Grid authorities have issued a record $10 million fine to an unidentified utility over more than 120 security violations spanning four years.
The North American Electric Reliability Corp., which sets and enforces cyber and physical security standards for the bulk U.S. power grid, said the series of lapses "collectively posed a serious risk to the security and reliability of the bulk power system," even though they aren’t known to have triggered any blackouts.
"Many of the violations involved long durations, multiple instances of noncompliance, and repeated failures to implement physical and cyber security protections," NERC concluded in a partially redacted notice of proposed penalty posted Friday.
The bulk of the violations took place between 2015 and 2018, though some cybersecurity vulnerabilities have yet to be fixed, according to NERC. The nonprofit grid overseer declined to identify the recipient of the fine, citing ongoing security concerns.
"This is huge," said former NERC executive and independent consultant Earl Shockley, adding that the notice has already shaken the industry.
Shockley pointed out that critical infrastructure protection (CIP) standards are the most frequently violated of all the voluminous grid reliability rules, which cover everything from pruning trees to modeling power system flows. The CIP standards are laser-focused on cyber and physical security safeguards and date back to 2008. They lay out requirements for controlling access to sensitive buildings like substations, while also setting baseline protections for "critical cyber assets" like the specialized computers in grid control centers.
The 127 violations listed in the penalty represent an "ad hoc, informal, inconsistent, chaotic" approach to NERC CIP, according to Shockley.
"I’m really surprised to see an organization fail at this magnitude, this deep into the NERC [CIP] program," he said. "It’s obvious to me as an ex-regulator: This is a culture problem."
Who was it?
Pacific Gas and Electric Co. held the previous record for the largest public NERC CIP-related fine, according to multiple sources. That firm was hit with a $2.7 million penalty last year for leaving sensitive utility data accessible online over nearly three months in 2016 (Energywire, April 17, 2018).
PG&E has since filed for bankruptcy protection over its exposure to up to $30 billion in legal damages from several deadly California wildfires.
The Federal Energy Regulatory Commission gets the last word on NERC security standards and related fines. That agency has until late February to formally sign off on the $10 million settlement reached between NERC and the "unidentified registered entity" in the penalty, a spokesman confirmed. FERC has only rarely tweaked such penalties in the past.
The structure of the NERC filing points to a large utility holding company with subsidiaries based across multiple parts of the country.
"It’s got to be a huge company that has assets all over the place — and not just generation," said independent utility consultant Tom Alrich.
The anonymous company has agreed to overhaul its program for complying with NERC’s CIP standards, according to the filing.
Most of the alleged violations outlined in the document were "self reported" to NERC, but more than a dozen of them were found only through audits of the utility’s security program.
The 765-page penalty notice reads like a highlight reel of grid security mishaps, from improperly vetting software updates to bungling firewall settings to potentially allow hackers to patch into critical computer networks.
In one case, a company manager fired an employee but neglected to notify the IT help desk so it could revoke their access to a sensitive computer system. Three days elapsed before anyone caught the oversight and blocked the former worker from being able to access the networks.
In another episode, a "security specialist" disconnected a network cable from the back of a critical cyber asset and plugged it into his laptop, which hadn’t been vetted for potential malware.
‘Substantial damage’
The timing of these security violations overlapped with some of the most aggressive cyber espionage campaigns ever directed against U.S. utilities, including a series of 2017 intrusions that targeted multiple companies that own nuclear power plants (Energywire, June 27, 2017).
For instance, on July 26, 2017, three technicians reported to an unidentified site to change passwords on some relay systems. Protective relay devices are designed to keep an eye out for faults on power grid systems, quickly severing the flow of electricity if something goes wrong.
"The technicians then began work at 1:40PM with their [redacted] issued laptops; not the dedicated CIP [transient cyber asset] laptop located at the site," a heavily redacted report from the utility said. "Technicians overlooked the label on the corporate issued laptops stating ‘Not permitted for communication with CIP Cyber Devices.’"
The threats from such oversights aren’t theoretical, although hackers aren’t known to have ever caused a power outage in the United States.
In 2015, suspected Russian hackers remotely logged into the control networks at three distribution utilities in Ukraine, tripping breakers and shutting off the lights to more than a quarter-million people for several hours. The hackers struck again in 2016, cutting off power at a transmission substation outside Kiev for several hours by using specialized malware.
On Tuesday, U.S. Director of National Intelligence Dan Coats warned in congressional testimony that hostile nation-states like Russia and China have similarly penetrated U.S. critical infrastructure systems, including gas pipelines and power utilities (Energywire, Jan. 30).
"Russia has the ability to execute cyber attacks in the United States that generate localized, temporary disruptive effects on critical infrastructure — such as disrupting an electrical distribution network for at least a few hours — similar to those demonstrated in Ukraine in 2015 and 2016," Coats said. "Moscow is mapping our critical infrastructure with the long-term goal of being able to cause substantial damage."
