Grid crisis plan gets pandemic test

By Peter Behr | 04/01/2020 07:16 AM EDT

The nation’s electric power industry, defending now against an all-too-real coronavirus pandemic, got a report card yesterday on an imagined nightmare — a devastating  cyber and terrorist attack that was simulated in a nationwide preparedness exercise in November.

The nation’s electric power industry, defending now against an all-too-real coronavirus pandemic, got a report card yesterday on an imagined nightmare — a devastating cyber and terrorist attack that was simulated in a nationwide preparedness exercise in November. Varistor60/Wikipedia

The nation’s electric power industry, defending now against an all-too-real coronavirus pandemic, got a report card yesterday on an imagined nightmare — a devastating cyber and terrorist attack that was simulated in a preparedness exercise in November.

The report on the nationwide GridEx V training exercise was released yesterday by the grid’s security monitor, the North American Electric Reliability Corp. (NERC). Online "war games" challenged more than 7,000 utility officials, government leaders and other participants to defend against and recover from coordinated physical and cyber assaults, including a hacking attack based on the real takedown of part of Ukraine’s grid by Russia in 2016.

NERC official Matt Duncan said yesterday the grid organization does not record how well or poorly the participating utilities did in combating a version of the CrashOverride malware unleashed in the 2016 cyberattack.


"We don’t grade the utilities," said Duncan, senior manager of resilience and policy coordination for NERC’s Electricity Information Sharing and Analysis Center, the industry’s cyberthreat quarterback. "They grade themselves." Of 526 utilities, agencies and organizations that participated, 148 submitted self-evaluations.

The cybersecurity firm Dragos Inc. warned in February that attacks on control systems worldwide continue to increase and said, "Correspondingly the potential risk due to a disruptive cyber event impacting the North American electric sector is currently assessed as high" (Energywire, Jan. 10).

Smaller utilities generally don’t have the level of cyberdefense expertise and resources that large power providers can afford, industry leaders acknowledge, and that issue surfaced again in the November tests.

"That’s a gap we need to be honest about and need to be getting better at," Duncan said. The industry is developing a Cyber Mutual Assistance Program to help smaller utilities, patterned after the mutual assistance and crew support utilities offer each other after major storms, he noted.

GridEx V did not add a pandemic to the scenario, but Duncan said the coronavirus emergency has improved teamwork among the electric power industry and the natural gas pipeline and telecom sectors, which was a top priority that those sectors practiced in the exercise. The cooperation has strengthened the grid’s preparations to deal with supply emergencies if the COVID-19 pandemic hits control rooms, he said.

Duncan was asked about parallels between the grid industry’s preparations for extraordinary "black sky" disasters and how the Trump administration, state governments and the public health sector have faced up to the coronavirus pandemic.

"I won’t speak to the federal or state response," he replied. "I will speak to the industry response and how it relates to GridEx training."

A top concern from the 2013 exercise forward has been "making sure you speak with one voice: providing guidance and one source of truth. That’s what’s playing out now in COVID-19," Duncan said.

"The only way we have grown that culture within the electricity industry is through GridEx and the opportunity to connect very senior utility executives with their colleagues at all levels of government and other sectors," he said.

A major addition to the exercise’s second day was a simulated cyber and armed attack across New York and southern Ontario, which was designed as a trial run of the extraordinary standby authority given by Congress to the Energy secretary to direct utilities’ actions in a presidentially declared grid emergency. There is no parallel for that process in a pandemic.

Narrowing the scenario to a single state gave senior utility executives and federal agency leaders the chance to consider how to handle emergency power shortages that would require decisions on who should be given priority for electricity — decisions with potential life-or-death consequences and uncharted issues of financial liability, officials note.

The report’s conclusions on future actions cited a need to "build consensus with [the Department of Energy] on the design of actions and liability protection for unprecedented operations actions" in grid security emergencies.

"The industry believes firmly that the government should set priorities in emergency situations," Duncan said. Then, government "should rely on utility expertise" to manage grid recovery, he added.

Supply chain risk

Map of 2015 GridEx security exercise simulation. Photo credit: North American Electric Reliability Corp.
A map showing simulated grid disruptions from the 2015 GridEx security planning exercise. The biennial exercise is designed to test how grid operators would respond to a wide-scale, combined physical and cyber assault on electric infrastructure.

The GridEx report noted progress — and much still to be done — in managing the supply chain risk to grid operators that critical controls from vendors may contain hidden malware.

The FBI sent a nonpublic alert Monday warning of ongoing state-sponsored cyberattacks against supply chain vendors. The alert, first reported by ZDNet, focused on attacks against the health care industry but said that vendors who supply software and services that maintain industrial control systems for the energy sector were also among those "heavily targeted."

Some supplier representatives, not named by Duncan, joined in the simulated attack exercise in New York and Ontario last November, opening conversations on how to deliver emergency spare equipment in critical situations, he said. "That was huge," he said.

But the report found more is needed to align a regulated, largely cooperative grid industry with an intensely competitive group of unregulated, high-tech suppliers, Duncan said. It proposed considering a shared inventory program for critical components and offering guidance to utilities on how to pinpoint their most crucial vendor equipment.

Duncan said utilities have generally done well in lining up personal protective equipment for operators’ use in the COVID-19 emergency because the industry has a track record of looking ahead three, six and nine months for spares that must be secured in advance from a global supply chain.

"I can’t underscore how important it is to have equipment vendors … involved in this process," he said.

The confusion over securing test kits, protective gear and ventilators for hospitals is an obvious contrast to common practice for utilities, where planning and forecasting are daily essentials, said Jim Cunningham, executive director of Protect Our Power, a grid security advocacy group.

"These are things that utilities for the most part are doing every day," Cunningham said in an interview.

The reaction to the coronavirus, on the other hand, has been splintered.

"Unfortunately, there were a lot of folks — and even some today — who don’t believe it’s a serious issue. That lack of belief gave us a late start," Cunningham said.

"Now, for the most part, our [national] leadership is taking it very seriously, but it took too long to get to that point and we’re paying the price for that," he added.

Yesterday, New York Gov. Andrew Cuomo (D) protested again at the hurdles governors are facing while competing with one another and with the Federal Emergency Management Agency for ventilators. "So FEMA is driving up the price. What sense does this make?" Cuomo said at a media briefing. "The federal government should have been the purchasing agent."

Caitlin Durkovich, the Department of Homeland Security’s assistant secretary for infrastructure protection in the Obama administration, said it’s easier for power providers "to wrap arms around things they know" than for the health care sector to plan for a pandemic of unprecedented dimensions. "It’s why the electric sector has gotten pretty darn good at hurricane response," said Durkovich, a principal with Toffler Associates, an emergency preparations consultancy. "They live it. They practice it and get together after each one to talk about how they can improve, not just as a company but across the sector."

‘Failure of imagination’

A "failure of imagination" was one of the missteps that paved the way for the Sept. 11 terrorist attacks, the official 9/11 Commission report concluded.

The COVID-19 pandemic could and should have been imagined and prepared for, Durkovich said, citing a threat briefing that the incoming Trump administration received from Obama administration transition officials.

In the case of the coronavirus, "too many people didn’t really believe it could happen, or if it did [government] would swiftly move into action and block it," Durkovich said. And then it was too late to stop the virus from spreading rapidly.

On several occasions, President Trump stated that this crisis had defied prediction. "Nobody knew there would be a pandemic or epidemic of this proportion. Nobody has ever seen anything like this before," he said at a news conference. "Nobody ever thought of numbers like this."

At the same time, he and his aides have defended the federal response. White House spokesman Judd Deere told The New York Times that "any suggestion that President Trump did not take the threat of COVID-19 seriously is false."

Duncan suggested that imagining a pandemic isn’t likely to be an issue for the next GridEx event, in 2021, though he said NERC officials will take a "hard look" at the pandemic and consider how it might fit in a future exercise.

"We say GridEx is designed to overwhelm even the most prepared utilities, and we use realistic scenarios" to challenge participants, he said. "We do make each GridEx tougher than the last, and there is no shortage of threats out there. But the only way we are going to get better … is practicing against those threats."

Reporter Christian Vasquez contributed.