Correction appended.
Power industry leaders and senior federal officials have set out to create a playbook for restoring electrical service after a crippling natural disaster or extraordinary cyber or physical attack on grid networks, grid officials say.
The task follows the second day of the GridEx III "war games" exercise last month, when utility executives and federal and state officials were confronted with a simulated catastrophic loss of grid facilities.
"We subject executives — CEOs from utilities and very senior government personnel — to a really bleak scenario that puts many, many people in the dark for a long time and puts them [the leadership] under stress to figure out what they’re going to do … to help the nation get back on its feet as quickly as possible," said Andrew Bochman, senior cyber and energy security strategist for the Idaho National Laboratory’s national and homeland security directorate, speaking to a meeting of the National Conference of State Legislatures last week.
"Grid planners and operators can identify things we can do today to deal with potential threats and incidents," said Scott Aaronson, senior director for national security policy at the Edison Electric Institute, which represents investor-owned U.S. utilities. "We don’t want to have those conversations for the first time during an incident."
Aaronson said he would not anticipate the specific recommendations from GridEx III, which will be published early next year by the North American Electric Reliability Corp. (NERC) after canvassing more than 4,000 industry and government officials who participated. NERC, the high-voltage grid’s security monitor, ran GridEx III.
Counting on ‘mutual assistance’
Several goals are obvious from the event, Aaronson and other participants said.
One is to create a plan for mutual assistance among utilities if potentially far-ranging cybersecurity attacks hit the grid, just as utilities follow when sharing repair equipment and crews after a natural disaster.
Another is to pin down how federal agencies would support the power industry in restoring power if attacks have damaged critical facilities. This task will belong to NERC’s Electricity Sub-sector Coordinating Council, senior executives from across the industry, working with top administration officials.
"There is a culture of mutual assistance" in the utility industry, Aaronson said. "We have been doing it literally for decades."
There are playbooks that are exercised in real emergencies, such as storms, fires or earthquakes, to send crews and bucket trucks to restore service in defined areas. "When it comes to cyber responses, it’s much less clear," he added. "Was it a cyberattack? Or was it an equipment failure? When does the attack stop? Do we have the appropriate number of human resources?"
A higher level of collaboration among industry and government agencies is needed on this front, he said.
Congress gave grid and federal agency officials a crucial assist when it passed new grid emergency legislation this month, said Paul Stockton, managing director of Sonecon LLC and a former assistant secretary of Defense for homeland defense. The emergency powers were tacked onto the Fixing America’s Surface Transportation (FAST) Act at the last minute as it neared final House and Senate clearance. President Obama signed the bill into law earlier this month.
The law gives the secretary of Energy limited authority "to protect or restore the reliability" of critical electric infrastructure facilities during a presidentially declared emergency.
"Determining exactly how the secretary of Energy might best employ this authority — the concept of operations by which government agency can most effectively support utilities as they restore power — is the challenge that lies ahead," Stockton said in an interview.
‘Islanding’ the grid
Designing a recovery plan for segments of the high-voltage power grid begins with existing protective strategies and requirements but can’t stop there, experts say.
"The grid has procedures to island itself" in the face of dangerous power flow disruptions, said Johannes Pfeifenberger, a principal with the Brattle Group consultancy. "A lot of protective relays are set to drop off and isolate sections" to prevent outages from cascading over wide areas.
"The relays act automatically. There would be blackouts, but you need to have that in the basic backbone of the grid so [we can] bring it back" after an emergency, said Jason Christopher, senior technical leader for cybersecurity at the Electric Power Research Institute.
"This is pretty much what has happened during some of the big reliability events," Pfeifenberger said.
In the 2003 Northeast blackout, the largest to hit the United States and Canada, massive power surges caused by a series of power line outages in eastern Ohio raced eastward into New York and around Lake Erie to Michigan, knocking out electricity to 50 million customers. But relays separated most of New England from the surge, and its grid "wobbled" but stayed up. The cascading outages were limited to eight states and Canada’s Ontario province by automatic relay operations, without human intervention.
"Every transmission operator has to have a restoration plan and enough ‘black start’ capability to bring the grid back" following an outages, Pfeifenberger added. Black start units — typically small generators — are able to start without the need for outside power. Power from the black start units then energizes large generators on the network.
Recovery plans, however, are divided among grid subregions, leaving open questions about how a widespread, devastating natural disaster or terrorist attack would be managed, some experts said.
"The companies that own and operate the bulk power system have procedures to posture the grid proactively should the need arise, such as in the event of severe geomagnetic activity," noted NERC spokeswoman Kimberly Mielcarek.
"Grid operators maintain detailed plans to restore service following a wide range of disturbances, including through the use of black start resources when the situation dictates. These plans are regularly exercised in drills and simulations," she said.
"Real-time operational decisions in response to emergency situations are made by the companies responsible for operating the grid," she added, with designated grid reliability coordinating centers overseeing operations in each of their geographic areas, under NERC emergency standards.
"Where it might get more complicated is when multiple points and multiple systems could be attacked at the same time," Pfeifenberger said. "You could imagine that something might happen that doesn’t allow the system to react as planned or as designed. And that could have far more widespread impact.
"If the outages damaged major equipment, that could create some widespread outages. Even with black start restoration, if some of the major transmission equipment is knocked out and damaged, you might not have the transmission system in place to fully restore it," he said.
The power industry has strong foundations on which to build advanced recovery plans, Stockton said. "But as we anticipate especially severe nontraditional hazards, whether they are coordinated kinetic attacks, cyberattacks or even very large seismic events, then additional work needs to be done on how to accelerate the power restoration industry" with government support of industry actions, he added.
"Currently, restoration plans are developed and implemented on a regional basis and not necessarily to meet national or international priorities," said Brian Harrell, a director of Navigant Consulting Inc.’s energy practice and a former NERC grid cybersecurity official.
"A ‘unity of effort’ is required to optimize electricity sector and government roles and resources, depending on the circumstances of the situation at hand. However, the sheer scale of a severe attack scenario would challenge existing emergency response processes in the electricity sector and government.
"It is fair to say that emergencies with national security implications present unique challenges that are unknown to utilities and may impede restoration," Harrell said.
The extraordinary attack on many points of the grid presented on GridEx III’s second day is not considered a clear and present danger by experts, some of the event’s planners said. Instead, it was deliberately exaggerated to push the participants with a worst-case scenario.
"You want to exercise what you would do the most extreme conditions you can imagine," EPRI’s Christopher said.
EPRI cybersecurity expert Annabelle Lee testified recently to a House committee that she would be surprised if a cyberattack succeeded in taking down a major U.S. city’s electrical supply. But the GridEx III exercise "was not overstated," she said in an interview. "To protect the grid, we have to think of everything. All an attacker has to do is think of one thing. … You don’t just ignore it."
There may come a day when adversaries will succeed, Aaronson said at the legislators’ meeting last week. "We as a sector have to be prepared to respond and recover from that day," he said.
Correction: An earlier version of this story misspelled the name of the Electric Power Research Institute’s Annabelle Lee.
