Grid planners put ‘black start’ technology to the test

By Blake Sobczak | 11/13/2018 07:34 AM EST

PLUM ISLAND, N.Y. — Three inflated air dancers flapped incongruously over a deserted laboratory. The bright tube puppets would have fit right in at a used car dealership, but on this closely guarded, government-owned island, they stood out like fireworks in the driving rain as the Defense Advanced Research Projects Agency ran a cybersecurity exercise called “Liberty Eclipse” last week.

A view from the ferry to Plum Island, N.Y.

A view from the ferry to Plum Island, N.Y. Blake Sobczak/E&E News

PLUM ISLAND, N.Y. — Three inflated air dancers flapped incongruously over a deserted laboratory.

The bright tube puppets would have fit right in at a used car dealership, but on this closely guarded, government-owned island, they stood out like fireworks in the driving rain.

Powered by electric fans, the props were meant to signal that this part of Plum Island, at least, still had power during a grid cybersecurity exercise held here last week.


Walter Weiss, a program manager for the exercise at the Defense Advanced Research Projects Agency, called them "high-visibility power indication devices," cheekily adding more jargon to a week already swimming in cybersecurity and electrical engineering terms.

Weiss, whose glasses never seemed to fog up in the inclement weather, was already thinking of new obstacles to throw at exercise participants. His job was to put seven DARPA-funded research projects to the test, aiming to uncover gaps in the power grid’s defenses under dire, "black start" conditions, in which a crippling cyberattack brought the bulk power grid to its knees and forced operators to start from scratch.

"We try to make this as painful as possible." he said. "We want to find the limits of the tools, so we drive them to the point where we see how far they get.

"Then we beat them back down, and that’s when they start getting really upset with us," he added with a grin.

Plum Island, a strip of windswept land off the eastern tip of Long Island, offered a rare spot to unplug from the wider grid and run a cybersecurity exercise on live, 13.2-kilovolt wires. DARPA orchestrated the weeklong "Liberty Eclipse" exercise alongside the Department of Energy, National Guard and Department of Homeland Security, which controls access to the island, roughly the size of Central Park.

"No one’s really tried to do this before," Weiss told a group of reporters invited to observe part of the event, which ran from Nov. 1 to Nov. 7 (Energywire, Aug. 3). "How do you go from a policy tabletop [exercise], to the technologies — the ones and zeroes — and then feed back to the policy community what actually worked on the ground?"

Restoring trust

Last Tuesday, the exercise participants were off to a good start. The yellow, red and green tube men flailed against the downpour above decrepit Building 257, which once housed a U.S. Army lab for researching germ warfare. The DARPA researchers and utility players had been tasked with preserving power to Building 257 at all costs, while working to re-energize the wider grid.

But Weiss and other planners periodically tossed wrenches into the works, simulating a steady onslaught of cyber and physical attacks. Later that afternoon, they would introduce a data "wiper," modeled off real-world cases of ransomware, which could send grid operators back to square one if they weren’t careful.

The event offered a dress rehearsal for nascent technologies in a three-year-old DARPA research effort dubbed RADICS, short for "Rapid Attack Detection, Isolation and Characterization Systems."

The RADICS program, which kicked off in 2015 with a $77 million federal funding announcement, is aimed at ensuring U.S. utilities can bounce back from a blackout brought on by a cyberattack. Grantees had to assume the worst: that utilities’ operational networks, including sensitive field equipment, have been compromised by hackers.

"How do I know that substation’s working? How do I know if I can trust that substation? How can I trust the communications, and what do I have to fix to make sure it’s working?" said Gary Seifert, an engineering contractor who helped build out the physical infrastructure for the Liberty Eclipse scenario. "There is a lot of steps that go into that."

Researchers from defense contractors Raytheon Co., BAE Systems PLC, Perspecta Labs and other DARPA funding recipients dreamed up a host of solutions, from adding a second, backup layer of sensors and communications channels tailor-made for grid emergencies, to sending up a specialized balloon to offer a bird’s-eye view of grid restoration, scanning for Wi-Fi hotspots and electromagnetic signals to map out where electrons are actually flowing.

Weiss said DARPA is working to prepare a public after-action report to map out next steps for the RADICS program and key in on any major weaknesses.

The Department of Energy will draft its own set of takeaways, having completed a related tabletop exercise in October in addition to joining the on-the-ground team at Plum Island.

Brian Marko, exercise program manager for DOE’s new Office of Cybersecurity, Energy Security and Emergency Response, cast Liberty Eclipse as a way to get utilities "in front of a safe and secure environment" to see how federal workers, researchers and engineers would band together during a disaster, "so we’re ready for the real thing, if, God forbid, it ever happens."

Worst of the worst

Dozens of representatives from major utilities and industry groups, including the New York Power Authority, Duke Energy Corp. and the National Rural Electric Cooperative Association, trekked out to Plum Island to take part in Liberty Eclipse.

The baseline scenario was built around mind-bending bad news: swaths of the U.S. grid had already been offline for a month, exhausting battery backups at power plants and substations alike.

Participants were sorted into two main groups, each with their own control center: Utility A and Utility B.

Both fictional power companies would need to build a "cranking path" to bring the lights back on, starting with a "black start" generator up to the task.

Much like building a fire, restoring power to a completely blacked-out grid requires utilities to start small. Black start generation resources — typically diesel or gas-fired units — provide the kindling. With enough electricity flowing through pockets of the interconnected grid, large coal or nuclear-fired power plants can be piled on like logs, and utilities can ultimately resume normal operations.

"When you go out to your car every day, you turn your key over and it starts the motor, right?" Seifert explained. "Any generator bigger than 30 or 40 megawatts has to have a smaller generator to give it power."

Black start events are exceedingly rare. The 2003 North American blackout, which affected some 50 million people in the U.S. and Canada, was one of the most recent real-world tests.

But U.S. lawmakers, including the chairwoman of the Senate Energy and Natural Resources Committee, have urged the power sector to review their black start resources, just in case.

Sen. Lisa Murkowski (R-Alaska) pointed out at a hearing on the subject last month that "the increasing risks presented by cyberattacks — and the threats of electromagnetic pulses and solar storms — make it more important that we be prepared" (Energywire, Oct. 12).

Liberty Eclipse imagined just such a disaster: "as bad as it possibly could be for the country," as Seifert put it. "In reality, hopefully it never happens."

Building 257

On Day 1 of the exercise, Utility B received an order from Energy Secretary Rick Perry: Whatever else happens, keep a critical facility online — Building 257.

The order was intended to practice real emergency authority that lawmakers granted DOE three years ago under the Fixing America’s Surface Transportation Act of 2015.

"DOE has the right to basically say: ‘Utility, this is more important than your normal black start; you have to get this one up first and keep it on all the time,’" Weiss pointed out.

He didn’t specify the precise nature of the critical facility.

Beyond the confines of the exercise, Building 257 served as an animal disease research center for decades after changing hands from the U.S. Army to the Department of Agriculture in the middle of the last century. Its real-world research mission passed over to another building on Plum Island, which remains in use and was kept isolated from the RADICS event on a separate grid.

During Liberty Eclipse, the worn-down structure, now a mess of peeling paint and chipped windows, was vaguely declared "critical to the national defense of the country."

"Utility B has quite a burden on their back: They have to keep their control center alive, and they have to keep their critical facility online," Seifert said.

At first, Utility B brought power back by reverting to "manual mode" — assuming computer systems couldn’t be trusted, and sending workers out to physically flip switches and restore electricity to Building 257 and its supporting substation.

In December 2015, grid operators in Ukraine relied on the same strategy to recover from one of the few real-world examples of a cyberattack on a power system. Facing a coordinated assault on their operational networks, three distribution utilities in western Ukraine resorted to sending line workers into the field to bring the lights back on in a matter of hours.

"We’re paying very close attention to what those real threats are," Weiss said.

He pointed out that the exercise series could return to Plum Island as soon as next spring to gather more data for researchers, policymakers and power utilities.

"How do we make this part of how people exercise and prepare?" Weiss said. "I think we’ve created something that other people don’t have yet."