Fourth of a four-part series. Click here to read the entire set of stories.
The aftermath of the cyberattack in Ukraine on Dec. 23, 2015, produced two unexpected lessons that U.S. grid operators have started to take to heart.
After cutting off power to nearly 250,000 homes and businesses in western Ukraine, the cyber terrorists delivered a final punch to the gut. The hackers wrecked some of the digital controls the operators needed to restart the system remotely. An aptly named cyber weapon called "KillDisk" hidden inside the Ukraine system erased parts of the operators’ startup software.
But substations across the Ukraine utilities’ grid networks still had Soviet-era manual controls, so crews were able to restore power by hand within six hours.
"It was the folks who got in trucks and knew where to go and drove out and found the breakers that had been tripped through the remote access tools," said Suzanne Spaulding, undersecretary of the Department of Homeland Security’s National Protection and Programs Directorate, in a blog interview.
Now, some leading U.S. grid officials, members of Congress and security experts are warning that old-fashioned protection might be needed for the more advanced U.S. power grid. Fail-safe cyberdefenses cannot be assumed in the age of the smart grid.
"We had this rush to automation over the last 15 years or so, on some level almost blind to security risks we are creating," said Scott Aaronson, executive director for security and business continuity at the Edison Electric Institute, which represents large, investor-owned utilities.
"It is good we have automation, which gives us better situational awareness. But it also increases the attack surfaces," he added, referring to the proliferation of sensors and controls that rely on software and connect to the virus-infected internet.
"Automation is driving incredible benefits," said Michael Assante, a director of the SANS Institute, a leading cybersecurity training firm. "We’ve consolidated and centralized a lot. You just need to keep in mind it also lets the bad guys do the same thing."
The brutal KillDisk finale in Ukraine demonstrated how attackers could conceal destructive malware that could re-emerge unless operators effectively cleansed their control systems. The Ukraine operators failed this test, experts agree.
Joint attack
The Ukraine attackers unleashed a second weapon that has jarred U.S. cyber strategists and corporate executives: the hacker’s ability to take down the utilities’ electric power distribution system and also attack at least one of the utility’s telephone call centers. The denial-of-service attack flooded the call center with counterfeit phone calls, preventing customers from getting through to report the loss of power, sowing more confusion and alarm among the grid operators.
"The attack in Ukraine gave us a taste of the threat to come," said Paul Stockton, managing director of Sonecon LLC and a former U.S. assistant secretary of homeland defense for the Defense Department. "That is just a small hint of the kinds of cross-sector attacks that may confront the United States."
The danger of such a one-two punch is a top-level conclusion in a new report to DHS Secretary Jeh Johnson by a cyber subcommittee of the DHS Homeland Security Advisory Council of corporate, academic and military and local government leaders.
Johnson ordered the subcommittee to address a major gap in federal cyberdefenses by finishing the DHS National Cyber Incident Response Plan (NCIRP). An interim draft of the plan was issued in 2011 but has never been completed. The lack of a final plan left key questions unsettled about how the federal government would respond to a major cyberattack on critical infrastructure, including how DHS and DOD duties would be divided, said Robert Dix Jr., a vice president for policy at Juniper Networks Inc., a Virginia-based network security firm.
The subcommittee released proposals last month calling for closer coordination of recovery plans by the communications, electricity and financial sectors. And it called on governors to work closely with federal agencies in the wake of a large-scale cyberattack.
"What we focused on was the wake-up call that the Ukraine attack should provide to the United States, in that it reflected a simultaneous attack on the communications and energy sectors," said Stockton, a co-chairman of the DHS advisory council subcommittee.
"It is the kind of attack that will require very intense cross-sector collaboration, of the sort that the new NCIRP needs to help be able to provide," Stockton said.
The case for simplicity
The assault in Ukraine dramatizes a crucial difference between the fallout after a natural disaster damages parts of the grid and the debilitating impact of cyberattacks that leave undetected but active malware hidden inside power systems.
"That is one of the big things about the Ukraine incident," Assante noted: If other utilities are attacked, how would they know that other malware isn’t still lurking after the initial attack ends?
"If they were hiding in other places, they could still be there," Assante said. "If we didn’t trust our electric substations and devices anymore, how do we deal with that? How would we bring it back? Those contingencies need to be considered."
Assante and two colleagues are among the experts arguing for a return to older control methods to safeguard the most important grid operations.
"The old analog relays and circuit protection devices were as reliable as the day was long," Assante, Tim Roxey and Andy Bochman wrote last year in a paper titled "The Case for Simplicity in Energy Infrastructure," published by the Center for Strategic and International Studies.
Roxey is a vice president of the North American Electric Reliability Corp. and head of its cyberthreat-sharing program. Bochman is senior cyber and energy security strategist at the Idaho National Laboratory.
"For every major piece of grid equipment, hundreds of digital devices have evolved to support it," the authors wrote. "Remote terminal units, intelligent electronic devices, programmable logic controllers, distributed control systems, field programmable gate arrays: these are specialized computers with circuit boards, memory chips, and communications circuits, the parts sourced from innumerable suppliers, and animated via instructions coded in software. And while the hardware brings loads of complexity, it’s in software that complexity truly runs wild."
One fallback position is to put more humans and nonprogrammable backup controls into systems on the most vital parts of the power grid, they said.
A generation ago, one of the authors recalled, utility systems were run by people like "Fred," he said, "who used to sleep at the substation with his dog. Give him an instruction to change a setting, and Fred would do it."
To defeat skilled cyberattackers, the most important grid components may need to rehire some "Freds" or create the equivalent with controls that are totally isolated from outside entryways, the authors argue.
Tom Fanning, chairman and CEO of Southern Co. and co-chairman of the Electric Industry Subsector Coordinating Council, the industry’s CEO-level security committee, has also been looking back to the future for security.
"What are the fallback positions? The electric power industry could be run manually," Fanning said at a recent conference. "We used to do it."
Not a cure-all
A revival of older controls is not a cure-all for every situation, said Marty Edwards, director of DHS’s Industrial Control Systems Cyber Emergency Response Team (ICS-CERT).
"Every entity has to evaluate that for itself," Edwards told EnergyWire. "It’s easier to maintain some semblance of manual control when you have human resources to deploy in one small region. But if you’re scattered across multiple states, that’s going to be tougher. So you have to make the determination where that’s important or critical."
However, the need for secure backup ways to restore power becomes vital once it’s accepted that attackers may get through the best defenses, said EEI’s Aaronson.
"It would be professional malpractice if we were putting all of our emphasis on ‘protect, protect, protect’ and not acknowledging that protection — while incredibly important — can’t be effective 100 percent of the time," Aaronson said.
Grid operators have to think about security holistically, he said: "Not just ‘protect, detect and defend’, but ‘respond and recover’; ‘security, not just protection.’"
"Are there things we can do today to be able to operate manually in the event of an incident: Go to a degraded state simply to keep the power running?" Aaronson added, speaking at a recent cybersecurity conference. "Those are the kinds of big decisions that we are taking as a sector and in partnership with the government, to begin to do the planning for those incidents that could have an impact for a longer term on the grid," he added.
"How do we make sure the inevitable bad day doesn’t become catastrophic?" he asked.
U.S. utilities — whose cyberdefenses vary widely in sophistication and strength — also have significantly different capabilities to recover from a major cyberattack, according to a top-level review issued in January.
A joint report by the Federal Energy Regulatory Commission and the North American Electric Reliability Corp., said a review of nine selected U.S. utilities showed that all had detailed plans for responding to and recovering from a widespread blackout. The nine utilities cooperating in the review were not named.
But the report went on to give 102 pages of ways in which the recovery plans should be bolstered, including increasing emergency startup and battery backup capacity to bring up systems after blackouts. It also called for upgrading restoration plans to account for a major change on the grid, including power plant closings. The idea is to test that recovery strategies can work in practice and confirm that spare systems and equipment will be available.
Lawmakers weigh in
The need for a fallback "manual control" has caught the attention of lawmakers in Congress.
The "Securing Energy Infrastructure Act," co-sponsored by Sen. Jim Risch (R-Idaho), chairman of the Senate Energy and Natural Resources Subcommittee on Energy, would task the Department of Energy’s national laboratories with testing "analog and nondigital" control systems’ ability to withstand remote cyberattacks. The legislation would free up $11.5 million to study the issue and report back in two years.
Not everyone is convinced that the modern grid needs a hands-on makeover.
Cris Thomas, a strategist at Tenable Network Security who also goes by the hacker name "Space Rogue," called efforts to pursue a manual mode at the expense of cyberdefenses "a step backward."
"It just seems like we’re spinning our wheels looking at this old stuff when we should be looking at the new," he said. Companies can apply better patches and use secure coding for state-of-the-art technologies.
Eric Spiegel, president and CEO of Siemens USA, a major developer of grid components, said at a recent cybersecurity event that much of the American grid "is old and needs to be modernized."
"A smarter grid will help prevent blackouts," he said. "But reliance on software and the Internet of Things means it gives more points of entry for people who want to harm us."
Russia’s ‘patriotic hacking’
The extent to which such a "smarter," more automated grid presents a risk to the United States also hinges on a candid assessment of the hackers who are capable of threatening it.
"There aren’t a lot of people globally that are capable of doing what happened in Ukraine," said Thomas, noting that most in that exclusive club are U.S. allies. "In the U.S., I think that you’re not going to see a similar attack against the power grid unless there are other factors involved, as well."
In Ukraine’s case, the attackers were widely believed to be based in Russia, and the sophistication of the attack pointed toward state sponsorship, according to multiple experts.
Jason Healey, director of the Atlantic Council’s Cyber Statecraft Initiative, agrees that accusing fingers point directly at Moscow. "I have almost no doubt in my mind, and that comes from a couple of lines of evidence, starting with people I trust who are savvy and not easily fooled," he said.
Some cybersecurity experts have cautiously labeled the Ukraine grid hackers a "Russian nexus," allowing for the possibility that they are advanced and organized attackers that nevertheless lack direct links to the Kremlin.
Healey suggested that such groups, if they were responsible for the power outages, still effectively work as proxies for the Russian state. "Russian ‘patriotic hacking’ goes back at least 10 years," he said. "If Vladimir Putin is sitting back and allowing these to happen, they don’t get a pass for that."
The question then becomes whether Russia would launch an attack on U.S. utility systems, given the response that would provoke.
Representatives from the United States and Russia carried out high-level talks on cybersecurity in Geneva, Switzerland, in April, according to a senior Obama administration official. Both sides brought up the possibility of expanding information-sharing between the two countries to reduce cyber risks to networks.
It’s not clear the Ukraine case was ever mentioned.
While the United States has publicly chastised and unsealed criminal indictments against hackers from rivals North Korea, Iran and even China, the administration has been quieter when it comes to Russia’s role as a global hacker.
"There isn’t the same level of signaling to the Russians, and I’m not sure why that is," said James Andrew Lewis, senior vice president and program director at the Center for Strategic and International Studies and a former foreign service officer who has worked on a range of cybersecurity and military issues.
Whether that silence will encourage Russia or other nations to push the boundaries of acceptable behavior in cyberspace remains to be seen. In the meantime, Lewis said, "we shouldn’t rest on our laurels" when it comes to grid cyberdefense.
"Does anyone feel confident in saying that if the Russians wanted to do this to us, that they would be unable to do so?" Lewis asked. "I’m not."
