Second of a four-part series. Click here to read part one.
A month after hackers blacked out power in western Ukraine, a team of U.S. security experts touched down in Kiev to piece together the extraordinary assault.
Interviews, cellphone video evidence and a crash course in Soviet-era grid equipment helped the dozen or so Americans untangle the Dec. 23, 2015, cyberattack on three utilities. The investigators traveled thousands of miles with one big question in mind: Could the methods used to hack the Ukrainian power distributors, or the hidden code behind the strike, pose a threat to the U.S. electric grid?
But two days into the five-day mission, analysts working in an opaque intelligence aggregator at the U.S. Department of Homeland Security reached their own conclusion. The Ukraine case did not pose any particular risk for U.S. systems, according to a Jan. 27 DHS memo marked "For Official Use Only."
Weeks later, a separate branch of DHS flipped that conclusion on its head, delivering the first in a series of stark warnings to electric utilities and other operators of U.S. critical infrastructure.
The conflicting and drawn-out response to the hack has triggered pointed criticism about DHS’s ability to deliver cyberthreat intelligence outside the walls of government. The agency is supposed to spread the word about fast-moving online threats to the networks that underlie everything from the bulk power grid to car factories. But in the case of the Ukraine hack, the first of its kind, it took two months for DHS to disclose lessons from the incident and three more months to provide additional guidance accounting for the attackers’ techniques.
"There was a credible threat to the U.S. grid, with realistic mitigations that could have been applied, and instead [DHS] decided to sit on the information," said Robert M. Lee, founder of Dragos Security LLC and a co-author of an influential SANS Institute analysis of the Ukraine case.
"In the midst of the first attack on a power grid that was public, there was no public word from the government," he said.
The war that had been raging in Ukraine for two years was a major source of frustration for U.S.-Russia relations. The agency was struggling to field requests from the utility industry and private analysts to share what the U.S. government considered sensitive information.
Some industry officials had an inside track on earlier attack details, including executives with security clearances and members of the CEO-level Electricity Subsector Coordinating Council, the industry’s principal liaison with the U.S. government on security issues. But the broader power sector would have to wait.
The hackers in Eastern Europe had preyed upon equipment and technological vulnerabilities also present in North America’s energy infrastructure, even repurposing a malware strain that was unearthed in U.S. systems in 2014.
As DHS officials kept largely quiet, utilities relied on private cybersecurity firms and media reports to fill in the blanks about the methods hackers used. Experts say the early lack of widely shared, actionable data could have left some companies exposed. And that has put DHS at the center of concerns about the effectiveness of cyberthreat-sharing from the U.S. government to the private sector, which controls the vast majority of the nation’s critical infrastructure.
"If the U.S. government is seeking to achieve a real partnership with the private sector, what is their value-added proposition?" said Susan Hennessey, a fellow in national security law at the Brookings Institution and managing editor of Lawfare.
Unimportant or ‘imperative’?
DHS was still trying to pin down details of the Ukraine attack a month after it happened.
During the on-the-ground investigation in western Ukraine from Jan. 25 to 29, DHS’s Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) joined representatives from the Department of Energy, the FBI and the North American Electric Reliability Corp. (NERC), which develops and enforces cybersecurity rules for the high-voltage bulk power transmission grid.
The itinerary was secret. But the fact-finding mission came as no surprise, given that the hourslong grid takedown in Ukraine was without precedent in the brief history of cyber conflict. Private security firms had already concluded in early January that evidence pointed to computer hackers, not some other form of sabotage or human error.
Still, officials at DHS headquarters saw no reason to wait for investigators to return from Ukraine before issuing a threat assessment. On Jan. 27, with the investigators on the ground, DHS’s Office of Intelligence and Analysis (I&A) published an analysis titled, in bold letters, "Damaging Cyber Attacks Possible but Not Likely Against the U.S. Energy Sector."
The report from I&A, which reports directly to DHS Secretary Jeh Johnson, said it "is unable to confirm the event was triggered by cyber means," citing "limited authoritative reporting."
I&A is tasked with analyzing top-secret intelligence, and it’s charged with being a DHS conduit to state and local authorities. Its direct access to DHS’s chain of command also puts it at the center of gravity as the agency considers rising threats. But the office has faced sharp criticism from Congress about its effectiveness, and it has fought turf battles with the FBI over who is tasked with distributing information about domestic threats.
The I&A report, which was later leaked and published by the Public Intelligence accountability and transparency research project, concluded that "this incident does not represent an increase in the threat of a disruptive or destructive cyberattack on U.S. energy infrastructure, which I&A assesses is low."
In explaining the reassuring finding in a footnote, I&A said it was based on the earliest views of the attack expressed at a Jan. 4 meeting that included DHS and industry officials.
But the I&A outlook crumbled fast. DHS’s view switched 180 degrees two weeks after the U.S. team returned home. In a February alert pushed out to electricity providers, DHS officials warned of a potential threat against utilities. The seriousness of DHS alerts to industry only escalated from there.
On March 7, the department released a detailed breakdown and alert about the attack and cited an "urgent need" for grid operators and other critical infrastructure owners to take "enhanced cyber measures" to protect themselves.
On the same day, Andy Ozment, DHS assistant secretary for cybersecurity and communications, and Greg Touhill, the deputy assistant secretary in the same office, stated that while there was no evidence of a Ukraine-level attack underway in the United States, it was "imperative" to raise defenses against what happened there.
The DHS alert put the risk in stark terms.
"It is the assessment of ICS-CERT that critical infrastructure [industrial control system] networks, across multiple sectors, are vulnerable to similar attacks," the alert said.
DHS officials rejected repeated requests from EnergyWire for interviews and information about the department’s response to the Ukraine attack and any lessons the agency learned.
By spring, senior DHS officials had switched gears from silence about the threat to elevating Ukraine to a top priority.
"It is incredibly important," said Suzanne Spaulding, DHS undersecretary for the National Protection and Programs Directorate (NPPD), in an April 12 podcast interview with a Washington law firm. "We are beginning a multi-city campaign across the country to make sure we get the word to critical infrastructure owners and operators about what happened there."
Spaulding said the "good news" is that the U.S. government knows how to protect against and mitigate a Ukraine-style attack on critical control systems. "But folks have to take steps. They have to take action. They have to understand this is not just something that has the potential to affect the electric grid," but something that could affect any Internet-connected critical infrastructure organization, she said.
NERC, the U.S. grid overseer, has maintained that the impact to the U.S. bulk electric power system would be blunted by best practices and binding federal critical infrastructure protection standards, the latest version of which took effect this month. But the standards rarely trickle down to small electric utilities.
"The grid in North America is larger and more diverse in the design and configuration of its equipment, including industrial control systems," NERC spokesman Martin Coyne said in response to EnergyWire‘s emailed questions. "As part of the industry’s best practices, these systems run on licensed software and are routinely screened for potential threats including malware, which is not the case in Ukraine."
A BlackEnergy link
But there is at least one known and ominous similarity between the Ukraine systems and U.S. electric utilities — the presence of BlackEnergy, a powerful, elusive intrusion malware that can give attackers a hidden opening to victims’ systems. DHS has issued a series of warnings that BlackEnergy 2 has broken into the U.S. grid.
The similarities between the U.S. and Ukraine strains were so striking that DHS reposted the technical indicators in its original 2014 alert on BlackEnergy 2 to help companies root out its newer cousin, BlackEnergy 3, which was spotted on the Ukraine system.
NERC said BlackEnergy 3 has not made its way across the Atlantic.
"There is no credible evidence that the incident could affect North American grid operations and no plans to modify existing regulations or guidance based on this incident," NERC spokeswoman Kimberly Mielcarek said Jan. 7, three days after her colleagues huddled in a closed-door meeting with DHS to talk about what happened in Ukraine.
The regulator posted a confidential alert about the cyberattack, including recommendations, to members of its information-sharing portal in early February and asked U.S. utilities whether they had defenses in place against the series of weapons unleashed against Ukraine.
A month later, NERC shared a public analysis of the attack prepared by experts at the SANS Institute, a Bethesda, Md.-based influential cybersecurity training and research nonprofit.
That paper concluded that "nothing about the attack in Ukraine was inherently specific to Ukrainian infrastructure." It could happen elsewhere.
Duane Highley, CEO of the Arkansas Electric Cooperative Corp., who serves as co-chairman of the CEO-level Electricity Subsector Coordinating Council, testified before Congress last week that the Ukraine event offered a case study for how government could improve information sharing with his industry.
"While the content of the classified and unclassified information from the government was very helpful, the timeliness of getting specific, actionable information to industry must be improved so that we can respond as quickly as possible," Highley said in prepared remarks before the Senate Energy and Natural Resources Subcommittee on Energy.
He elaborated on his concerns in a follow-up interview last week, describing how private-sector experts pointed out that there was a vulnerability but were initially barred from sharing the details.
"We’ve got to become a closer partner with the Department of Energy and DHS, and we need to continue to develop greater trust, because we are on the front lines of the war," Highley said. "It used to just be the army fighting the war; now we’ve got the private sector fighting the war" in cyberspace.
Highley was optimistic that information sharing would improve post-Ukraine, based on follow-up conversations with administration officials and steps to implement information-sharing legislation.
"They just need to move a little faster," he said.
In fits and starts, DHS and Congress have worked toward reorganizing the agency to prioritize its industrial cybersecurity mission, in part by renaming the nondescript National Protection and Programs Directorate as the Cybersecurity and Infrastructure Protection Agency.
SANS Institute’s Lee, a critic of DHS, said the agency threatened to take legal steps to block private analysts from sharing their early findings about Ukraine on security grounds. But he drew a distinction between the agency’s experts and DHS’s political arms.
"They’re patriots. They’re doing amazing work," he said. "The problem is with the bureaucracy of the larger government, where senior government leaders do not understand the technology, they don’t understand the impact, and they don’t understand the threat, but they’re trying to limit what is said to the community."
‘Are we prepared?’
The early warnings about the Ukraine attack’s threat came from private industry cyber forensic specialists. For cybersecurity researcher Chris Sistrunk, the alarm arrived on Christmas Eve in a Twitter message from a trusted colleague in his close-knit circle of security professionals, Marina Krotofil. She enclosed a link to a Ukrainian-language news article.
Sistrunk couldn’t believe what she was sharing: Hackers had reportedly knocked out power to hundreds of thousands of Ukrainian electricity customers.
"We were questioning if it was a real attack or not," Sistrunk said.
When he and Krotofil found mentions of the cyberattack on power companies’ public websites and Facebook pages, their suspicions inched toward certainty. The Ukraine outages looked like they really were the work of hackers.
Sistrunk dialed ICS-CERT, the government’s first line of defense against cyberthreats to electric infrastructure. He also notified NERC, which runs its own secure threat information-sharing site.
He said leaving the messages was "just a good thing to do" on the chance there was a similar intrusion and takedown playing out at U.S. utilities.
Get the tactics, techniques and procedures out the door fast enough, the thinking goes, and hackers won’t be able to use the same tricks twice. The sooner utilities can learn about specific vulnerabilities, the smaller the window of time during which they can be exploited.
The Cybersecurity Information Sharing Act, which Congress passed late last year, called on DHS to strengthen distribution of classified cyberthreats and bring information sharing up to "machine speed." This spring, the department launched its Automated Indicator Sharing (AIS) capability to cut out the need for phone calls like Sistrunk’s in the future. Only a few electric utilities are participating in the new venture at this point. The power industry’s main source of cyberthreat information from the government is the Energy Department.
Such machine-to-machine warning tools work best when fed with concrete data: Which URLs have been hijacked? What internet protocol addresses are the hackers known to be using?
A few of these clues emerged as the dust settled and the lights came back on in western Ukraine. On Jan. 11, DHS published digital signatures that could be used to search for the malicious BlackEnergy payload thought to have been used during the course of the attack.
"We cannot confirm a causal link between the power outage with the presence of the malware," the agency said, adding that it still "strongly encourages" companies to look for BlackEnergy.
But there was much more to the Ukraine attack than strings and conditions, zeroes and ones. This was not a threat that could be spooled through a computer, diagnosed instantly or quickly tamed. The threat was distinctly human, down to the bogus telephone calls designed to hamper the Ukrainian power utilities’ ability to respond.
Brookings’ Hennessey said DHS’s assessment likely evolved as the focus shifted from the immediate attack to its broader implications, though she, too, took issue with the delay.
"It’s the natural way that the U.S. government tends to respond to threats: first, in a very specific sense, because that’s where the most rapid response potentially is needed," said Hennessey, who formerly worked in the Office of General Counsel at NSA. "Then they move on to the larger questions: What about the electric grid in general? What about critical infrastructure in general? Are we prepared?
"I think DHS was shaken by the outcome of that inquiry," she said.
5.8 out of 10
The limitations of DHS’s capacity to navigate cyberthreats may have stemmed from diplomatic sensitivities and bureaucratic hurdles.
DHS was only able to visit Ukraine with a green light from Kiev, according to sources with knowledge of the inquiry.
"Given that this was politically sensitive, and who the likely perpetrator was, and that the U.S. was asked to come in, all of that tempered what could be said to the public," said one informed industry official.
Sources also said DHS analysts were hampered by the Ukraine government’s reluctance to publicly broadcast details of the attack. Ukraine’s worries are evident from its utilities’ about-face on the news blasts that went out in the immediate aftermath of the hack. Kyivoblenergo, one of the three electricity distribution companies that hackers hit hardest, circulated an announcement warning that "third parties" had made "illegal entry" into its control systems. The utility later deleted the Dec. 24 post.
Since then, Ukrainian authorities have been quick to pin the cyberattack on Russia while avoiding detailed discussions of its causes and implications. One cybersecurity expert involved in the investigation declined comment, saying, "We try not to raise this topic anymore." The attack is dead and gone, the thinking goes — any takeaways have long since been debated, adopted or cast aside.
But many of the technical lessons took more time to trickle down to U.S. utilities.
On May 31, more than five months after the Ukraine blackout, DHS posted a warning about a commonly used piece of hardware.
In its advisory, DHS described a security glitch in a 7400-series Moxa device designed to translate serial communications in industrial environments to the modern Ethernet protocol. Moxa devices are widely deployed across the United States and worldwide, including in electric substations.
The vulnerability in question was hardly a slam-dunk, according to DHS, which ranked its severity a 5.8 on a 10-point scale. "Crafting a working exploit for this vulnerability would be difficult," the agency’s Industrial Control Systems Cyber Emergency Response Team concluded, without mentioning Ukraine.
Yet on Dec. 23, remote hackers managed to disable dozens of the devices in the first-of-its-kind cyberattack on Ukraine’s power grid.
Once corrupted by malicious firmware updates, the devices were impossible to repair. Grid operators in Ukraine had to buy and install brand new serial-to-Ethernet converters across affected substations.
Moxa has since stopped producing the UC 7408-LX-Plus device with the critical flaw.
To SANS’s industrial cybersecurity expert Lee, DHS’s response to the Moxa problem defied explanation.
"We know for a fact that the adversary took advantage of a vulnerability to overwrite the firmware on a Moxa device during a nation-state cyberattack on the power grid," he said. "And how does DHS classify it? ‘It would take a really skilled attacker to do this, and we’re giving it a 5 out of 10 for vulnerability rating.’ What?"
Moscow, maybe
DHS’s website is littered with warnings about insecure industrial products coming from big manufacturers like Siemens and Schneider Electric down to smaller companies like Malaysia-based Ecava.
In written testimony before the Senate Armed Services Committee earlier this year, U.S. Director of National Intelligence James Clapper listed threats from "cyber and technology" on Page 1.
"Devices, designed and fielded with minimal security requirements and testing, and an ever-increasing complexity of networks could lead to widespread vulnerabilities in civilian infrastructures and US Government systems," he said.
Clapper didn’t hesitate to name names. "Russia is assuming a more assertive cyber posture based on its willingness to target critical infrastructure systems and conduct espionage operations even when detected and under increased public scrutiny," he said later in the annual report.
Despite the hint, no senior U.S. official has laid the blame for the Ukraine cyberattacks on Russia.
The Dec. 23 attack "serves as a wake-up call for all types of countries, especially countries like the U.S., where everything is connected," said Nadiya Kostyuk, a fellow at the EastWest Institute’s Global Cooperation in Cyberspace Initiative. "And I do hope that the countries discussing potential ways of cooperating move a little faster on these types of issues."
In its Jan. 27 report, DHS’s intelligence office came closest to pinning the attack on Russia, but experts don’t want hand-wringing over attributing cyberattacks to come at the expense of communicating the tactics behind them.
"Government needs to understand that asset owners need to know quickly the technology behind [an attack], not whodunit," said Marcus Sachs, senior vice president and chief security officer of NERC, at a grid security event last week in Washington, D.C., noting that the two priorities are sometimes "tugging at each other."
He said that events "like Ukraine help us get closer to that kind of understanding; I think we still have a long way to go."
The third story in EnergyWire’s Hack series examines how U.S. cybersecurity rules might have fared against a similar attack against the U.S. grid.
