Eastern Europe was blanketed in a heat wave last summer. In Kiev, Ukraine, a state of desperate resignation had set in as fighting intensified between pro-Russia rebels and Ukrainian forces to the east. Separatists closed highways and attacked ports. Meanwhile, a silent incursion had started to worm its way into the email accounts of employees at media outlets, national railroads and power distributors in the western half of the country.
The digital-era Trojan horse looked like a call to arms from the nation’s embattled capital. The subject line read simply, "Mobilization."
As Ukraine’s civil war raged, a few mouse clicks at three local power companies set in motion the covert intrusion. It was the first successful attempt at planting a bug, then disabling an electric grid serving hundreds of thousands of people.
At 3:30 p.m. on Dec. 23, 2015, lights winked out in parts of the Ivano-Frankivsk regional capital. A minute later, another part of the grid went down. Soon, a third utility — and almost one-quarter of a million households and businesses had lost electricity.
Workers at the Prykarpattyaoblenergo, Kyivoblenergo and Chernivtsioblenergo utilities watched helplessly as cursors moved across their workstation screens at the intruders’ commands, shutting down substations. Other hidden commands destroyed vital equipment. The attackers were invisible and precise, and they showed the world how fragile critical infrastructure is when hacking is used as a weapon of war.
Ukraine’s battle to wrest control from the hackers elevated the story of frequent blackouts in a poor country to the latest in a series of cyberattacks with implications for the United States. Months in the making, it represented an escalation in attack methods that frightened U.S. authorities and executives. The hack methodically corrupted standard programing and subverted controls. It laid bare the work of persistent planners.
Seven months after the Ukraine attack, U.S. security officials are still trying to understand whether the much larger, and more sophisticated, North American power grid is equally as vulnerable to a determined, insidious assault. A more ominous warning has been sounded to utilities and federal agencies: Step up preparations to recover from a cyberattack that may one day break through.
Hackers didn’t simply crack a code and pull the off-switches at local substations — they rendered some crucial station devices inoperable. Then, they corrupted software and servers designed to turn the power back on.
The unparalleled grid strike in Eastern Europe has led to stronger, more frustrated complaints by industry and security experts about the performance of the U.S. Department of Homeland Security as a source of rapid, actionable cyberthreat intelligence for the electricity sector. It also has raised concerns that federal guidelines applicable to the high-voltage interstate grid don’t guarantee the security of local utilities that distribute power to millions of homes and businesses.
A four-part investigation by EnergyWire found that relationships between DHS and outside experts with deep knowledge about grid security became badly frayed in the weeks after the December hack. For several months, DHS put out conflicting internal and public messages about the dangers posed by the Ukraine hack, compounded by tug of wars around the use of closely held information inside the diffuse intelligence community.
What resulted was a slow and halting response by the U.S. government in the aftermath of the Ukraine takedown.
In an age that pits state-sponsored hackers against private companies like power utilities, critics of congressional inaction and government secrecy are starting to hammer at what they view as glaring failures around threat-sharing.
Ukraine is one of a cluster of cyberattacks in the past two years that grabbed headlines. The November 2014 attack on Sony Pictures Entertainment mushroomed into a national security and free-speech entanglement with North Korea. The U.S. Office of Personnel Management (OPM) disclosed last summer that computer breaches included the theft of Social Security numbers of 21 million Americans. Hackers also stole fingerprints of government workers and compromised security clearances.
The Obama administration in September 2015 publicly acknowledged suspicions that China was the source of the OPM breach.
In the Ukraine case, top administration officials have kept quiet, refusing to give credence to experts’ widely held view that Russian hackers likely planned and executed the first-known takedown of a power grid.
Reconnaissance
The email messages snuck through the Ukrainian utility servers in droves last summer, asking employees there to "enable content" to read the attached document. When they clicked, the file unloaded the BlackEnergy attack software on their office computer systems, burrowing deep into the information technology side of the business.
Once out of sight, BlackEnergy established hidden lines of communication the hackers could use to extract information and download malware, all under the noses of the grid operators. There was plenty of time to inventory systems and search for passwords and pathways that would take the intruders from business-side computers into the protected heart of the utilities’ operations, including the grid operators’ control room.
Over many months and perhaps after additional runs of "spear phishing" emails, the hackers were able to find utility officials with credentialed access to the operating systems and steal their passwords, too. With login info, the hackers could escalate from espionage to attack. All they needed was a trigger.
Incitement
On the eve of the assault, utility employees in Ukraine, Russia and a few former Soviet states were slapping each other on the back for a job well done.
It was Energy Day, a holiday to recognize and celebrate electricity workers in parts of Eastern Europe. In a Dec. 22, 2015, speech, Russian President Vladimir Putin congratulated his country’s power industry professionals. He singled out their "great, strenuous and highly demanding" work keeping the lights on in Crimea.
It was not an offhand comment. Putin’s forces annexed the predominantly Russian-speaking peninsula in 2014, setting off a violent and ongoing territorial conflict that leaked into eastern Ukraine. Crimea, despite its de facto status as a Russian republic, remains connected to Ukraine’s electric transmission network. Power outages in the contested territory have been attributed to Ukrainian saboteurs.
Observers have suggested those blackouts set the stage for Russian vengeance, a theory that would explain the timing and rationale for the Dec. 23 cyberattacks, and why the biggest impacts were confined to pro-Western portions of the country.
The message in that scenario: You hit our grid in Crimea, we hit yours.
Attack
The takedown itself was quick and clinical, backed as it was by months of planning.
Western Ukrainian grid operators could only watch as hackers booted them from workstations, dragging cursors around control system screens to achieve their own harmful ends. The attackers changed passwords so the Ukrainians couldn’t log back in to grab the reins.
Utility industry workers were sidelined during those first frantic minutes. In hacker parlance, they had been owned.
No one online could be trusted. The computers, however, were inherently trusting. They dutifully carried out the hackers’ commands to open high-voltage circuit breakers at dozens of substations across western Ukraine, knocking out power. The machines had never been programmed to question why so many users would simultaneously log in from unusual internet protocol addresses. The virtual network tied to the operational workstations asked for a username and password — nothing more — to grant unfettered access.
With credentials in hand, the attackers still had to understand Ukraine’s Soviet-era electricity infrastructure to do any real damage. After all, they were dealing with three different power distribution management systems at three different companies.
At 6 p.m. on Dec. 23, the situation moved from bad to worse for one operator when the hackers cut off the backup power source to a critical control center, preventing the system from rebooting. Once merely spectators, utility workers were now blind to what was happening in the far-flung power distribution networks that had once been under their control.
Impact
The blackout itself lasted less than six hours in most places. It was hardly calamitous for Ivano-Frankivsk, which is "no Manhattan," as one Ukrainian source put it. The two other, mostly rural areas affected by the outages were similarly accustomed to power disruptions.
Many Ukrainians put up with electricity rationing for the better part of a decade following independence from the Soviet Union in 1990. More recently, war in the East has made reliable energy something of a luxury there. While western districts are more secure, critical services — hospitals, key government buildings and the like — still keep backup generators as standard practice.
December normally brings brutal cold for much of Ukraine, with the capital, Kiev ,averaging just 28 degrees Fahrenheit throughout the month. But highs on the Wednesday of the cyberattack stretched into the 50s across swaths of the country, meaning the outages were less likely to endanger human lives.
Still, until that day, hackers had never carried out such a surgical strike on civilian infrastructure. And even if Ukrainians pride themselves on their resilience, no one, from Ivano-Frankivsk to Manhattan, likes living without electricity for long.
Interference
Prykarpattyaoblenergo offered the first sign the blackout was more than just a routine interruption.
The day before much of the West would celebrate Christmas Eve, the Ukrainian utility warned its customers not to call in to report power outages, so its workers could get a grip on the unfolding crisis.
The Ivano-Frankivsk-based company’s phone lines had been swamped by a barrage of calls that investigators later traced to Russia.
This "telephone denial of service" would be bad enough during a normal blackout. But Prykarpattyaoblenergo’s technical director added that "outsiders" had tampered with the utility’s control systems, forcing the company to revert to manual operations across its territory.
The hackers didn’t want to switch off power only to see their work undone minutes later. They were actively sabotaging recovery efforts, leaving no trace of doubt as to their intent.
Coercive updates went out to boxes used to translate data from grid equipment, a move that effectively turned the converters into bricks. Analysts later found that the attackers must have tested the malicious code beforehand, so they knew it would cause delicate field devices to fail.
Before calling off their assault, the online attackers left behind another nasty gift: "KillDisk" malware to wipe victims’ computers and render them useless. It was the digital equivalent of kicking the power companies while they were down.
Aftershocks
At least three other energy companies in Ukraine were targeted in the campaign, but were able to stop the hackers from causing physical damage. Meanwhile, the hardest-hit power companies spent months in "manual mode," unable to trust their networks enough to go back to normal operations.
The attackers weren’t done yet. In January this year, more emails made the rounds at energy companies in Ukraine, but they ditched the BlackEnergy malware in favor of a different, less sophisticated Trojan horse.
This time, the emails claimed to come from Ukraine’s main electricity market overseer, the National Power Company "Ukrenergo."
"When I read through the email that [hackers] prepared for the Oblenergo companies, originating from the regulator, it was perfect," said Vlad Styran, co-founder and manager of operations at the Kiev-based cybersecurity firm Berezha Security. "There was either a lot of work behind that, or they had access to an insider" who knew the writing style and types of files normally shared by the Ukrenergo, he added.
The targeted utilities were on guard following the attacks, and no follow-up physical disruptions are known to have happened in Ukraine beyond Dec. 23.
Styran, whose day job challenges him to think about ways to break into companies as a "white hat" hacker, has suggested the Dec. 23 cyberattacks were a warning shot rather than an attempt to disable Ukraine’s infrastructure indefinitely.
Among the dozens of security experts contacted by EnergyWire, the vast majority agreed that the level of sophistication, organization and time that went into the attack points to a state-sponsored hacking group with control system expertise.
Russia remains the most likely culprit, even though two of the three utilities harmed are majority-owned by Russian businessmen, including one reported ally of Putin. The Security Service of Ukraine wasted no time in blaming the Kremlin for the attacks. Russian authorities have yet to comment on the case.
"It was a demonstration of power, maybe just field testing of the tools and the tactics," Styran said. "That’s how we can treat it. If the goal was to make harm, the targets would be completely different."
From theory to practice
The attack was a call to arms in the inchoate language of cyber warfare.
To Robert Lipovsky, who was among the first cybersecurity analysts to examine the Ukraine case, the events of Dec. 23 showed "that things such as this aren’t just theoretically possible," he said, "that things like this can happen."
"It shouldn’t have been so easy for the attackers," said Lipovsky, a Slovakia-based senior malware researcher at cybersecurity firm ESET.
He cited the attackers’ ability to hop from utility computers on the business side over to operational workstations that directly communicated with control systems. Such a threadbare "air gap" between operational and informational networks is not unusual in other parts of the world, he noted.
"There are definitely loopholes, generally. It’s not just limited to Ukraine," he said. "This shouldn’t be taken lightly."
Part 2 in EnergyWire‘s "The Hack" series explores the Department of Homeland Security’s response to the Ukraine attack.
