Part two of a two-part series. Click here for part one.
While cyberattackers and defenders duel for advantage in ever-more-complex digital battlegrounds, a set of basic vulnerabilities affecting power grids, factories and pipelines has gone largely unaddressed.
A vast number of sensors and other remote controllers that send vital data to control rooms -- and receive instructions in return -- weren't built with cybersecurity in mind.
That leaves their relatively unprotected communications prime targets for attackers, cybersecurity officials and consultants say.
Many of these devices were installed before "cybersecurity" even existed as a term. Others are brand new but can still be "insecure by design," according to security consultant Dale Peterson, or "designed to be insecure," as cyber technician Eric Byres puts it.
These core communications links are not included in the cyberdefense operations of the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), the special unit of the Department of Homeland Security formed to coordinate protection of control systems in energy and manufacturing sectors, said Marty Edwards, the ICS-CERT director.
"What Dale [Peterson] and others are referring to [by] 'insecure by design' is the entire control ecosystem [of] the clear text protocols that go back and forth between all these devices. I can't fix that through the vulnerability coordination process," Edwards said.
ICS-CERT routinely alerts industrial control system operators to flaws in their software (EnergyWire, Oct. 15).
But "the vulnerabilities are secondary to the fact that the whole damn protocol is broken," said Byres, chief technology officer of cybersecurity defense contractor Tofino, referring to the protocols or rules for sending data over the networks.
Streams of data from distant devices drive wall-sized system maps and terminal displays on control-room SCADA (supervisory control and data acquisition) computers that allow operators to monitor and adjust to shifts in pressure along pipeline networks hundreds of miles long, or instantaneous changes in power transmission.
Byres explained that decades ago, grid operators in particular were less concerned with security than with reliability. "The problem wasn't keeping people out; it was making it work," he said.
"The whole architecture was, make this simple to connect -- it was a trusted network, a happy little family," he added. "Well, things changed."
An unpatchable problem
ICS-CERT is a clearinghouse for finding and fixing software flaws in industrial control systems. Utilities, equipment vendors, cyberdefense firms and independent investigators that uncover holes in software report them to ICS-CERT, whose experts assess the vulnerabilities. Private warnings go out to affected companies as vendors devise patches for the flaws. Public alerts typically follow.
"It works extremely well for routine software errors," Edwards said, "the things that can be fixed with a patch."
Generic vulnerabilities in the "insure by design" networks aren't patchable, he added.
"We certainly look for ways to promote the higher security within that ecosystem. It's more an incentive to push the market toward adopting more secure products, more secure protocols and adopting a security culture," Edwards said. "We do that through outreach and awareness and dialogue. It's not something that alerts and advisories can fix."
That limit to ICS-CERT's scope has left security researchers such as Peterson unsatisfied with the way the government handles industrial control system cybersecurity. Peterson, founder and chief executive of the Digital Bond cybersecurity consulting firm, is among the toughest critics of DHS for not moving more aggressively to deal with critical infrastructure vulnerabilities.
"I am still waiting for DHS to state that insecure-by-design protocols and devices in the critical infrastructure need to be upgraded or replaced," he said.
When important devices are fundamentally unsound, ICS-CERT's advisories can seem like trying to prop up a wall while its foundation sinks.
"We have to take a more holistic approach and start to work on those insecure-by-design areas, because that perimeter defense model can only hold up for so long," Edwards said.
The reason that vulnerable critical infrastructure networks haven't suffered a major breach may be due more to attackers' forbearance than to stout defensive walls, he added.
"I believe even the bad guys understand the potential impact of taking a control system offline. So I think there is a little bit of, call it hesitation," he said.
Billy Rios, founder of the cybersecurity startup Laconicly, is broadly supportive of ICS-CERT for its vulnerability reporting system but said assuming attackers will steer clear of damaging critical networks is a "ludicrous" idea.
"Our security defenses should be built on engineering, so it is actually difficult to attack these devices," said Rios, who is an expert in embedded devices. "Relying on the attackers' goodwill is not a good strategy."
Defendable by design?
The way utilities' control networks are set up does give them some innate defense, other experts say. The systems were typically designed to fit each infrastructure operator's unique requirements, creating obstacles for would-be attackers who must unravel the peculiarities of each system.
Mike Kuberski calls this feature "security by obscurity." He manages grid protection and automation for Pepco Holdings, which provides electricity service in the nation's capital, its Maryland suburbs, and parts of Delaware and New Jersey.
"You build things on proprietary systems," Kuberski said, making them "much more of a challenge" for hackers to penetrate.
Robert Lee, who researches cyberspace conflict and critical infrastructure cybersecurity at King's College London, pointed out that defending industrial control systems should -- from a purely technical standpoint -- be much easier than attacking them.
"You can have two substations from the exact same power company, and they are going to be configured differently. And that confuses the hell out of hackers," he said. "They have to go learn about it before they can start attacking it."
Energy infrastructure operators, from utilities to pipeline companies, should first focus on mapping out their critical devices so they know what hackers may be after.
"If defenders actually understood what was on their network and understood what their network is, then they already have an upper hand on the attackers," Lee said.
That's not to say fundamental flaws such as insecure-by-design protocols aren't a problem. But "just because a vulnerability exists in a system doesn't mean you're going to be able to exploit it," said Nadya Bartol, senior cybersecurity strategist for the Utilities Telecom Council.
Although insecure-by-design vulnerabilities may not have been exploited in a big way so far, that doesn't mean they can be ignored, experts say.
But the lines of authority for further action are not clear. Mandatory federal cybersecurity rules don't extend to common communications between remote devices and control centers that travel on commercial "serial" telecom pathways (rather than the Internet), consultants note.
The federal rules, drafted by the North American Electric Reliability Corp. and approved by the Federal Energy Regulatory Commission, don't apply to state-regulated distribution utilities.
Perry Pederson, principal with the Langner Group cyber consultancy, said he's "not a witch doctor" and can't predict whether design flaws will be exploited in a large-scale attack.
"Is somebody planning and plotting? Could they, if they had the motivation and the money? Yes," he said. "I'd rather put my finger on potential problems and mitigate them rather than just hope that no one is interested, that nothing happens, that nobody figures this out."
Pederson said deciding whether an attacker will get through to cause a blackout or some other destruction "is the wrong question."
"The question is, can you live with the consequences? If not, deal with it," he said.
Byres of Tofino has argued that the cost of locking down SCADA networks by rapidly replacing vulnerable devices would be prohibitive.
Instead, the industry should step up actions to reduce the "attack surface" by narrowing the communications that the networks carry to the truly essential data, and by making sure control systems at the end of the line are bulletproof, he said. Operators using such networks need to abide by airtight authentication rules, he added, while communication procedures need to be continuously strengthened.
"I think we're going to need compensating controls for the next 20 years," he said, until network vendors have replaced the old units with secure new ones.
Peterson, who debated Byres at a SCADA security conference this year, was glum about the threat. "Nothing changes until something bad happens," he said. "I don't want to live in that world."
Want to read more stories like this?
E&E is the leading source for comprehensive, daily coverage of environmental and energy politics and policy.
Click here to start a free trial to E&E -- the best way to track policy and markets.