ORLANDO, Fla. -- The small city water company in Arnold, Mo., went online last November. Seventeen hours later, a hacker using the SHODAN attack search engine had identified and penetrated an internal computer address leading to the control system that operated the pumps at the heart of Arnold's operation.
It was a virtual water company linked to the Internet, a fake "honey pot" created by Trend Micro cyberdefense consultant Kyle Wilhoit on his computer to mimic the control network of a real utility and uncover the intruders.
Wilhoit was a featured speaker at a conference here of the National Association of Regulatory Utility Commissioners, brought in to dramatize the cyberthreat to utility officials and regulators. If the same attack succeeded with a real water company, "it could and probably would cause catastrophic water pressure failures," he said in his briefing Tuesday. "The only difference was, it was happening in my basement."
The back door into the dummy water company system was quickly discovered and repeatedly exploited by amateur "hackavists" and state-sponsored cyber saboteurs, with China and Russia as leading origin points, he determined. An expanded version of the trap captured 74 infiltration attacks between March and June.
While the targeting activity is documented, the overall vulnerability of the U.S. utility sector is closely guarded. Industry and congressional surveys indicate widespread differences in defensive capabilities, particularly between large and small utilities.
Joseph McClelland, director of the Office of Energy Infrastructure Security at the Federal Energy Regulatory Commission, outlined the office's efforts to protect the most critical parts of the high-voltage power grid.
His office is modeling weak points in the electric grid, he told a conference panel. "We've identified key nodes, substations and generators that if we were attackers we would target." The results were reviewed with the industry to verify conclusions, he added.
Beginning with large utilities, FERC has gone over company safeguards against cyber and physical attacks and best practice defenses, he said. "Who is targeting their facilities, what methods and means they're using, what technologies, and what they can use to protect their systems," McClelland said.
Several grid officials said they didn't know of any electric power utilities whose supervisory control and data acquisition (SCADA) networks -- the operational nerve centers -- actually face the Internet and so would be exposed to attackers' probes to SCADA openings, as was Wilhoit's creation.
But officials have a list of concerns. Some gas utilities may not be as well-protected as grid centers. Smaller electric utilities don't have resources to match cyberdefense investments by the largest investor-owned power companies. Utilities have smart meter networks and large business and customer-linked operations that do run on the Internet, requiring secure firewalls to protect operations networks. And in several high-profile security breaches and attacks, malware has entered secure systems through contractors' laptops and USB jump drives, officials say.
Joe Rigby, chairman and CEO of Pepco Holdings Inc., said by the middle of next year, 1.4 million of Pepco's 2 million utility customers will have smart meters that are designed to exchange data with control centers.
"We have an enormous amount of customer information that could be very disastrous if it fell into the wrong hands," Rigby said at the conference.
Attempts to breach the company's networks are inevitable, he added. "We don't look at this as 'if.' It's a matter of when it happens," he said.
Rigby said cybersecurity is a board-level priority. The company works closely with federal security and law enforcement agencies, and Pepco has run defense training exercises, he said.
But, he added, "If in fact we have a problem, I think it's going to take us quite a bit of time to recover depending on what the issue is."
Defense contractors step in
Some of the largest utilities have joined with leading defense and information technology contractors to make their cyberdefenses more sophisticated and formidable.
NARUC officials have toured Lockheed Martin's NexGen Cyber Innovation and Technology Center, a 25,000-square-foot research and cyberdefense installation in Gaithersburg, Md., north of Washington, D.C., where technicians monitor and combat cyberattacks against the company defense business and its utility customers.
"As the grid becomes more automated and the information technology merges with the grid operations technology, then that inherently makes the grid more accessible electronically and more vulnerable unless it is protected," says Roger Flanagan, director of Lockheed Martin Energy Solutions.
Lockheed Martin coordinates a Threat and Information Sharing Working Group of about 20 large utilities that share cyberdefense best practices and training. The project was sponsored by American Electric Power Co. Inc. through an American Recovery and Reinvestment Act grant.
"If we uncover something, we have formalized mechanisms to share information with the government," Flanagan said. Threat information is quickly shared with the utilities, he said.
On one recent afternoon, Flanagan led the way past a secured door into a large room where a score of technicians were operating the company's cyberdefenses. At that moment, Google searches for links to a Lockheed Martin missile were displayed on a large monitor. A second cyberdefense team works in a nearby room.
"I think utilities are making great strides in this area. I think they are taking action, the ones that we are working with, at least. ... We are working with them on methods and best practices and tools for either triage or response," he said.
The industrywide response is far from uniform, he and other experts say. There is a gradation in how utilities are responding, he said.
"We are doing work with some smaller utilities that may not have the large operations and information technology staffs to either fully deploy or even afford some of the tools and techniques that are available," Flanagan said. "Those are usually on smaller distribution networks and probably on less critical infrastructure parts of the grid, but very important still."
Creating a 'cyber-reactive culture'
The Obama administration's cyberdefense plan for the power grid and other critical infrastructure installations, issued in draft and due to be completed in February, relies on voluntary, company-by-company adoption of best practice defenses, said Patrick Gallagher, director of the National Institute of Standards and Technology (NIST), speaking to the NARUC audience this week.
The administration concluded an industry-based defense had a better chance of succeeding against evolving cyberattacks than a federal plan built on static government standards, he said.
But it had also become clear that a federal regulatory approach could not get through Congress, he added.
"There was a debate happening in Congress about whether this implied new regulation, new regulatory roles and responsibilities, and you might imagine this was not a uniformly positive idea. It was a very contentious issue," Gallagher said.
The president's executive order initiating the voluntary industry plan "to a large extent has sidestepped that debate," Gallagher added.
Instead, NIST has led an industrywide development of best cyberdefense responses with more than 3,000 industry, government and academic institutions participating, he said. "It is the industry's framework. And that is really the key ingredient."
Rather than keeping score, it sets "an aspirational goal [and asks] organizations to adopt a continuous approval model that strives to get there," Gallagher said.
"The framework is not going to threat-proof anybody. We don't believe there's any such thing as threat-proofing. ... It is about creating a cyber-aware and cyber-reactive culture. It's like the immune system."
NSA anger trickles down
Robert Kolasky, senior adviser in the Department of Homeland Security's infrastructure office, said DHS is working to strengthen and speed up the process for reporting government-detected cyberthreats to grid companies and utilities. It will also seek to coordinate new incentives to encourage more rapid adoption of the new framework.
But he agreed it is not a government fix.
"Given where we are, with the budgetary situation, the economic condition, philosophy of government ... this is not going to be a government-mandated program, nor is it going to be a government-funded set of incentives that will very quickly give a lot of money to industry to promote adoption of the cybersecurity framework. The reality is, for a number of reasons we don't think that is good policy, and it doesn't work in this budgetary environment."
In response, Pepco's Rigby was plainly dissatisfied with the government's current threat reporting process, even as he praised the administration for helping improve the company's cyberdefenses.
"It could be a very, very difficult process ... where, with all good intent, you have multiple agencies working, [and] it might be difficult in that moment to collaborate, you could have a lot of confusion," he said.
Washington Utilities and Transportation Commissioner Philip Jones, the outgoing NARUC president, said flatly, "Information sharing is not working, and I don't know who to deal with in the federal level. ... I know we need legislation to deal with this, but I don't think we're going to get it."
The furor over National Security Agency surveillance programs has shelved congressional interest in expanding sharing of confidential security information with the utility sector, NARUC staff concludes.
DHS's Kolasky said government security agencies "continue to shift from a need-to-know to a need-to-share culture" and are working on a new executive order on information sharing, he said. "A lot of it [involves] cultural changes within the government, and that takes longer.
"It's not just a technical problem. It's a cultural problem," he said.