An insidious cyberattack on a small Florida water utility that came to light Monday has spotlighted vulnerabilities in a sprawling sector that is overseeing itself as threats continue to grow.
Investigators reported no leads yesterday on who broke into the control system of the water system of Oldsmar, a city on the north side of Tampa Bay, on Friday. But authorities said the hacker took over a control room workstation in an attempt to pump dangerous levels of treatment chemicals into the water supply of 15,000 people.
The break-in was both remarkable and totally predicable, some officials and experts said yesterday.
It was remarkable because no such attack on the nation's tap water supplies had been reported before among the more than 25,000 community water supply systems. Water systems are constantly defending against probes from hackers, and there are more breaches than are publicly reported, some experts say.
It was predictable because the cyberassaults are so incessant, and absolute security is so hard to achieve.
Rare though the incident is, it has opened questions about the vulnerability of water systems and the oversight of their defenses.
Could the operations of a water utility be manipulated by a hacker? "The reality is, the answer to that question is yes," said Kevin Morley, manager of federal relations for the American Water Works Association, a trade association for about 4,300 water utilities.
"It is reasonable to assume a 100% probability of being attacked," he said — whether from relatively simple "ransomware," which locks up victims' computer files and demands payment to unseal them, or from a "very targeted" attack.
"I think there's a false perception that there's an opportunity for complete risk elimination. It's just not possible. This is about risk management," Morley said.
In Oldsmar's case, the hacker tried to hike levels of poisonous sodium hydroxide, or lye, to over 100 times their normal concentration, according to local law enforcement. An operator at the water treatment plant was able to quickly switch levels of the chemical back to normal before any of it leached into the water supply (Energywire, Feb. 9).
Daniel Groves, cybersecurity business sector leader for West Yost Associates, a Phoenix-based engineering, planning and operations services firm for the water sector, said his company conducts a lot of cybersecurity assessments for water systems.
"There are very few where we don't find evidence that someone has been tampering with it," he said.
Still, he said the Florida case is unusual because the Oldsmar utility "came out and said what happened."
"I was happy to see that," added Groves, whose firm prepared a new cybersecurity risk management guide for AWWA.
However, Michael Arceneaux, chief operating officer of the Association of Metropolitan Water Agencies, said he doesn't believe what happened in Florida is a widespread issue. As managing director of WaterISAC, the sector's threat sharing organization, Arceneaux has called on utilities to create strict barriers between the internet-facing information systems and control systems.
"I think the water sector, as big and diverse as it is, has done well over the years and I think people can feel confident that their water is safe, not only because of the measures that utilities have taken but also because of the redundancies," said Arceneaux. "Had that chemical actually entered that system, alarms would have gone off and changes would have been made quickly."
There was no formal cybersecurity oversight of water utilities until congressional passage of the America's Water Infrastructure Act of 2018, which directed community water systems serving populations of 3,300 or more to assess hacking risks to their systems, certify completion of the reviews to EPA and prepare response plans. Compliance with the law is staged, with the largest utilities going first. Oldsmar is in the group serving the smallest number of customers, whose review certifications are not due until June 30.
Initial reports of the Oldsmar incident exposed an apparent glaring gap in defense.
In a press conference Monday, Pinellas County Sheriff Bob Gualtieri said an intruder gained access to the control system through a software program called TeamViewer, which gives technicians remote access to the system for monitoring and repairs. Although the tool had been replaced six months ago, it remained active, Gualtieri told The Wall Street Journal, providing an unguarded back door for an attack.
The sheriff added that safeguards were in place that would have detected and blocked the higher concentrations before they could taint the water supply.
Arceneaux said that although he has no direct knowledge of the incident, what appears to have happened would be a violation of basic cybersecurity hygiene, which requires strict separation between internet-facing business systems and operating systems.
A new approach to security?
Morley at AWWA said utilities "have been pretty good" at separating control systems from business networks.
"I'm not going to pretend that separation is 100%, because there is data on the process control side that is needed for business purposes," he added. "And so there is some potential through traffic there. I'm not going to pretend like system architecture is perfect everywhere since we know that's not true in any sector."
Under the risk reporting system created by Congress, an incident like the Oldsmar breach would not have had to be disclosed publicly. EPA's public reporting is confined to listing the utilities that have completed a self-assessment, with nothing about the results.
"AWWA requires a utility to certify that they have completed the risk assessment of their critical assets including cyber," Morley said.
"When a utility goes through that process, they are basically certifying that they have done their due diligence and taking ownership of the risk" — a goal that AWWA is pressing, Morley said. "The details of that risk are obviously security-sensitive and not something that you would want to be in the public domain."
When asked whether there's sufficient cyber oversight of the sector, Arceneaux responded, "I think that's a discussion that could be had."
Groves of West Yost Associates said the Florida breach is sending a message about the need to protect the most critical operating systems for water utilities — a cause that becomes a priority for cyberdefense experts at the Energy Department's Idaho National Laboratory.
INL's Andrew Bochman and Sarah Freeman have described the strategy as a forensic analysis by utilities of their most important operations.
"It zeros in on the comparative handful of most vital processes and functions on which the success of the company or military organization hinges entirely," Bochman and Freeman wrote in their new book, "Countering Cyber Sabotage." "The question is posed: 'What would kill your company?'"
Its basic assumption is bleak: "That well-resourced, adaptive adversaries have already taken up residence, performed extensive reconnaissance, remain undetected, and are preparing their cyber-physical attack."
The INL cyber analysts warn that automation of control systems is removing control room personnel in favor of increasingly complex computer systems, expanding the danger that the operators could not fend off a serious attack. The most essential systems must be shielded, or backed up with manual controls, or otherwise given fail safe protection, they urged.
A retired manager of an East Coast water system, speaking on the condition of anonymity for security reasons, said that the utility had become aware of a cyber vulnerability that could have allowed a sophisticated attacker to seriously damage operating equipment. "We pursued that, thinking about critical assets and what you can do protect them from damage," he said.
Part of the strategy included trying to operate the water system without the computer-based management systems known as supervisory control and data acquisition, the former manager added.
"That's really important. That was something we were able to practice," he said. "We never took the whole control system down, but we got to practice operating different elements of the water system manually."
Anticipating and dealing with worst-case scenarios is vital, he said. "We can do a lot to defend our water systems' controls, but if somebody wants to break in, and they have the skills, they'll break in."
Reporter Christian Vasquez contributed.