Last month, hackers tied computers into knots at a small Colorado water utility.
It wasn’t the first time the Fort Collins-Loveland Water District and its wastewater counterpart had been hit by "ransomware," a type of malware that encrypts victims’ computer files and demands online payment to unlock them.
While operations weren’t harmed, the infection prompted the water district to switch out its information technology service provider and call in the FBI. The case, first reported by the Coloradoan, remains under active investigation. FCLWD and the South Fort Collins Sanitation District treat and distribute water to 45,000 customers in northern Colorado.
Colorado water officials aren’t alone in their cybersecurity woes. The nation’s nearly 70,000 water and wastewater utilities are struggling to keep their heads above a rising tide of online threats, based on interviews with security experts and water company operators.
As one IT manager at a midsize water utility put it, "It’s not a question of if, it’s a question of when" hackers disrupt vital U.S. water systems. "Most small and midsize utilities are overstressed," said the manager, who requested anonymity.
Some larger utilities are well-positioned to thwart an attack by hackers backed by a foreign government, said Michael Arceneaux, managing director for the Water Information Sharing and Analysis Center, the industry’s clearinghouse for getting the word out about the latest hacking threats and vulnerabilities.
But in a sector that encompasses tens of thousands of local water systems, securing America’s vast and disparate drinking water supply remains a significant challenge.
"Drinking water utilities run the gamut in terms of cybersecurity preparedness," Arceneaux said. "What we try to do to compensate for that is make sure people are aware of the threats, so they have some motivation to invest the resources that should be invested."
He said the ISAC and its membership recently reached the level of maturity needed to start partnering with other sharing and analysis centers, including the multistate government ISAC and the electric power sector’s E-ISAC.
Water utilities and power distributors share similar industrial control systems, rely on many of the same equipment providers and can encounter similar cyberthreats.
While the water system is inherently not as interconnected as the U.S. electricity system, "it’s very plausible that the water sector is less prepared than the power sector for dealing with cybersecurity threats," Arceneaux said. "We are so fractured, so the water sector as a whole is at a little bit of a disadvantage."
What keeps you up at night?
The decentralized nature of the U.S. water industry has left policymakers with a dilemma. Cybersecurity for water treatment and supply networks is only loosely monitored at the federal level and is often ignored by state utility commissions that may have limited cybersecurity expertise and tend to focus on water quality.
"Water cybersecurity is not on everyone’s — or certainly not every commissioner’s — radar screen, although I’ve tried to make it that way," said Mary-Anna Holden, a commissioner on the New Jersey Board of Public Utilities.
In many emergency planning exercises, it isn’t the lack of electricity that triggers chaos and widespread casualties. It’s the lack of clean water that forces people from their homes.
"Nobody thinks about wastewater systems until they break," said Holden, who chairs the Committee on Water at the National Association of Regulatory Utility Commissioners.
New Jersey is one of the few states to have taken any regulatory action on the issue of water security. It’s required utilities to report cyber events to state environmental officials and directed regulated utilities to include cybersecurity in risk management plans.
"If someone’s hacked into the operational network and can control chlorination, do something to the [wastewater] digesters or can get control of the wastewater plant, that’s the thing that keeps me up at night," Holden said. "You could cause cholera or dysentery downstream, which could be a major city. How do you counteract that?"
In New York, staff members from the Department of Public Service conduct "frequent and regular audits of company defenses" against emerging cyberthreats, according to a spokesman. The agency conducts annual reviews of water companies’ cybersecurity plans and is weighing data security requirements for certain firms that receive sensitive personal and billing information from water, electric and gas utilities.
‘Scotch tape and bubble gum’
In the summer of 2013, a small New York dam saw its control systems hit by a suspected Iranian hacker. The Justice Department accused Hamid Firoozi of gaining access to the dam’s supervisory control and data acquisition (SCADA) system, costing city managers in Rye, N.Y., roughly $30,000 to remediate the threat.
While a few flooded basements might have been the worst-case scenario from the dam breach in New York, the case still drew scrutiny to SCADA system security and the water sector writ large.
In 2016, the same year DOJ unveiled its charges against Firoozi, the security branch of telecom giant Verizon Communications Inc. reported responding to another bona fide hack of a water utility’s control systems.
In the report detailing Verizon’s data breach investigations, its analysts used a pseudonym called "Kemuri Water Co." to stand in for multiple data breaches at water utilities of varying severity.
Verizon reported that the hackers modified settings without knowing exactly what they were doing to the control system. The unnamed water company was able to quickly identify and block what they were doing.
Other industrial cybersecurity experts said that the baseline scenario outlined by Verizon was plausible: a hacker gaining a foothold in a water utility’s business networks; crossing over a firewall or "DMZ" to land in the industrial control network; and from there fiddling with valves, flows and physical processes.
Dave Weinstein, vice president of threat research at Claroty Inc., said water and wastewater are among his top concerns. "It gets the least attention, and they’re probably the least mature sector from a cybersecurity standpoint," he said.
Many tiny water utilities simply don’t have the bandwidth to square off against hacking teams that may be backed by foreign militaries, he explained.
"Small mom-and-pop operators, I’ve visited a number of those sites. A lot of them are being held together by Scotch tape and bubble gum," Weinstein said. "It’s pretty precarious."
The number of exploitable cybersecurity weaknesses found in water-sector equipment also appears to be growing, based on statistics from the U.S. Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) and analyses by Kaspersky Lab’s ICS-CERT.
Sixty-three cyber vulnerabilities were uncovered in the "water supply" sector in 2018, according to federal data, accounting for 15 percent of all industrial security problems. Only the energy and manufacturing sectors had more vulnerabilities in 2018.
Risk and resilience
Federal lawmakers have started to take note.
Last fall, Congress passed the America’s Water Infrastructure Act of 2018, sponsored by Sen. Amy Klobuchar (D-Minn.) and signed into law by President Trump on Oct. 23.
Any water utility serving 3,300 or more people is now expected to carry out a "risk and resilience" assessment of its networks, including a review of cyber defenses. The nation’s biggest water providers have until next March to comply, while smaller companies can wait to act until June 2021.
EPA is now the go-to agency for water cybersecurity. It’s tasked with issuing guidance to utilities on implementation of the new law and following up on compliance. "EPA is very aware that cyberattacks are a significant threat to critical infrastructure sectors, including water and wastewater systems," an EPA spokesperson said in an email. "EPA works through a voluntary partnership approach to help water utilities enhance the cybersecurity of their water systems."
The voluntary approach signed off by Congress means utilities are not required to hand over data to the federal government that could help it assess how vulnerable the nation’s water systems are. That contrasts to electric utilities, which are required to report major cybersecurity incidents and gaps through both the Department of Energy and Federal Energy Regulatory Commission.
Robert Powelson, a former FERC commissioner who’s now CEO of the National Association of Water Companies, said the federal push to ramp up cybersecurity guidance and regulation includes water. "Looking at the posture with DHS and the Department of Energy’s new cyber office, everyone’s like, ‘We’ve got to bring water into this conversation.’ I think it’s healthy; I think it’s a great opportunity."
Industry groups like the National Rural Water Association and American Water Works Association have released some of their own assessment tools and cybersecurity resources, warning members to ignore cybersecurity at their peril.
‘Perfect target’
News of a few water-sector cyber intrusions has trickled out publicly, including an attack on a North Carolina water utility in the aftermath of Hurricane Florence last year.
Jeffrey Hudson, CEO of the Onslow Water and Sewer Authority in southeastern North Carolina, announced on Oct. 15, 2018, that "a sophisticated ransomware attack" had effectively wiped out many of the small utility’s computers. He emphasized that the safety of the water supply and the environment was never jeopardized.
"ONWASA will undertake the painstaking process of rebuilding its databases and computer systems from the ground up," Hudson said, rather than pay off the digital hostage-takers.
Cybersecurity experts project that targeted ransomware attacks are set to rise. In a recent threat outlook, analysts at Booz Allen Hamilton predicted "a plausible uptick in state-sponsored attacks and intrusions at water utilities," citing a March 2018 alert from DHS that claimed Russian hackers had already targeted U.S. water networks.
Booz Allen described the water sector as a "perfect target" for hackers.
So far, the U.S. water utilities hit hardest by hackers have been "collateral damage," explained Booz Allen chief technologist Kyle Miller, falling prey to common threats like ransomware.
"Most water utilities have less robust, less mature network security than a lot of the other [infrastructure] verticals," Miller said. "A lot of that comes down to size and funding. It’s hard to compare a county water system to a Fortune 100 oil and gas company."
Miller said he’s most concerned about targeted threats, as water companies follow global trends in industrial automation and digital connectivity.
Even if supply interruptions or chemical releases don’t become a full-blown crisis, a hack that causes people to lose faith in the quality of their water is "certainly within the realm of possibility" for nation-state hackers, he said.
The stakes are high. "Nearly every facet of life relies on clean and reliable water to function," he said.
