‘What nightmares are made of.’ Hackers hit Fla. water supply

A hacker tried to poison a Florida city’s water supply by spiking levels of a dangerous chemical, law enforcement authorities said yesterday.

The unsuccessful cyberattack Friday on a water treatment plant in Pinellas County — the first documented attempt to hack into and contaminate a U.S. community’s water supply — raises questions around critical infrastructure security as water and energy utilities move to digitize their operations.

A hacker gained access to a water treatment facility serving around 15,000 people in the city of Oldsmar, changing the levels of sodium hydroxide pumped into the water supply to a "significant and potentially dangerous" level, Pinellas County Sheriff Bob Gualtieri said at a press conference yesterday.

Cybersecurity experts say that although digitizing the so-called operational technology (OT) that keeps water flowing and electricity running can boost productivity, it also increases ways that hackers can infiltrate a system.

"The attack against the city of Oldsmar’s water treatment system is what [OT] nightmares are made of," said Marty Edwards, vice president of OT security at cybersecurity firm Tenable Inc. "If successful, the damages of the attack would have been catastrophic."

Early Friday, an operator at the Oldsmar water treatment facility outside Tampa noticed that someone briefly logged in to the system, authorities said. But as supervisors regularly access the computer network, the worker did not raise the alarm.

Later that day, an operator watched as the hacker regained access to the facility and in less than five minutes raised the levels of sodium hydroxide to dangerous levels, from 100 parts per million to 11,100 parts per million, Gualtieri said. The chemical never leached into the water supply, and the operator was able to quickly switch the levels back to safe amounts, officials said.

Sodium hydroxide, also known as lye or caustic soda, is a very corrosive, odorless, white crystalline solid used in water treatment facilities to raise the pH of water to minimize corrosion. It’s also a common ingredient in commercial drain and oven cleaners, and it is used in everything from making soaps and explosives to petroleum products.

Sodium hydroxide can cause irritation to the eyes, skin and mucous membrane, according to the Centers for Disease Control and Prevention, but the level of harm depends on dose and duration.

Swallowing sodium hydroxide may cause severe burns in the mouth, throat and stomach; vomiting; diarrhea; a drop in blood pressure; severe scarring of tissue; or death.

But even if the hacker had successfully locked in the higher sodium hydroxide levels, Gualtieri said, there are safeguards in place that would have detected and blocked the higher concentrations before they could taint the water supply.

"We’re very fortunate that this was not a bad situation. It could have been, and we want to make sure that it doesn’t happen," Gualtieri said.

Industrial cybersecurity firm Dragos Inc. said that many facilities have controls that can prevent unsafe conditions from occurring should a hacker breach critical infrastructure systems.

"That being said, remote access to industrial control systems (ICS) is common and increasingly so due to the need for people to work remotely," Dragos said. "This incident underscores how important it is for asset owners and operators to assess and secure their remote connections, especially internet connected remote access, and to ensure their incident response plans are current."

Dragos CEO Robert M. Lee called on critical infrastructure operators to harden their networks and make potentially deadly attacks harder to pull off.

The hacker accessed the facility by using the software TeamViewer, which was password protected, Gualtieri told E&E News. TeamViewer is a widely used program for remotely accessing computer systems.

The investigation is still in the early stages, and there is little information on who the hacker was or their motives, Gualtieri said.

"We don’t know right now whether the breach originated from within the United States or outside the country. We also do not know why the Oldsmar system was targeted, and we have no knowledge of any other systems being unlawfully accessed because of this security breach," Gualtieri said at the press conference.

The FBI is working with Oldsmar and the Pinellas County Sheriff’s Office to investigate, Andrea Aprea, spokesperson for the FBI’s Tampa Division, said in a statement.

The Secret Service is also helping, Gualtieri said. An agency spokesperson said the Secret Service does not comment on "ongoing investigations."

"These cases will only become more frequent and illustrate the need to be proactive in cybersecurity in the utility context," said Tracy Mehan, executive director of government affairs for the American Water Works Association, which represents major water suppliers worldwide.

An EPA spokesperson said that what happened in Oldsmar underscores the "importance of vigilance by water utility employees and staff in addressing the threat of cyber intrusions" and that the agency has tools to help water and wastewater utilities identify, respond to and recover from cyberattacks.

Under a 2013 presidential policy directive, EPA is the designated sector-specific agency for ensuring the nation’s water sector is prepared for any hazard, including cyberthreats. Legislation passed by Congress in 2018 tasked EPA with collecting data from water utilities by requiring them to carry out "risk and resilience" assessments of their networks, including a review of cyberdefenses.

Echoes of grid hacking

Authorities said the water utility employee in Pinellas County watched the hacker move the mouse cursor across the screen before the operator could reset the sodium hydroxide amount to safe levels. The ghostly maneuver harks back to the first-of-a-kind hack of Ukraine’s power grid in 2015, when grid engineers watched as Russian hackers methodically moved the cursor around, changing controls and causing a blackout that left more than 250,000 people in western Ukraine without power for hours in the middle of winter. The hackers in that case took the extra step of locking grid operators out of their own workstations, forcing grid operators to flip switches in the power system by hand.

Worries over critical infrastructure security have steadily increased in recent years as the convergence of IT and OT has opened new avenues for hackers to exploit.

Daniel Kapellmann Zafra, manager of analysis at cybersecurity firm Mandiant Threat Intelligence, said in an email that last year saw an increase in cyberattacks by low-level hackers looking to access and learn about remotely accessible industrial systems. The newfound interest is likely due to the rise of tools and resources available for malicious hackers, Zafra said, adding that most hackers have little real-world impact.

"Many of the victims appear to have been selected arbitrarily, such as small critical infrastructure asset owners and operators who serve a limited population set," Zafra said. "While the [Oldsmar] incident does not appear to be particularly complex, it highlights the need to strengthen the cybersecurity capabilities across the water and wastewater industry similarly to other critical infrastructure sectors."

Last year, the National Security Agency and the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency warned that foreign hackers are using internet-accessible OT systems, like those that manage the power grid, as an access point for cyberattacks (Energywire, July 28, 2020). DHS and CISA declined a request for comment.

"The days of isolated OT networks are long gone," said Tenable’s Edwards, who formerly led the DHS team that responded to cyber emergencies in industrial control system networks. "In its place is a highly dynamic and complex environment of smart OT technology, modern IT and everything in between."

Edwards noted that the hack in Florida is an example of how quickly potentially deadly changes can be made in an industrial environment and is why security experts have warned for years of rising threats to OT systems.

"This wasn’t the first attack of its kind, and it certainly won’t be the last," Edwards said.