When a group of hackers penetrated the Snohomish County Public Utility District in Washington state seven months ago, they had inside help.
Some employees of the state’s largest PUD opened an email cleverly disguised as work-related, and unsuspectingly downloaded an attack payload.
This particular nightmare scenario in the electric utility sector did no damage. The "attackers" were from select cybersecurity elements of the Washington state National Guard, including a top executive at Microsoft Corp. and other experts. They had been invited by the utility to try to break into Snohomish’s system to test its defenses.
"I wanted them break in," said Benjamin Beberness, the Snohomish PUD’s chief information officer. "I didn’t tell them how." And break in they did, in 22 minutes, added Jessica Matlock, Snohomish director of government relations.
Acknowledging that a cyberattack broke through is normally the last thing a utility would admit, but Snohomish did so to demonstrate the serious consequences for the electric power sector if its own employees neglect cyberthreats.
The critical importance of employee cyber awareness is a headline recommendation in a new cybersecurity defense primer issued by Snohomish, the Washington National Guard, and other government and private-sector members of the ad hoc Energy Sector Cybersecurity Working Group.
The publication, the "Cybersecurity Guide for Critical Infrastructure for the State of Washington," is aimed particularly at smaller utilities and municipal and cooperative electricity utilities that lack the staff and resources to invest in layers of costly cyberdefenses.
"We wanted to see if disparate players could come together and do something for small- and medium-sized utilities," said Gordon Matlock, cyber practice lead at Bridge Partners Consulting, a member of the group and Jessica Matlock’s husband.
For them, enlisting their own employees as the first line of defense is both essential and free, said Anita Decker, executive director of the Northwest Public Power Association, also a working group member. Her organization provides training and other services to nearly 150 consumer-owned utilities in that region. "This has really pulled out the key things that small utilities can walk through, in a much simpler manner," Decker said of the guide.
"People do take it very, very seriously," she said of her organization’s members. "But it’s not that everybody is at the same level" of preparedness.
"If you are a small co-op who has someone maybe watching your firewall once in a while, you are definitely someone we want to reach," Beberness said. "It’s not supposed to be a silver bullet, by any means, but it lets them know what resources are available to them."
The guide seeks to summarize and simplify the cybersecurity framework of defense strategies issued by the National Institute of Standards and Technology (NIST).
It includes top-level questions for executives and managers, such as: "Who in my organization is responsible for cybersecurity?" "What are the rules that govern my use of company resources?" "Does my organization have a policy on bringing personal devices into the workplace?"
Other questions challenge managers and employees about their cyber awareness. "What am I allowed to connect to my company’s system?"
"Basically, safety is part of every work process and people do not think about it as something separate from getting the job done. Cybersecurity should be treated the same way," the guide advises.
"What that document tries to do is take the NIST framework and break it down into small enough chunks, so that a smaller organization could get some understanding of where their gaps are," said Lt. Col. Thomas Muehleisen, chief cyber planner for the Washington Military Department. "Then they can decide to make investments in programs or people."
"You can do some easy things, like two-step passwords, training your employees, monitoring your contractors and consultants, instead of shelling out hundreds of thousands of dollars" for the latest advanced defense program, Gordon Matlock said. "Reach out, coordinate and collaborate with others with similar challenges. Find the information-sharing groups in your region. These don’t cost a penny."
The guide’s advice to focus first on point-of-entry threats is exactly right, said Miles Keogh, director of grants and research at the National Association of Regulatory Utility Commissioners. Keogh and his colleagues issued a cybersecurity primer and checklist of key questions that state regulators should consider when reviewing utilities’ cyberdefense preparedness.
"The human element is almost free," Keogh said. "It costs effectively nothing to have well-trained users. What we find is that the nation-state actors’ assured penetration techniques start with phishing or spear phishing" to trick insiders into clicking on links with hidden attack software. Such attacks "account for the enormous majority of initial penetration," he said.
Baiting the hook
That was the case in the Snohomish PUD exploit. It grew out of a relationship between the utility and the Washington National Guard’s cybersecurity units during two cybersecurity summits sponsored by the Snohomish PUD and the Pacific Northwest National Laboratory and other participants over the past two years. The summits were the foundation for the guide’s development, Muehleisen said.
The Guard units draw on technically trained members of Air and Army units, and cyber experts from the state’s technology companies who have enlisted in the state Guard unit. The latter group includes Russ McRee, director of security response and investigations for Microsoft’s Windows and Devices Group. McRee and cyberdefense developer Billy Rios were key contributors to the design of the Snohomish intrusion, Muehleisen said.
Rios, founder of WhiteScope cybersecurity resource firm, is a major in the Washington Air National Guard. "We do this for a living," he said of the civilians on the National Guard team that did the cyber probe of Snohomish.
The team was invited to try to break through the utility’s outer firewall defenses. Once they had shown the ability to get through, they could continue to go after defenses on a smart grid test unit at Snohomish, which replicates its operating system. That allowed the attack team to dig deeper into vulnerabilities without exposing the actual control system, Beberness said.
"They didn’t go through the firewall," he added. "They could have spent days trying that. It was so much easier to send out a phishing email and get in the back door."
Beberness said of the email, "Let’s say, it was enticing and appealing to people to click on the link."
Rios said, "This is a benefit of an elite team. We demonstrated to them we could do a lot of crazy things" inside the utility’s system. Neither he nor Beberness would add any details. Muehleisen said he has had to turn down other utilities that want the Guard unit to have a go at their systems. "The Guard is an extremely finite resource," he said. Future penetration tests will have to be justified as a training or emergency response exercise and meet the governor’s priority list, he said. Other utilities should turn to the cyber guide, he added.
"At the end of the day, we got to tell them about all the things we discovered," Rios said. The team "hammered home" that a utility cannot allow its control system to be exposed by a single failure point, he said. "We could sit at the same table and talk about what they could do to get the best bang for the buck."
If smaller utilities limit their cyberdefense investments to low-end actions, could they create a weak link that could pose a wide threat to the power grid?
The Western Interconnection grid network links all utilities west of the Rocky Mountains, large and small. Utility regulators point to the Southwest power blackout of September 2011 as an example of an outage that cascaded because of operating and planning failures among adjoining utilities of varying sizes.
Decker said small utilities aren’t thought of as priority targets. "They aren’t typically going to generate that cascading outage." However, she added, "If somebody wants to practice on the small folks before something large happens, there are a lot of them."
Rios said not enough is known about how a successful breach at one utility, regardless of its size, might impact the larger grid. "We need to ask ourselves — the leadership, the policymakers — what these utilities really mean to us. Is it OK that they are barely scraping by with the bare minimum" of cyberdefenses? "We have some theoretical understanding" of the question, he added. "But we don’t have enough data to understand what that really means."
