ATLANTA — A Midwestern utility lost communications with a substation when a telecommunications carrier’s line failed, at 6 p.m. on a Friday. If connections couldn’t be restored in short order, the utility could be facing $1 million a day in fines for violating Federal Energy Regulatory Commission rules.
But the telecom company said it no longer offered round-the-clock service. Its service trucks rolled during the day, Monday to Friday, said Dan Belmont, whose Chicago-based firm, West Monroe Partners, had to scramble to create an alternative communications channel for the utility.
And executives of an upstate New York utility were stunned last year to learn that their telephone carrier was about to terminate the phone lines used for their operations — an earlier warning was misplaced in the mail.
The incidents, related during the Utilities Telecom Council’s conference here this week, dramatize a jarring change affecting the communications networks that link most utility substations to control centers.
AT&T, Verizon and other carriers are putting utilities on notice that they are closing down the old, standard telephone services that carried the utilities’ operating data.
It is another of the operational challenges landing on the desks of utility CEOs that are the consequence of incessant technological change — in this case, consumers’ embrace of cellphones or phone over Internet rather than "plain old telephone service."
"Those networks are dying," said Belmont, his company’s senior director for energy and utility solutions.
Traditional residential phone service has been falling off, and that’s the same equipment utilities use, said Mark Madden, regional vice president for North American utilities at Alcatel-Lucent.
"We’re getting to the point where the big telecom carriers are no longer willing to invest in these networks," Madden said. It’s not going to happen everywhere, all at once, he added. "But utilities are going to have to do something, and they don’t have much time," Madden said.
Belmont and Madden are two of the principal authors of a white paper on the network transition published by UTC.
"Those old services are going away. No matter how satisfied you are … they’re not going to be there much longer," said Keith Porterfield, vice president for telecommunications for Georgia Systems Operations Corp., grid operator for member-owned utilities in the state.
A costly day of reckoning
Telecom operators have secure options to offer, Belmont said. The banking industry, for example, has invested in secure advanced telecom networks embedded with cybersecurity measures.
But the new options — dedicated fiber-optic channels, or MPLS (multiprotocol label switching services), for example — can cost as much as 10 times what the utilities were paying for their communications, Belmont said. That is a tough pill for utilities to swallow, and tough, too, for the regulators who see ratepayers’ expenses rising for grid automation, cybersecurity and extreme weather defenses, and new transmission lines.
Utilities had notice that this change was coming, but some put off the reckoning.
"For us, the key driver was a hard date," Porterfield said. AT&T announced that GSOC would lose its frame relay service (so-called because data moves in discrete frames or packages) at the end of December last year. "AT&T drew a line in the sand."
"We got the message and began the transition," he told an audience at the UTC conference. "It’s been challenging, but we’ve made it 95 percent of the way through." GSOC was able to negotiate the per-site transition cost nearly in half, he said. His company’s replacement for critical functions was built around a corporate network with zoned firewalls, with maintenance and billing in different zones. "We are not on any publicly accessible IP space."
Cheaper options, including cellular communications channels, aren’t acceptable to utilities from a reliability standpoint, Porterfield said.
"It was not just a ROI [return on investment] calculation," he said. "We were also looking at the future." The new system allows enhanced security and provides lower maintenance costs.
Utilities must take into account the data requirements of their equipment, Madden said.
An example is a high-voltage circuit breaker programmed to respond to queries about the status of the line within 10 to 12 milliseconds, or the controls will assume there is a fault and the breaker will trip, shutting down that section of the line, he said.
Some of the new connections telecom companies are offering utilities may not be able to handle traffic that quickly, and a false distress signal could occur. "If the communications fail, you could be back in a situation like 2003," he said, referring to the Northeast blackout that year, caused by a series of line outages that ended in a cascading failure.
"Utilities have been relying on security by obscurity" for a long time, he said, meaning that their older, unique software controls didn’t draw hackers’ interest. That goes away, too, if utilities wind up on state-of-the-art networks.
"In IP [Internet protocol] networks, the vulnerabilities are well-known and there are many more attack surfaces, but there are also a lot of defenses," he said.
"Utilities have to make the decision, ‘Do I do it myself’" or buy a new service from a utility, Madden said. "Replacing the service is not trivial," he said.
