Five times over three years, a desperate scenario has played out on Plum Island, an isolated spit of land just off the northeastern tip of New York’s Long Island. A large part of the power grid has gone down, leaving the population in the dark and critical facilities such as hospitals growing desperate. A team of utility operators and cybersecurity experts scrambles to get the grid back up, while hackers try to keep it down.
Each emergency was a drill held by the Defense Advanced Research Projects Agency (Darpa), the Pentagon’s moonshot research arm. Its goal was to expose utilities accustomed to dealing with hurricanes, blizzards, and other challenges to the reality of a successful cyberattack on the U.S. electrical grid.
Concern about such an event has been mounting within the U.S. government for years. Darpa began laying the groundwork for its drills in mid-2015, part of a five-year, $118 million project called Rapid Attack Detection, Isolation and Characterization Systems—or Radics—after chilling congressional testimony the previous year from then-National Security Agency Director Mike Rogers. Rogers told lawmakers that hackers had been breaking into U.S. power utilities to probe for weaknesses and that Russia had been caught planting malware in the same kind of industrial computers used by power utilities. “All of that leads me to believe it is only a matter of when, not if, we are going to see something dramatic,” he said.
The problem has seemed especially urgent in recent months, following a series of ransomware attacks on U.S. facilities and rising tension with China and Russia. Russian troops are massed on the border of Ukraine, a country whose power grid has been hit twice by Russian cyberattacks. Last year the White House launched a 100‑day sprint to accelerate longer-term projects fortifying America’s power infrastructure against similar attacks.
In late December, U.S. officials privately warned utilities they could be targeted if relations with Russia deteriorate, telling them their security teams shouldn’t take the holidays off, according to two people familiar with the briefing. On Jan. 11, U.S. officials publicly called on utilities to comb their networks for signs of Russian intrusions. Secretary of the Army Christine Wormuth recently told reporters that the power grid would also be a target in a conflict with China over Taiwan.
The drills on Plum Island starkly illustrated the chaos hackers could unleash. Attackers hijacked critical safety equipment, shut down communications, and sent fake data to confuse operators making crucial decisions. Utilities that were once confident they could keep from being hacked are no longer so sure. “What we’ve seen as a country is the adversary is going to be successful,” says Walter Weiss, Radics’ program manager. “The issue then is, what do you do next?”
While the government periodically practices such scenarios, utility operators rarely do. Until it ended in 2020, Radics offered the 15 utilities that participated near-real-world conditions to test new technologies, some of which they’ve since implemented. It also jolted them out of any complacency they may have had, says Brian Lynn, a lead trainer for PJM Interconnection LLC, the country’s largest grid operator, who advised Darpa throughout the program. “Anyone who was there really had their eyes opened up,” he says. “And they were able to go back as a firsthand witness to each of their companies and say, ‘Hey, this is a real thing.’ ”
Most histories of cyberattacks on physical infrastructure start with Stuxnet. The 2010 attack, believed to have been carried out by the U.S. and Israel, destroyed more than 1,000 Iranian nuclear centrifuges by manipulating the industrial computers that controlled them. Modern power grids are also heavily computerized, making them more resilient during storms and other weather-related disruptions but also opening new vulnerabilities for cyberattacks.
Russian hackers carried out the first major cyberattack on a nation’s electricity grid in late 2015, taking down part of the Ukrainian national grid for six hours. The following year they staged another attack on Ukraine, infiltrating a transmission substation north of Kyiv and tripping every circuit breaker, briefly severing the flow of power to a section of the city. The hack was meant not only to punish Ukrainians but also to show what Russia could do to other adversaries, according to Andy Bochman, senior grid strategist at the Idaho National Laboratory and one of the top U.S. experts on cyberthreats to the grid. “Both attacks in Ukraine were demonstrations,” he says. “And the whole world was watching.”
Shortly afterward, Russian malware was discovered inside as many as 10 U.S. utilities, including the operator of a nuclear plant in Kansas. Government officials hastily convened a series of secret briefings with utility executives, prompting the power companies to spend months scrubbing their systems, according to three people familiar with the incident. The U.S. Department of Energy continued briefing power company executives, warning that, among other things, potential adversaries had been caught manipulating grid components during manufacture, according to two people familiar with the briefings’ contents.
Although the most sophisticated attacks are likely to come from nation-states, the spate of ransomware attacks over the past year shows how widely the ability to paralyze physical infrastructure through cyberattacks has spread, says Ang Cui, founder of Red Balloon Security and a participant in the Plum Island exercises. “What an attacker can do to these embedded devices today makes Stuxnet look like caveman technology,” Cui says.
Hackers who want to bring down a grid would likely manipulate the computers that keep it in balance. Operating a modern grid requires constant realignment to make sure the amount of power sent into the system is equal to the power that households, businesses, and other customers pull from it. Eric Hittinger, an expert in energy policy who’s an associate professor at Rochester Institute of Technology, likens this process to a bicycle rider constantly shifting her weight to stay upright. If that balance is disrupted badly enough, he says, “everything starts to fall apart. Different parts of the system will start to turn off in unpredictable ways. You end up with cascading failures. You fall off the bike.”
This can happen because of natural events, such as the winter storms in Texas last year, when electricity heading into the grid fell after the weather took power stations offline. The Electric Reliability Council of Texas, the organization that operates the grid, responded by shutting off power in major population centers, averting a cascading collapse by just minutes, according to three people who reviewed data from the incident.
One of the hardest parts of a grid failure is repowering it following a collapse, and the biggest outages could require a tricky maneuver known as a black start, which involves restarting the grid without power from outside the blackout zone. A particularly nightmarish scenario—and the one that Darpa simulated in its drills—would be a cyberattack where hackers stay in the system and repeatedly disrupt the restart process. In this situation, a blackout that would’ve lasted hours could extend to weeks. “When it comes to cyber, it’s like you’re repairing the damage from the hurricane while it’s still on top of you,” says PJM’s Lynn. “And I just can’t fix it and know it’s going to hold. I’ve got to keep asking, ‘Did I miss something? Is something still infected?’ ”
The first drill on Plum Island took place in 2018, with subsequent exercises occurring until October 2020. The action centered on Fort Terry, a now-abandoned part of New York City’s coastal defenses. The remote island, which is also home to a high-security lab used to study contagious animal diseases, is accessible only by a ferry behind a guard post; the 100 or so participants in each drill spent a week to 10 days there, returning each evening to hotels on Long Island. Employees from utilities and people from National Guard units role-played at their day jobs, while Darpa brought in cyberwarfare experts to act as the hackers.
Participants are loath to reveal many details for fear of giving attackers useful information, but among the biggest challenges was a culture clash between seasoned utility operators and experts in cybersecurity. “You have to get your systems operations guys, who don’t speak cyber, to talk to your cyber guys, who don’t speak systems operations. And that’s just very challenging,” says Donnie Bielak, a colleague of Lynn’s at PJM who also consulted on the exercises. He remembers the initial attitude of a shift operator from New York as typical: “Basically, he arrived saying, ‘All right, all you cyber nerds, hands off, I got this.’ ”
Once the exercise started and the power went off, though, the inadequacy of the typical recovery techniques became clear. The attackers manipulated data coming from sensors—showing a circuit breaker as open when it was closed, for example. Even unsophisticated-seeming tricks were disruptive. In one case, attackers dimmed the brightness on a device’s screen, leading recovery crews to waste time misdiagnosing what seemed like dead equipment. In another, operators accustomed to checking the status of components with an online app were flummoxed when the hackers disabled the portal so they couldn’t log in.
At the beginning of one exercise, Weiss reminded a group of cybersecurity experts of their own lack of preparedness by simply flipping the circuit breaker to the conference room where they were gathered. Anyone who hadn’t brought extra laptop batteries or had forgotten a headlamp was basically out of commission.
The exercises generally started with failure, followed by slow progress as the defenders learned to work with experimental technology developed for Radics—and how to operate in their new, contested environment. Eventually, Bielak says, the utility workers who initially brushed off the cybersecurity experts began to work closely with them. Defenders had to come up with procedures to clean each substation of malware, for instance, before connecting it to the larger grid. If they missed anything, the hackers might flip a breaker at the wrong moment or send manipulated data that could confound the recovery effort.
Even seeming victories were fleeting. During one exercise, defenders managed to restore the grid and reestablish the operations of the model utilities. Power began flowing again to key sites across the island, Weiss recalls, and everyone involved erupted in cheers. A few moments later the grid came crashing down again.
Radics, like many Darpa programs, had a limited run, and the exercises ended when the program did. Some of the test equipment remains, and the Energy Department continues to test cybersecurity technologies on the island. But most of the utility workers are no longer involved, a situation that Lynn worries will result in participants letting their newfound skills go stale. “You want that muscle memory,” he says.
Even if they were to continue, the Plum Island exercises involved a small number of utilities, and experts are concerned that the country remains unprepared for a dangerous and unpredictable threat. The situation on the Ukrainian border highlights how foreign conflict raises the likelihood of an attack. Even an attempt to apply political pressure by conducting a limited cyber operation could set off failures beyond the attackers’ control, according to Tim Roxey, former security chief for North American Electric Reliability Corp., the grid regulator. “The collateral damage,” he says, “can sometimes be far different than what the intent was.”
Plum Island has had some lasting tangible effects. The National Rural Electric Cooperative Association, whose members are mostly small, not-for-profit operators, recently began using a new security tool that it first tested on Plum Island, according to NRECA Chief Scientist Emma Stewart. Once fully deployed, the tool, called Essence, will help the co-op members, which have comparatively small security budgets, detect cyberattacks on some of their most sensitive equipment.
Cui, whose company makes some of the tools that were tested on Plum Island, says the U.S. has a long way to go in preparing for cyberattacks and in training the utility workers who will be on the front lines. His team recently examined several common industrial computers used to control the grid and uncovered serious design flaws that could be leveraged by hackers. To prove his point, his team remotely hijacked one of the devices and shut off power to a scale model he built of Manhattan.
“How at risk is this country? Maybe a better question is: How much have we done to prevent something like those scenarios at Radics from happening?” Cui says. “I think it’s pretty clear that we haven’t done nearly enough.” —With Jordan Robertson and William Turton
Source: Bloomberg
