Repost: Hacking the power grid through air conditioners

This is a repost of a blog that Joe Marshall (@ImmortanJo3) and I wrote on February 22, 2016 and @da_667 posted to his blog (which is now defunct, but he has given me permission to post here).

It’s not that easy..

Ladies and gentlemen, I am proud to host another guest work on blindseeker. This article was a collaborative work between two gentlemen with a wealth of knowledge regarding ICS/SCADA systems, including security and operations on the US Electrical Grid.  Before we get started here, these professionals wish to present this caveat:

We stress that this article  relates to the U.S. grid and it’s customers – it is difficult for us to talk authoritatively about grids not here in North America (though there is certainly some uniformity in how power grids work). The threat of something like this might be more pronounced in Europe or other parts of the world,  and it’s not for us to say one way or another. Also, this article and its opinions are solely of its authors, and the alien overlords who telepathically command us.

So, without further adieu,  here ya go.
~DA_667

Recently, Wired.com produced an article “How to hack the power grid through home air conditioners”. The article attempts to paint a picture that the power grid is at threat if a malicious attacker can cycle a residential or commercial A/C unit via RF signals, it would cause an interruption to the power grid and affect service delivery and reliability (read: blackouts).

We question the feasibility of this as a viable means to affect the health of the U.S. electricity grid. But for us to explain that, we’re going to have to take a deep dive into how power is distributed to homes and businesses. Hopefully, at the end of this, you’ll agree with us and realize that while there are some serious threats to the power grid, this one might not be as bad as you think. To begin, let’s talk about Demand/Response.

Demand and Response: A Basic Primer

As a rule, it’s not very practical to store power on ye olde power grid. Unless you start putting together batteries the size of skyscrapers, there’s just no way to keep power stored for long periods of time. As a result, supply and demand are kept pretty darn close to each other. You produce just as much power as you need, and use as much as is being produced. Pretty simple concept, yeah?

Any number of situations can result in an increased demand for power. Let us imagine a simple scenario: A heat wave over a large geographic area is causing energy usage to ramp up  – after all, most energy consumption comes from homes and businesses keeping their places climate controlled (as well as other power-hungry devices, such as water heaters), because it’s no fun dealing with hot weather. Those kinds of energy demand spikes can be a problem for utilities and RTO/ISO’s (ISO is an Independent System Operator and an RTO is a Regional Transmission Organization[1]). During peak demand, the transmission and distribution grids can get heavily taxed, and that can affect power delivery and reliability.

Enter Demand/Response (Henceforth known as D/R) to the rescue! Utilities figured out early that it was easier to control the load on their grids rather than manage generation. Subsequently, all utilities have some kind of voluntary smart energy saving program, which allows a utility to control your thermostat and/or compressor in the event of a peak usage warning – compressors are notoriously power hungry, and having control over it can lessen the load on the grid. Typically this means that your utility installs a thermostat into your home, that can be remotely controlled via radio frequency. For the inconvenience of this, they cut you a break on the price of energy you pay year round, or during an actual D/R event.

Technical details of D/R systems

Traditionally, most D/R systems worked over a radio waves, usually UHF/VHF RF. These systems are very similar to the one-way pager systems of the 90s .

The thermostats themselves tend to be very basic in both form and function. Once a D/R event is issued, somewhere a button is pushed in a control center, and your utility blasts out a one way signal to your thermostat to cycle off your A/C compressor. Not all systems are the same – some will give you control over your thermostat manually after a signal is sent – some will outright prevent you from managing your thermostat until they send out an all clear. It varies by ICS vendor and utility.

There are a lot of inherent problems with this pager-like system; Wired’s article notwithstanding. Utilities have no way of tracking whether or not the RF signal actually reached your thermostat. Maybe the thermostat isn’t on? Or perhaps the signal can’t reach certain homes?  Generally speaking, it’s just not terribly reliable. However, the idea is quantity over quality: if the utility sends out enough requests to their wireless thermostats, then some of them are bound to get the request and adjust the air conditioning units accordingly, achieving the goal of reduced load on the electrical grid. Since they’re audited by RTO/ISO’s for load management (read: keeping them honest), utilities have yearned for a better way to initiate and track an actual D/R event.

So, now that we have explained both what D/R is, and how it has worked traditionally, let’s talk about the attack.

The Attack

The supposition is this: By snooping RF commands sent via a cellular/RF network, an attacker can reverse the commands and send commands to a thermostat to cycle a compressor in the guise of a D/R event. With rapid, repeated cycling of load created by engaging a compressor via thermostat on a large scale, this would create an unbearable load on the local grid, causing outages on a large scale. With this attack , a bad guy could use power delivery as a weapon for any number of motives.

Where things get shaky is the implications as to what it means for for the distribution network that delivers power to said home/business. Could this destabilize a local grid, or larger distribution and transmission network? In short, we believe the answer is ‘no’. Why? A combination of factors make the likelihood of affecting the grid on any meaningful scale an unlikely possibility.

The Rebuttal

Here are some facts:

1. With a few notable exceptions, Demand/Response is voluntary. And for the right reasons – you might have vulnerable family and pets at home, and wouldn’t want them to endure high temperatures during a D/R event.You might operate a business that heavily relies upon climate control. These reasons (and others), might explain the somewhat tepid enrollment in D/R to begin with. In 2014, only 9.3 million out of 147,373,702 energy customers were enrolled in the entire country (that’s about 6.3%). It’s simple: If you aren’t enrolled in D/R and thus have a vulnerable thermostat, then this attack doesn’t apply to you. In aggregate this lessens any kind of perceivable risk to the grid.

2. Distribution transformers and feeders are designed to handle inrush current, which is the initial inrush of current when an electrical device is energized. When designing and maintaining a distribution network, utility engineers take into account the electrical load from every type of customer, from a small house to large box stores to hospitals and even large industrial customers. When a motor is started, the inrush or starting current can be 50 times larger than the normal running current.  Motors are everywhere in the average home: power tools, ceiling fans, refrigerators, washing machines, dryers, dishwashers, and the air conditioning system. The largest in the home is typically the air conditioner condenser motor, which can range from 1 to 5 horsepower in size. For example, a 5-ton ac unit can have 150 locked rotor amps (LRA) at 240 volts.

Now let’s imagine a situation. Let’s say that a distribution feeder serves 1000 customers. A squirrel causes a problem next to the substation and the feeder breaker trips, causing the 1000 customers to lose power. It’s a 95 degree August day, and everyone is running their air conditioners when the outage occurs. The squirrel is removed and the power is restored to the feeder. This instance is called cold-load pickup, where the inrush current is many times greater than the normal current on the feeder due the massive residential load, especially ac units, turning back on all at once. If the design of the feeder and its protection is not correct, then the feeder could trip back out again, and the customers would not be happy. This is why engineers account for cold load pickup in feeder design, especially in the Southern United States, where most homes have air conditioners.

3. Many utilities are phasing out these antiquated RF-based D/R systems in favor of newer systems and protocols, such as zigbee or wifi-based D/R systems utilizing commercial off the shelf thermostats (such as google’s NEST and many others). These newer systems deliver both enhanced reliability and better metrics/reporting from these devices out in the field (i.e. in your home!). These new systems promise better reliability, control, and granularity. And as a nice bonus, these flavors of D/R aren’t vulnerable to the stated RF attack. As a caveat, however, there is no percentage breakdown in the the electric sector as to what competing systems are used (Traditional RF v. Zigbee v. Wi-fi). For the sake of simplicity, we will argue that certainly a small percentage of D/R systems do not utilize the traditional RF flavor of D/R, and that number is growing as the older systems are being phased out.

 

4. Even if you are actually enrolled in a D/R program, and even if they use a vulnerable RF based D/R, and even if their distribution grid design isn’t up to the task, there’s still the problem of actually getting the RF command to your thermostat. The authors will concede there are numerous ways for a bad actor to saturate an area with RF, but keep in mind – utilities have a very large and very expensive RF infrastructure – and still have no way to reliably ensure that RF signals sent to participating customers in a demand response system are being received (as mentioned above in “Technical Details of D/R systems”). We’re talking massive service areas here – for example, in Baltimore Maryland, the local utility, BG&E, has a service area of 2300 square miles, and still can’t ensure RF reception for reasons beyond their control. There are numerous complications with using the older RF systems, chief amongst them is that it’s one-way. The lack of return telemetry makes gauging the effectiveness of the system highly problematic. The advent and popularity of smart meters has helped utilities gauge program response, but doesn’t solve the problem if a thermostat isn’t switching when ordered to, and this happens more often than you would think. On the bright side, thermostat technology is changing [pdf] (see #3).

Attack Impact and Viability

We find that exploitation of RF signals sent to RF-enabled D/R systems is entirely plausible on a small scale; kudos to the researchers for discovering this weakness, and pointing it out. It is entirely possible for an attacker to replay commands to a D/R thermostat, to cause a compressor to cycle. If you live in home with a D/R thermostat or ac compressor switch, you have a right to feel a little concerned. If you have vulnerable pets or family members, it’s worth being aware that this could possibly affect your air conditioning. We would give you some hope, however, that this attack has a reasonably modest barrier of entry, and the solution would literally be as easy as switching out your thermostat if you suspect you are a victim. Keep in mind, this might unenroll you from any D/R program you are in, but it’s no fun having a heat stroke.

It’s worth noting that the old D/R system has been around for a long time. It perfectly fits the utility model of “If it’s not broken, don’t fix it”, the bane of all information security programs. With security researchers shining lights on antiquated and vulnerable ICS technologies, hopefully it can spur better security innovation in an industry that is loathe to change itself.

Edit: 2/26/2016 – [Removal of “Context Matters”] After some discussion, we have decided to remove a single paragraph here discussing the vetting and the quality of the Wired article referenced. The paragraph in question didn’t contribute to the discussion in a meaningful way. Considering how difficult information security is in general (ICS security in particular) we should be tearing down walls, instead of building them up.

Conclusion

We find that the attack based on snooping/replaying RF commands to a thermostat valid. There are no security checksums or validation of RF commands sent from most D/R systems to thermostats; They will respond to any properly formatted request as per design. A hat tip to the two security researchers who discovered this are in order. However, don’t buy into FUD. The real danger this vulnerability exposes is to homes and business owners who have vulnerable thermostats, who rely upon climate control. We find the threat to the stability of any actual grid not plausible.

This article was co-authored by @ImmortanJo3, team leader of Cisco Talos’ ICS security division, and @chrissistrunk, an ICS security professional for Mandiant. With some minor edits made by @DA_667.