Featured image by NASA/Goddard Space Flight Center Scientific Visualization Studio.

This will be a long post.

Hurricane Rita

10 years ago today, Shannon and I were staying with Mom and Dad. We had evacuated from our house in Beaumont, Texas a few days or so before. Hurricane Rita was so awesome/frightening on the satellite picture…just like Katrina was. I was assigned to the evacuation team at work. The drive took a toll on us and was grueling because we got stuck in traffic, Shannon got lost and we were extremely tired. We had our two dogs, two cats, and little Cecelia packed into my car and a rental while our minivan was in the shop. Hurricane Rita hit that day…just a month behind Hurricane Katrina.

Remember Xanga? I found my post:

 

Thu, 22 Sep 2005 12:24:08

We left Beaumont at 1am thursday morning with our daughter and our 2 dogs and 2 cats. We finally made it to Calhoun,LA around 9:30. 8.5 hours for a normally 5-hour drive. Not too bad considering we don’t live in Houston!

I am tired…I have been up for 30 hours…Shannon hasn’t slept for 2 nights in a row.  I need a shower…I have been wearing the same clothes for those 30 hours. And it’s freaking HOT!

Well, for my peeps in North Louisiana….wanna hang out? I ain’t got much to do till at least Sunday.

I’m pooped….and I need a margarita….not a hurricane rita. Play it Stevie!

HOLLA!

Chris

The Aftermath

We watched the TV as all the weather folks were in Port Arthur, High Island, Cameron, and especially the one that set up at the Entergy Edison building in Beaumont. http://www.cnn.com/2005/US/09/24/news.update.sat/ I remember the bricks falling off of the Transmission Operations Center (TOC) parking garage.

I had a conference call the next morning with my boss (in SCADA system support) and the Beaumont TOC folks that stayed during the storm. They were okay, but they were pretty scared that night. My boss said that they felt the building creak and they had to put the tiny little tiles back in the Transmission SCADA map board as some had popped out.

I asked a coworker to drive by my house and see if it was okay…he sent me a picture from his phone. That was the first time cameras were on cellphones.

I headed back to work on Monday. I remember getting to the roadblock near Beaumont. I showed the Guardsman my Entergy badge and he immediately waved me through saying “Get to work!” I saw our neighborhood in Old Town Beaumont…and looked like Sherwood Forest fell down. Many houses on our street and neighborhood had trees that fell on them. Our tiny house was spared. The first thing I did was clean out my refrigerator. I was just in time to save it from certain death. Check out this footage from by my house and the rest of the SETX area.

When I got to work we only had power to the SCADA server room and the operator arena. We all slept in the server room…it was so so so cold. The rest of the building was very hot. My office was 90°F or so in the day. A few days later we were able to get the 500kV line from Mount Olive to Hartburg heated up…after blowing lightning arresters in the process. We finally got Travis Street sub back on and power to the building…then I could sleep in my office.

The SCADA database ran out of tags…which was very interesting to fix. We worked long hard days…to help coordinate the crews as they heated up Transmission lines and substations.

The best beer I ever had was when we got power to the convenience store…it was a huge Budweiser. I sat in my car in the parking lot before going back to my “room” and drank the coldest and most refreshing beer ever.

One day I went to see United Methodist Temple, our church in Port Arthur where Shannon was youth director. It was demolished. The amount of wind damage was astonishing. Our minivan was somehow spared (almost have 200k on it now). One of the church staff ended up making hand crosses for people out of the main sanctuary cross that was destroyed.

We were out of power for 18 days to our house. I called Shannon and it was time to come home. And yes…that was a large car rental bill, but we didn’t have a choice. I did get a chainsaw from FEMA…and put it to good use.

I’ll never forget the smell of those rotten refrigerators that were on the side of the street.

http://www.beaumontenterprise.com/news/hurricane-rita/

Incident Response

Katrina and Rita…a double-whammy. We at Entergy never imagined two back-to-back storms. We got everyone that could take power (over a million customers) in like 40 days. Two years later we’d have another double-whammy (Gustav and Ike) that was the same number of outages…but we got them back on in 18 days. Working storm restoration was absolutely amazing. Pure adrenaline, working safely for 16-hour days, and the rush of getting people’s power restored was just a few of the things that I experienced working storms. In times of despair (small or multiple states in the case of a natural disaster)…this is when humans really shine bright. Helping each other when it truly matters.

Entergy is still winning awards for storm restoration. Their Hurricane Drill (“Incident Response Plan” to the security people reading this) is a well-honed machine that has been formed by the winds and rain of many hurricanes. The playbook is a sight to behold…and is unlike any security Incident Response Plan I have ever seen since working at Mandiant. We in the security and DFIR business should get some cross-pollination from these type of disaster recovery plans to our own IRPs. And power companies, oil & gas, water, and other critical infrastructures should incorporate computer-related disasters into their Disaster Recovery and Business Continuity playbooks and plans. No need to reinvent the wheel. This is why the electric sector, lead by NERC, conducts the GridX series of drills.

Remember

I’ll never forget ‪#‎HurricaneRita‬. <insert pic of Jim Cantore that he sent me here…as soon as I find it>

To all of my friends and colleagues in the power industry…you’ll always be my brothers and sisters because of events like this. Thank you!