Powerless: Lessons learned at Super Bowl XLVII

This audio was created using Microsoft Azure Speech Services

New Orleans, February 3, 2013. One minute and 38 seconds into the second half of Super Bowl XLVII something extraordinary happened: Half the lights went out inside the Superdome. With 111 million viewers and another 70,000 in attendance, it may be the most infamous power outage in history. But it could’ve been a lot more than merely inconvenient. After all, the lights eventually came back on and the game resumed. Imagine a halftime power outage. Or what if the backup generators had failed?

You can play the “what if” game forever, but the point is – it could’ve been worse. And yet, the power outage was a breach of our sometimes unrealistic expectations of instantaneous, dependably reliable power. Many of us would expect ‘power outage’ to join ‘robot invasion’ and ‘salad shortage’ at the bottom of a list titled “Super Bowl Concerns.” In the lead-up to the Super Bowl, however, Entergy, the local utility, took nothing for granted.

Entergy began evaluating the Superdome’s electrical systems two years prior to the big game and stationed nine people at the event. Months prior to the game, Superdome management also replaced several cables leading into the stadium. Clearly, power management was a high priority for both parties.

In the end, it appears an electrical protective relay caused the outage by tripping a circuit breaker. But, more questions remain unanswered:

• What caused the relay to trip the breaker?
• Was it improperly configured? Was this human error?
• Was it in response to a short circuit or overload?

Getting these answers isn’t exactly a simple process. To better understand why, I tapped into the expertise of our Power Reliability team:

• Electrical events are measured in cycles and milliseconds. For a 60-hertz power system, one cycle equals 17 milliseconds. (For reference, the blink of a human eye is about 300 milliseconds.)
• A few examples of electrical events:
  • Utility voltage sags and surges (several cycles)
  • The duration of a lightning stroke (microseconds, or a few thousandths of a millisecond)
  • Short circuits causing ‘cascading’ faults, where a number of circuit breakers trip (several milliseconds)
  • Breaker switching (3-5 cycles)
• The time resolution required to record electrical events is, not surprisingly, also milliseconds. Supervisory Control and Data Acquisition (SCADA) systems just like the one in the Superdome can ‘poll’ Intelligent Electrical Devices (IEDs), such as meters and relays, but need several seconds between updates. Too slow to quickly identify the event that actually caused the outage in this case.
• A large electrical system will likely have hundreds of IEDs. Reconstructing an accurate timeline of the electrical events leading up to the outage is no small task. IEDs must be synchronized to a common time source down to the millisecond.
• That requires another system altogether: a sort of electrical event “black box” called a Sequence of Events Recording (SER) system and experts trained to use it.

So, what do we know? Are we certain Super Bowl XLVIII won’t be interrupted by another power outage? Unfortunately, no. SERs can’t prevent electrical events just like real “black boxes” can’t prevent plane and train crashes. But, SERs are definitely a step in the right direction toward comprehensive energy risk management as the most advanced systems available in diagnosing outages.

Extra precautions have already been taken in preparing for this weekend’s game. But, just to be on the safe side, if I’m in New Jersey, I’m asking the football gods for good weather and reliable energy. As of now, the forecast looks promising, but given recent events around Ft. Lee, maybe New Jerseyans are more worried about too much power.