Lessons Learned from Hurricane Sandy

This audio was created using Microsoft Azure Speech Services

The devastation that we saw in areas such as Manhattan after hurricane Sandy naturally prompts lots of discussion about what went wrong and whether data center downtime directly caused by the storm could have been prevented. We can and should learn lessons from any event that affects our data centers, whether it’s a relatively routine weather event or a more serious one like Sandy.

At the same time, we should keep in mind the risks associated with any data center construction project are typically thoroughly assessed, and by-and-large known from the start.

For all of us who have been involved in any aspect of the business of data centers, there is a general recognition that “intelligent compromises” are nearly always necessary to enable a design or construction project to progress from concept to completion; no data center would ever be completed without ascertaining and accepting certain risks to the project, and to the enterprise.

However the successes of some data centers and the weaknesses (or failures) of others usually depend on those pragmatic, as well as often clever design details that impart certain essential characteristics of a data center, that may eventually be favorably judged.

In many respects, the risks that were realized during Sandy were probably identified by the data center designers, builders and owners and judged at the time to be acceptable because they were rarely seen. Perhaps next time maybe those same risks will be judged differently.

That said, Sandy did make a few things clear about how we design data centers and supporting infrastructure. And keep in mind, that while the level of devastation we saw from Sandy was indeed of exceptional magnitude, many of the problems involving water in individual buildings that occurred during Sandy could have happened in other ways; say a major water main break, for example.

Uncontrolled water is, obviously, a risk to any data center. That’s why most companies have done a good job of getting IT infrastructure and vulnerable data center network assets out of the basement and even above the first floor.  However many firms have left behind a large portion of the supporting, so called “grey space” infrastructure, including electrical switchboards, and switchgear, and essential mechanical equipment such as pumps and chillers.

The experience during Sandy reasserts that modern grey space infrastructure is just as susceptible to water damage as a server. It’s probably time that a greater level of consideration be given to moving all of the data center’s physical assets out of the lowest levels of the basement and in some cases, even off the first floor of a building, leaving behind only those electrical and mechanical assets that can safely withstand and remain operational through and after an inundation.

Avoiding single points of failure throughout the entire stand-by generator system is important, too. For example, the little transfer pumps that move fuel from a bulk storage tank to a generator’s day tan, have for a long time been the Achilles heel of generating systems, suffering failures from localized mechanical and environmental issues.   Even in cases where generator sets were many floors removed from the flooding, and adequate fuel was available in tanks with tightly sealed fill pipes and tall vents, exposed transfer pumps got drowned and ceased to work, leaving the diesel generators starved for fuel.

For data center stand-by generator systems, it’s now time to take a fresh look at the same pump technology has been widely employed in gasoline stations for more than 25 years. In these systems, the actual pump is submersed in the fuel of the tank, safely protected (not to mention cooled) by the fuel within the tank, generally impervious to the prevailing environment.

Another area where I expect we’ll see greater interest is generators that can run on either diesel or natural gas or a combination of both within the same engine, or a generator plant with a mix of separate diesel and natural gas powered generators.

During Sandy we saw cases where data centers that were well prepared, with generator fuel storage that was high and dry, but they ran into problems with fuel diesel deliveries. At the same time, there was ample natural gas available, (which is generally considered an “interruptible” fuel).

Some technologies will certainly get a fresh look. They include spark ignition natural gas generators, and commercially available retrofit packages that convert diesel engines to operate on a continuously variable mix of conventionally injected diesel and throttle body injected natural gas. Innovations in diesel/natural gas injection technologies presently available to, and getting favorable reviews from the over-the-road trucking industry, will hold promise as higher horsepower applications are developed and commercialized.