Hit the switch and test all your connected systems
Professional trainers know that the best way to teach a new skill, particularly an important skill, is to make it as close as possible to the real scenario. Rather than simply mimic, they try hard to recreate or immerse their students in a near-authentic scenario. It’s referred to as going ‘to the wall’. The closer to the wall the training, the more effective and valid the learning.
What’s the right load?
The same thing goes for testing backup power systems. The closer to a real-life test, the more merit the test will have. Unfortunately, in many data centres the methods used to test gensets may not be providing most valid assessments. A very common test relies on using load banks to test generators – where the load bank applies load to the generator(s), and consequently dissipates the power output. It’s supposed to imitate the actual load a power source will see in a real life outage. But unlike real life, a load bank gives a predictable, convenient load. A true utility outage is going to be dispersed, unpredictable and indiscriminate.
To be fair, testing a generator without sufficient load is hard on a diesel engine, which makes load banks a good idea if you want to prove your genset can carry their rated load. But it’s unwise to equate a load bank test with a valid backup power system test. It’s just too far from the wall; it doesn’t test the full system, and it doesn’t test other services in your facility. It leads to dodgy decisions about your backup system and misguided confidence in its performance during a power loss event. It can also lead to other issues with the generator such as wet stacking, where the exhaust system is gummed with unburned fuel or carbon buildup when generator run times are too short or loads aren’t sufficient. Testing only the generator and not the complete backup power supply is similar to testing the ending in an aircraft, but ignoring the wings.
Getting closer to the wall
As I mentioned earlier, the closer to the wall we can get, the more valid the test. So how about transferring the whole data center load to the backup generators to test the system under true load? It’s about as close to the wall – and as authentic – as you can get. Throw the switch at the utility, and you’ll test the UPS systems, standby power system, transfer switches, security, mechanical services, load staging and so on, all in one shot. You’ll also discover very quickly if facility wiring is reliable, and how much generator capacity you are really using
But who would take such a risk? Even though one feels confident their back-up systems are in working order, throwing the switch risks a shutdown if, in fact, there is a problem with the system. And a shutdown caused internally is just a little difficult to explain to management.
Perhaps the reason generators were not tested as frequently in the past is because the test involved stop watches, a significant number of personnel, and manual recording of test parameters. Test procedures were tough to coordinate and difficult to effectively measure and record results. It was a costly and inefficient endeavor, and there was a clear need to find a way to ensure backup power reliability without creating a lot of extra work for facility staff.
Automating the test
But how about automating the test? Automated testing greatly increases reliability with accurate monitoring and recording of test parameters. It includes detailed reports, and even traceability in case of unanticipated power generation system issues. It also greatly reduces staffing burden, and gives you demonstrated proof of due diligence of facility maintenance and testing should an outage occur and questions be raised. So perhaps at a time of low risk, you could decide to do the first test. Make sure there are qualified people on site to see firsthand how the systems behave and interact (and there’s no substitute for good engineering to keep the risk low). Be sure to explain to management and potentially affected parties that you’re doing the test. Once the first ‘low business risk’ test has been successfully completed, future tests can be done on a more consistent schedule.
As an added bonus, many utilities offer a variety of demand response programs. On days when the grid demand is heavy, the utility or their CSP call on enrolled companies to reduce consumption for a few hours (maybe 2-3 times/year). It’s a sound environmental policy and will probably save you from the risk of having a true blackout. Demand response programs can offer significant reimbursement per MW per year. If your data center generators can carry 3 MW of load on your generators, there could be a substantial return to be earned.
Participation in a demand response program helps ensure your generators work for several hours, your wiring is good and that you’re safe from extended blackouts. The best part is that if something goes wrong and you want to switch back to the grid you can, you just earn a little less return. Plus a demand response event will count as your monthly test! So while testing under your live load makes sense for a lot of different reasons, getting some compensation is a pretty great idea too.
And in the end, you’ll know. You’ll know your transfer switches and associated infrastructure function properly, and you’ll be very certain you won’t be getting any midnight calls about a data center gone silent with a botched backup system.
