Disaster Recovery Lessons Learned: The Seventh Anniversary of Hurricane Sandy

The following article was written by Matthew Hilsenrad, Director of Disaster Recovery at Abacus Group, and originally appeared on Disaster Recovery Journal.

During Hurricane Sandy in 2012 the New York Stock Exchange and NASDAQ were closed for two days and the super storm ended up costing New York State a staggering $32 billion, New York City $19 billion and the U.S. economy an estimated $65 billion.

Seven years later, it’s fair to ask what has Hurricane Sandy taught us?

Preparing for disasters, whether man-made or natural, has never been more critical for financial services firms because so much is at stake. With risk mitigation a priority, it’s hard to believe that many firms still do not have a business continuity plan in place. And some firms have plans, but they are authored without cross communication between business units, relying on each department to produce their own continuity strategy.

My firm is an IT/Cloud Services provider to alternative investment firms. The storm taught us the importance of geographic data diversity. Since Sandy we have opened new cloud data centers, always considering proximity to a secondary site so that our clients are prepared for a regional disaster scenario. For each data center we want to ensure there are different power grids, flood zones and alternate connectivity providers.

Sandy also showed the value of proactively moving essential client services to a secondary site ahead of a predicted major event. Firms with this type of disaster recovery strategy were able to work through Sandy.

We stayed open and learned a lot from Sandy. After the storm, our firm took further action to protect our clients. During Sandy, our secondary site was near Philadelphia, PA which for some clients was less than 100 miles from their primary site. Following Sandy, we moved all client data from their onsite offices to our more dispersed data centers. Within a year we started planning out a migration project which moved our secondary site to a location 1,500 miles from the primary data center.

Here are some additional best practices regarding business continuity and disaster preparedness:

• Ensure that you have a written business continuity plan (BCP). It may seem like a monumental task initially, but once you’ve started, upkeep is much easier.
• Make sure your plan is “holistic” – covering the entire firm. Make sure it is not “compartmentalized,” where one department doesn’t know what the other is planning.
• Your plan must be reviewed and approved by senior management (including boards of directors). Ultimately, they are legally responsible to customers as well as regulators.
• The BCP should combine both business and technology needs. It should also be accessible to all employees.
• The BCP should identify and include key services (connectivity, voice, email, data, applications, etc.) and vendors, with their contact information.
• Ensure that the BCP is reviewed on an annual basis, including documentation of all testing done since the last update.
• Work with your in-house IT staff or managed service provider to “stress test” your system and prepare for large scale outages like Sandy. Identify business critical workflows and include them during tests.
• Have a strategy to deal with office inaccessibility (where users work remotely). If possible, include multiple methods to access your data during a disaster scenario.
• Schedule a BCP day annually, ensure that your users are familiar with working remotely. Document takeaways and provide feedback to your IT staff on what worked and what didn’t.

Beyond environmental emergencies, it’s important to prepare for human-initiated disruptions. They can range from malicious cybersecurity attacks to something as simple as someone unplugging the wrong cable. You can’t anticipate every possible outage, but you can plan, stay organized and test.