Two IT Disaster Recovery Misses – and One Big Win
Five hours isn’t all that long in the grand scheme of things – but it might as well be an eternity during a data center outage. That’s more than enough time for a company to run up sizeable bills 
attempting to rectify the situation. It’s also plenty of time to give customers a negative experience that can take longer – and cost far more – to undo.
The airline industry has had its fair share of PR problems in recent months. Go back a little bit further to late last summer, and there were issues of a more technical variety. On August 8, a Delta Airlines operation center in Atlanta suffered an electrical surge and lost power. This outage took the computers needed to fly planes and book flights offline for roughly five hours. The immediate results of this were no doubt painful as Delta had to cancel nearly 1,000 flights on the day of the outage. That was followed by an additional 1,000 flights being canceled over the next two days as the airline worked to dig itself out. All told, the remediation efforts, the vouchers and refunds issued to upset customers, and more, cost the company $150 million – or $30 million for each hour of downtime.
While $150 million is an impressive total, it wasn’t even a record total last summer for airline IT issues. Southwest spent an estimated $177 million righting the ship following a data center outage in July that grounded flights for three days.
At least one insider chalked these outages up to an industry-wide failure to modernize. Systems have been built piecemeal over time and they run 24/7. Without the seamless integration, redundancy, automation, and other features found in newer backup and disaster recovery solutions, any company, in any industry, could be a sitting duck for a damaging outage of its own.
Internal threats to the data center
Data centers can be taken offline by a great many things that are out of IT’s hands – but sometimes it’s those hands that are responsible for the outage. In some instances this could be malicious, the work of, say, a disgruntled former admin. Other times, it’s simply an accident. Case in point, late last year, the global networking company Level 3 Communications dealt with an outage the company attributed to the always-dreaded “human error.”
The error occurred as provisioning work was being performed on Level 3’s network. An admin editing a routing table failed to include the necessary limitations on a configuration change, which snowballed into an outage. The erroneous entry was deleted to solve the problem, but that didn’t stop plenty of customers from voicing their outrage across social media.
In general, a simple typo such as this one can cause businesses a lot of grief. The risk of that happening only goes up as IT pros are tasked with handling larger and more complex workloads. That says nothing of what can happen when an incident occurs and IT is working at a break-neck pace to correct it. Whether it’s incorporated during day-to-day tasks or in a worst case scenario, automation can be a lifesaver.
Up and running but still unavailable
Even if a data center is powered up and running, that doesn’t mean everything is OK. Say, for example, if a business falls victim to a ransomware attack. This is a different kind of outage than the cases described above, but it’s no less devastating. In the event of a ransomware attack – should a business not have the appropriate backup and disaster recovery solutions in place – it will be faced with the choice of paying cyber crooks in order to regain access to encrypted data or saying goodbye to it, likely for good.
Last year, one of Arraya’s customers in the manufacturing industry fell victim to CryptoLocker, a particularly nasty form of ransomware. Rather than pick between paying up and losing its data, this manufacturer had another option. Instead, with Arraya’s help, the company was able to recover its data in under two minutes thanks to recently-made upgrades to its backup and recovery environment. Without those upgrades, the Arraya team, and the customer’s forward-thinking approach to backup and disaster recovery, this story likely would have had a very different, and costlier, ending for the customer.
Dive deeper into backup and disaster recovery
Arraya’s Data Center team is well-versed at deploying the tools and solutions needed to deliver wins such as the one in that last scenario. Start a conversation today by visiting: https://www.arrayasolutions.com//contact-us/. You can also reach out to them – or leave us a comment on this post – through social media: LinkedIn, Twitter, and Facebook.
Businesses interested in learning more about the latest disaster recovery and business continuity solutions can do so at the 3rd Annual Arraya Solutions Tech Summit on June 8. This free event features a full day of courses dedicated to the most valuable technologies and trickiest challenges encountered by modern IT pros, all led by our team of engineers to ensure a day that is truly “for techs, by techs.”
Included in the agenda will be a course entitled DR/BC Solutions that Guarantee Faster Recovery and Peace of Mind. During this session, attendees will explore leading business continuity and disaster recovery solutions from Dell EMC and VMware and how they help minimize downtime and remediation costs. They’ll also learn the differences between these two very distinct concepts that are regularly treated as one.
Arraya’s Tech Summit will be held at the Sheraton Valley Forge in King of Prussia, PA. To register, head to: https://events.arrayasolutions.com/.
