Request Information
1.877.459.4347

Request Information

or Call 1.877.459.4347

The monster costs of Frankenstorm Sandy

November 28, 2012 / Evolve IP

When Superstorm Sandy slammed into New Jersey and flooded lower Manhattan with a record-breaking tidal surge, many companies suddenly discovered they had woefully underestimated how much a storm like this could hurt them. The storm was large enough, for example, that some companies had both their headquarters’ data center and their backup emergency data center knocked out.

This “Frankenstorm” has exposed the hidden costs of disaster recovery and disaster planning — costs companies either didn’t think through in their planning, found too difficult to quantify, or dismissed as being intangible. But now, some of these intangible or difficult-to-quantify costs are very real and very large.

In the aftermath of the storm, Evolve IP has been highly focused on servicing customers that have been impacted. We are also talking to non-customers, trying to help restore their businesses because they didn’t plan properly. Many of these non-customers have fallen victim to the hidden costs of disaster recovery.

The unexpected costs these companies are incurring can be a guide for all of us as we assess our own disaster readiness. This is the second hurricane to hit the East Coast in a year, and the insurance industry tells us that, although these disasters might or might not be getting more frequent, they are certainly getting more expensive.

The top four hidden costs that we are seeing among the companies we are working with in the aftermath of Sandy are:

  1. Longer than expected recovery time for the restoration of production data from traditional tape backup
  2. Higher than expected recovery costs
  3. Lost revenue and sales
  4. Downtime per employee per hour

Some of these hidden costs might appear obvious on the surface, but they become troubling when you drill into them and see how damaging they really are. In this post, I’ll lay out how you can most accurately uncover and predict these hidden costs, and how working with a cloud provider like Evolve IP can help to minimize them.

First, you need to consider the two most important factors of disaster planning: First, What are your target Recovery Time Objectives (RTO) on an application by application basis? Second, what are your target Recovery Point Objectives (RPO) on an application by application basis? The recovery time is simply how long it takes you to get each application back up and running and into a usable production environment.

A typical target might be to have everything back online in 24 to 48 hours. For critical applications, the window is almost always much shorter, in some cases, less than one hour before it starts to have detrimental impacts to the business or revenues.

The recovery point deals with how much of your up-to-the-moment information you can afford to lose. In other words, what is the maximum age of the data? This also must be measured by application as the requirements will be very different for a transactional database or email than for a development web server or simple file storage.

For example, if your disaster recovery depends on restoring from a backup tape, then anything that happened after your most recent tape backup will be lost. The hidden costs of disaster recovery all relate back to those two decisions and the steps you take (or fail to take) in planning to achieve those objectives.

Hidden cost #1: Longer than expected recovery time for restoring production data from traditional tape backup

What it is: Companies are suddenly discovering that it is taking them longer to recover than they had planned. This is particularly true of those who are relying on tape backup. For disaster recovery purposes, tape backup has a number of built-in delays:

  • You have to get the tape from wherever it was being held off-site and to the location where it can be restored into production. This alone can cause a delay of 24 to 36 hours, depending on the location of the storage. It takes longer when the highways are closed.
  • You need a tape drive and auto-loader to recover the data off of the tapes. Some companies had both their primary and emergency tape drive systems knocked out by the storm, which meant they had to find, purchase, and install a tape drive and auto-loader at a off-site location before they could begin recovering the data. Add another 24 to 36 hours.
  • Tape drives can only go so fast. There is a reason tape backups are performed at night — their data transfer rates are much slower.The same is true when you try to recover from tape. What happens to your overall recovery time (RTO) if 12 hours of recovering data from tape has only restored 10 percent of your data due to the volume?
  • Tapes are almost always out of date. Tape backup is a point-in-time data protection strategy, so the restore point of tape backup is never up-to-the-minute or what many would consider near real-time. Anything that happened after the last backup is lost.

Why it matters: The amount of time it takes to recover data to restore critical applications into a production is critical. Data equals business, and 99 percent of companies can’t operate without their data. Any delay in recovery ends up having a cascading effect, causing a lot of other unplanned costs. Backing-up data to any media (tape, disk, etc.) is necessary to maintain historical copies of data and is best suited for recovering files, corrupt databases, or even full machine images. But they are only copies of a particular point-in-time. Backing-up is a data protection strategy; it is only a small part of meeting business continuity objectives in the face of a disaster. In a disaster, restoring from backups is a method of last resort, not the method of first resort.

How to address it: Compare the vagaries of on-premise, tape-based recovery to the cloud alternatives, where you have multiple data centers all interconnected with a fast network connections. The restoration process in the cloud is generally measured in minutes to a couple of hours, rather than days. If the connection is fast enough, the data can be mirrored or replicated from an on-site data center to a cloud data center in near real-time, so that the restore point provides a greater level of concurrency. With Evolve IP, these restoration processes are built into our service offering as either general DR procedures, data protection options, or as continuity service offerings to meet these types or requirements. We can simplify, and in some cases, automate many of the restoration processes for companies with more stringent needs.

Take into account: The potential risk that on-premise technology is obsolete. It’s not that you can’t keep using older equipment, but if you can’t read your backup tapes because your tape unit is destroyed and you can’t find a replacement without ordering one from the manufacturer, then this delay will extend the amount of time you are out of business.

Hidden cost #2: Higher than expected recovery costs

What it is: The costs of the third-party services you rely on to get yourself back in business.

Why it matters: When disaster strikes, a lot of the essential resources you use every day are no longer available to you. Even if they are, the daily resources are in place to operate under normal course of business. Most companies do not have excess resources sitting around waiting to jump into disaster mode. As a result, you often have to rely on third parties to supplement your needs, especially as it pertains to application work for critical system. Not only are you paying employees, but you are also now paying for the time of the third-party staff and outside vendors to get back in business. Those rates aren’t cheap to start with, and you are now paying overtime and perhaps even premium rates.

Another potential cost is the hike in insurance premiums the insurance companies will impose on those companies that don’t plan ahead. Up to 30 percent of the estimated $7 billion to $20 billion in claims expected for Sandy will be for business interruption claims from companies forced to shut down because of wind damage, flooding, or power outages.

How to address it: Have a good, fully informed disaster recovery plan in place that accounts for every contingency and every potential cost. For applications that are deemed critical to the business, ensure a continuity or avoidance strategy is employed. If you have 20 applications running, it is likely the business can continue to operate with perhaps three or four of those applications still in production. Trying to protect the entire business is a big undertaking. Focus on the applications that are most critical and figure out the best way to ensure the maximum downtime does not exceed your recovery time objectives.

Once you understand your true costs and your true risks, you can make good decisions about the resources you need. Even more importantly, begin thinking less in terms of disaster recovery and more in terms of business availability. It is probably far cheaper to ensure that your business remains available than it is to try to recover it once it has been taken down for an extended period of time by a disaster. With Evolve IP, disaster recovery and business continuity is a consideration that is factored into the available service options when the customer signs with us.

Take into account: Everything is more difficult and will take longer under disaster conditions. If the disaster is widespread, then everyone else will be competing for the same scarce resources.

Hidden cost #3: Downtime per employee per hour

What is it: The money it costs you to pay employees while the business is closed. The median salary for an employee in Manhattan is $50,000 per year ($23,000 for the U.S. median). If you have a company of 500 median people sitting around waiting for you, that’s over $12,000 an hour in employee downtime, at a time when the company isn’t generating revenue.

Why it matters: Even though an organization can’t do business, the employees are still on payroll. (You can force salaried employees to use their vacation days for the time you are closed, but that’s not good for employee relations — and just imagine how popular that will make the IT department.)

How to address it: Again, work closely with others in the organization to understand the true cost of employee downtime so that it is fully accounted for in your disaster recovery planning. Cloud, again, can help you avoid this issue altogether because the right cloud provider (with redundant data centers) and comprehensive recovery options can ensure that your business infrastructure stays available no matter what is happening to your company headquarters.

Take into account: Technologies, such as virtual desktops and cloud-based call centers, enable businesses to continue as if nothing has happened, all while employees stay safely at home with their families.

Hidden cost #4: Lost revenue and sales

What it is: This one’s obvious of course, but I’ll say it anyway: The money you can’t make because you can’t be open for business.

Why it matters: The lifeblood of a business is its revenue. Every hour that you are not open for business is costing your company money.

How to address it: These are not numbers that the IT department controls, so it’s often overlooked or underestimated when it comes to IT disaster planning. Work with your CFO and your finance department to get a solid understanding of the revenue impact per head of each hour of downtime. This can help you make more informed decisions about the appropriate recovery time and the investment required to ensure that you meet that deadline.

We are getting a lot of calls from potential customers who, before the storm, had been talking with us about financial justification for moving to the cloud. Now, when the disaster is causing them to lose revenue, they realize that being in the cloud would’ve avoided this loss of business. The financial justification of the cloud almost becomes a non-issue.

Take into account: When it comes to avoiding the hidden costs of disaster recovery, it’s better to be in the cloud than on-premise during a disaster like Sandy. But it’s also important for more than just a storm like Sandy; there are disaster risks every day in fire hazards, pipes bursting, rolling blackouts in heat waves during the summer, you name it.

In the cloud, as long as you have power and Internet, you can be up and running. Essentially, your disaster recovery plan becomes your business continuity, or “disaster avoidance,” plan. You should never be down, period.

The major benefits of being in the cloud during a disaster are availability, peace of mind, and cost savings. Or at least, those should be the benefits, if you have chosen your cloud provider carefully. Many cloud providers lack the redundancy in their facilities or have poor organization and control processes around things like disaster recovery and continuity. Anyone can put together a piece of marketing material claiming that they are “highly available.” Events like Sandy separate the rookies from the pros.

As for the businesses that were down for a week or more without power, perhaps flooded, with employees stuck at home, they lost over 100,000 of dollars per hour. Had they enabled their workforce to be mobile and centralized with their infrastructure and communications in a protected, cloud-based data center, they would have been in business making money, not losing it.

Categories: Cloud Computing
Recommended For You
The Evolve IP Compliance CloudTM

Compliance is a way to do business … not an afterthought when clients need it.

At Evolve IP we have a dedicated compliance and security practice and work with two of the world’s top 3rd-party compliance auditors, Grant Thornton and Ernst & Young, to enable customers to extend their compliance to our fully audited cloud. This focus allows us to deliver the documentation and assurances that other’s simply cannot including HIPAA / HITRUST, PCI-DSS (all 12 sections), SOC 2/3 and more. The Compliance CloudTM includes true client isolation, encryption in transit and at rest, private VLANs, firewalls and dozens of other security measures.

 
 
What Our Clients Say
  • “High-level technology, better compliance, cybersecurity, and communication capabilities are expected of all providers of financial services, regardless of size. With the rate of change and costs involved, it is difficult for any size credit union to keep pace. So when we were looking for a cloud technology partner, we emphasized the need to provide to our members with technology solutions that help them compete with larger banks and financial institutions, at a cost they can afford. Evolve IP offers all that, plus broad financial services industry experience and a pedigree of providing first-rate service to support growth and continually enhance the entire member experience.”

    David Frankil, Chairman of the NJCUL Services Corporation and President/CEO of the New Jersey Credit Union League

  • "The people that Evolve IP are more personable; you don't feel like there's necessarily a script when you're talking with them, they’re easy to understand, quick to get a hold of, and they follow through on what they say they're going to do."

    Watch Testimonial

  • "Evolve IP has been a vendor partner that has grown with us, that has helped us, and that you know stands by us and stands by their word."

    Watch Testimonial

  • "I was a bit of skeptical because I've been told that by sales guys before...but you've all came through on every one of their promises and not only that, but even throughout our subsequent years of working with us as partners, you’ve stepped up to the plate whenever we needed something and provided a helping hand"

    Watch Testimonial

  • "…we were assigned a project manager for our implementation they weren't just force-feeding one process down. they listened, they work with us, they adjusted schedules and they held our hand every step of the way…"

    Watch Testimonial

What the Experts Think

Our analyst-acclaimed solutions are built on a world-class, compliant architecture that leverages the blue-chip technologies organizations already know and trust.

Gartner

We deploy best-of-breed solutions including: Disaster Recovery, Contact Center, Unified Communications, DaaS, IaaS. Our services are analyst-acclaimed, vendor-validated, client recommended and award-winning.

HITRUST

Evolve IP is proud to have achieved the honor of being HITRUST CSF certified! Certification to the HITRUST Common Security Framework (CSF) affirms that all of Evolve IP’s cloud computing and cloud communications services adhere to the strictest security standards for electronic protected health information (PHI). The HITRUST security standard was developed by and for the healthcare industry as a means of going above and beyond the compliance requirements of HIPAA.

The HITRUST Common Security Framework (CSF) was developed to address the multitude of security, privacy and regulatory challenges facing healthcare organizations. The HITRUST CSF was developed by healthcare and IT professionals to provide an efficient and prescriptive framework for managing the security requirements inherent in HIPAA. HITRUST CSF rationalizes healthcare-relevant regulations and standards into a single overarching security framework. An important part of the “What is HITRUST” answer is understanding that the CSF is risk-based and compliance-based so that organizations can tailor the security control baselines and vendor management programs that they follow based on their specific organization type, size, systems, and regulatory requirements.

HIPPA

The Privacy regulations of the U.S. Health Insurance Portability and Accountability Act (HIPAA) require health care providers, organizations, and their business associates, develop and follow procedures that ensure the confidentiality and security of protected health information (PHI) when it is transferred, received, handled, or shared. This applies to all forms of PHI — paper, oral, and electronic, etc. Those who fail to adhere to HIPAA can suffer from huge fines climbing into the millions of dollars for major violations.

The Compliance Cloud™ fully enables covered entities and their business associates subject to HIPAA regulations to leverage a secure environment to process, maintain, and store protected health Information (PHI) featuring among other controls.

SSAW 16 Service Organization Control II (SOC 2)

Evolve IP has received an SSAE 16 SOC 2 Type II report on our internal controls relating to how we assess and address the potential risks associated with the security, availability, and confidentiality of not only the cloud-based services that we provide, but also our physical and logical infrastructure. Evolve IP utilizes the Certified Public Accounting firm of Grant Thornton to perform its annual audit and attestation in accordance with the Statements on Standards for Attestation Engagements No. 16 and the associated Trust Services Principles, as published by the AICPA, to evaluate the effectiveness of Evolve IP’s service organizations controls.

Forbes

While Forbes regularly features coverage and recognition about Evolve IP, they've most recently recognized Evolve IP as being the "Best Cloud Computing Companies And CEOs To Work For In 2017".  They've ranked Evolve IP in the Top 3 just behind Google and Microsoft in the Cloud Infrastructure classification.  (Feb 2017). Forbes  also recently recognizes Evolve IP for bringing Singer Equipment Corporation, a mainstream business based in PA, into the cloud by means of unified communication. (Sept  2017). Last year, Forbes recognized Evolve IP's survey of 1,080 executives citing that the number one reason to go to the cloud is the same reason that it is avoided. (Mar 2016).

Unified Communications Product of the Year

TMC and Internet Telephony Magazine have named Evolve IP’s unified communications platform as a 2017 Unified Communications Product of the Year Award winner. This marks the 6th time Evolve IP has been honored with this prestigious award and follows a series of product innovations that have allowed the company to rapidly expand its international coverage.

Evolve IP’s business collaboration tools and IP phone system dramatically improve employee productivity in the office and on the road with a Unified Communications as a Service (UCaaS) platform that fully integrates voice, video, instant messaging & presence (IM&P), desktop sharing, audio/web conferencing and more. The company also provides a sophisticated Web-based management portal, OSSmosis®, that allows administrators to easily configure system functions and quickly modify users without the need to reach out to a third party for changes.

Inc.

Inc. magazine has recognized Evolve IP in the 34th annual Inc. 500|5000, an exclusive listing of the nation's fastest-growing private companies. The list will be unveiled in the September issue of Inc.

The story of this year's Inc. 5000 is the story of great leadership. In an incredibly competitive business landscape, it takes something extraordinary to take your company to the top," says Inc. President and Editor-In-Chief Eric Schurenberg. "You have to remember that the average company on the Inc. 5000 grew nearly six-fold since 2012. Business owners don't achieve that kind of success by accident.

Payment Card Industry Data Security Standard (PCI DSS)

Evolve IP has achieved Payment Card Industry (PCI) Data Security Standard (DSS) compliance covering all 12 sections of the PCI DSS. The PCI data security standard is a comprehensive set of standards that require merchants and service providers that store, process, or transmit customer payment card data to adhere to strict information security controls and processes. It was created by the founding brands of the PCI Security Standards Council, which includes American Express, Discover Financial, JCB International, MasterCard Worldwide, and Visa Inc.

CSA STAR

Evolve IP is also a registered and participating member of the CSA Security, Trust & Assurance Registry (STAR). The CSA was formed to encourage transparency of security practices within cloud providers. It is a free, publicly accessible registry that documents the security controls provided by various cloud computing offerings, thereby helping users assess the security of cloud providers they currently use or are considering contracting with. CSA STAR is open to all cloud providers, and allows them to submit assessment reports that document compliance to CSA published best practices. The searchable registry will allow potential cloud customers to review the security practices of providers, accelerating their due diligence and leading to higher quality procurement experiences. CSA STAR represents a major leap forward in industry transparency, encouraging providers to make security capabilities a market differentiator.

Deloitte’s Technology Fast 500TM

Evolve IP has been ranked for the second consecutive year on Deloitte’s Technology Fast 500™, a ranking of the 500 fastest growing technology, media, telecommunications, life sciences and energy tech companies in North America – both public and private. Technology Fast 500 award winners are selected based on percentage fiscal year revenue growth from 2012 to 2015. The list is a veritable Who’s Who of technology that has included tech companies like Google, VMware and Facebook.

Technology Fast 500 provides a ranking of the fastest growing technology, life sciences, and energy the companies – both public and private – in North America. Technology Fast 500 award winners are selected based on percentage fiscal year revenue growth during the period from 2012 – 2016.

Red Herring

Red Herring has named Evolve IP as one of the Top 100 Companies in North America.  Red Herring’s Top 100 recognizes the leading and most promising private companies from around the world. Among the over 20 criteria used to analyze companies for the award, Evolve IP was noted for its financial performance, technological innovation, customer footprint, the DNA of its founders and addressable market.

Red Herring selects the award winners for North America from approximately 1,200 privately financed companies each year in the US and Canada. Since 1996, Red Herring has kept tabs on these organizations and its editors were among the first to recognize that companies such as Facebook, Twitter, Google, Yahoo, Skype, Salesforce.com, YouTube, Palo Alto Networks and eBay would change the way we live and work.

Entrepreneur

Evolve IP has been recognized as one of the “Best Entrepreneurial Companies in America” in Entrepreneur magazine’s Entrepreneur360™ Performance Index, a study involving a comprehensive analysis of private companies in America. Based on this study forged by Entrepreneur, Evolve IP is recognized as a company that exemplifies growth, not just in top and bottom line, but in sustainability and the ability to achieve lasting success.

According to Entrepreneur, after evaluating approximately 10,000 U.S. based firms, the team of editors and researchers behind the E360 Performance Index collected more than 250 pieces of data from the finalists, focusing on growth drivers and challenges, goal setting, resource allocations, and reward systems. The analysis uncovered a class of leading companies, including Evolve IP, whose continued success is largely based on superior value creation for their customers, building an adaptive learning culture, and aggressive geographic expansion—placing them amongst the most dynamic firms in America today.

Latest Press Releases

UC-One in Antarctica
July 8, 2019 / Evolve IP
TXi, a partner of Evolve IP, made news for the company bringing and using UC-One to Antarctica, while remaining connected, despite the remote isolation. TXi noted that the cost savings…
Evolve IP Revolutionizes DaaS Delivery with High Security, Flexibility and User Experience
June 25, 2019 / Evolve IP
WAYNE, Pa. - June 25, 2019 - Evolve IP®, The Cloud Strategy Company™, today announced the availability of the Evolve IP OneCloud Desktop, the next generation of its highly acclaimed Desktop-as-a-Service (DaaS) solution.…
Partner Day for thevoicefactory, now Evolve IP
June 18, 2019 / Evolve IP
UCToday's Rob Scott, highlights his experience at the annual partner event for thevoicefactory last week. Our very own Guy Fardone took the stage to discuss the exciting opportunities available in…
Unified Communications Transformation and Evolve IP
May 29, 2019 / Evolve IP
UCToday recognizes Evolve IP’s unique collaboration approach to unified communication. Our very own Scott Kinka offers a look into what we're currently up to at Evolve IP, M&A growth direction,…
View More
close

Contact Us

or Call 1.877.459.4347