October 11, 2016 / Peter Eisengrein In the wake of Level3’s massive outage, which had rippling effects across all virtually all carriers in North America, questions persist about VoIP redundancy and alternate routing of calls. Even well-networked carriers experienced over an hour of downtime, while Level3 was completely unreachable; even their website was down. So how could a single carrier cause such a headache for so many?Simply put, the answer lies in what can and cannot be controlled.In the world of telephony, any service provider worth their salt will have more than one or two carriers with whom they interconnect. And having multiple routes means that you can control to which carriers you send traffic. This is usually a fairly simple task that can be accomplished in a few minutes. But did you catch the operative word?To.It was subtle, but those two letters makes a world of difference. So, here it is again: having multiple routes means that you can control to which carriers you send traffic.Figure A: Normal Outbound RoutingA service provider – any service provider – can control their outbound routes, the “to”. A call is made, the list of possible routes considered, and a route selection is made and the outbound call is set up, all in a couple microseconds. The time it takes to verify the source of the problem as being with Carrier X usually takes longer than the act of removing Carrier X from your route tables. Depending on the nature of the problem the phone switch may automatically re-route to the next best route choice without any manual intervention.Figure B: Outbound Overflow To Alternate Route So, this leaves us with three lingering questions:
if it is so easy to route around a problem carrier, why can’t it be done for inbound calls;
if you’ve routed around the problem carrier why am I still having problems completing some calls;
if your carrier is dependent upon other carriers, why shouldn’t you just use one of them instead?
All fair questions. Let’s look at them individually.First, why can’t we control our inbound routes? The short answer is, because the call hasn’t reached us yet so we have no control over it. The longer answer is, just as we can control outbound routes to Carrier X, an inbound call to us is an outbound call from another carrier and they control how to get the call to us. If you had only a single route (trunk group) with that carrier and they could not complete the call for some reason then the call would fail. If you have multiple trunk groups, they may attempt other routes to get to you. However, if their network is having issues they may fail the call altogether. The next thing to consider is that the Public Switched Telephone Network was designed so that every local number can only belong to a single provider and therefore that single provider alone can complete the call (we’ll discuss toll free calls in a moment). What that means is that virtually every local number that is “owned” by Level3 likely had issues during the outage.Figure C: Inbound Failure Never Reaching The Terminating CarrierSecond, if Level3 was removed from routing why were some outbound calls still having problems? This is typically because the PSTN is a mesh of carriers who all interconnect and the overflow carrier attempts to send the call to the carrier having a problem; this probably wasn’t much of an issue for most carriers on October 4, 2016 because most, if not all, were aware of the magnitude of Level3’s issue.Figure D: Impacted Overflow RoutesThird, while Tier 1 carriers like Verizon and Level3 and Competitive Local Exchange Carrier (CLECs) may provide excellent products over their massive network, they are their own single point of failure and cannot offer multiple carriers to serve their customers’ needs.So, what can be done to minimize the impact when carrier outages occur? First, ensure that your service provider has ample connectivity to the PSTN. If they don’t then you’re probably not using the right provider. Next, if you have multiple locations, ask that the numbers of the locations be split among different carriers. While this doesn’t eliminate such outages, it can help by minimizing the impact when they occur. Last, consider using a Toll Free number instead of a local number for your inbound calls. Toll Free numbers are unlike local numbers and do not have a single “owner” but instead can be handled more like outbound calls where routing and multiple carriers can be controlled by the Responsible Organization (“RespOrg”).Evolve IP stands out among VoIP providers by having scores of routes to over 10 different PSTN carriers enabling us to adjust our routes in the ever-changing network landscape, minimizing the effects of a mammoth outage. For more information about how Evolve IP can help your business achieve maximum availability please contact firstname.lastname@example.org.Categories: Unified Communications
About Peter Eisengrein
Peter Eisengrein is a founding partner at Evolve IP and is responsible for the UCaaS and CCaaS networks. He has spent the last 25 years surrounding himself with talented experts who build and manage world-class voice networks. Outside of work, Pete enjoys time with his wife and kids, exuberant dog, and acrimonious cat, and has a black belt in Shotokan karate, is an amateur musician, photographer, and artist.
Compliance is a way to do business … not an afterthought when clients need it.
At Evolve IP we have a dedicated compliance and security practice and work with two of the world’s top 3rd-party compliance auditors, Grant Thornton and Ernst & Young, to enable customers to extend their compliance to our fully audited cloud. This focus allows us to deliver the documentation and assurances that other’s simply cannot including HIPAA / HITRUST, PCI-DSS (all 12 sections), SOC 2/3 and more. The Compliance CloudTM includes true client isolation, encryption in transit and at rest, private VLANs, firewalls and dozens of other security measures.
What Our Clients Say
"Yesterday was, perhaps, my busiest day of client interaction either by phone or email since I have been a PM, and I don’t think any of my clients knew I was working from home unless I told them. I was also able to do trades behind the scene and interact with my team. So, for me, the technology has been working great. As an old guy, I am constantly dazzled by technology in general, but being able to do this stuff from home is amazing!"
James C. Hunter, CFA, CFP, AIF, Senior Portfolio Manager, Principal
"Hey, IT people, As I’m working away in my home office, I just wanted to say thanks to you for all you’re doing, and have done in the past, to make it possible for us to run our company virtually. Not many of us JICers have jobs that everyone in the firm sees and could stop us from doing business. But you have this job, and do it well. Thank you for having the foresight and wisdom to get us in a position to succeed in a pandemic! You’re awesome."
Michael D. Barnes, Esq., CTFA President, Principal
“That’s the type of proposition I like to bring to a Board of Directors. When I can say, ‘we can get everything new, be completely redundant, it can meet all of our needs and oh, by the way, we are going to save over $300,000 a year.’ It makes it easy for me to sell!”
"The people that Evolve IP are more personable; you don't feel like there's necessarily a script when you're talking with them, they’re easy to understand, quick to get a hold of, and they follow through on what they say they're going to do."
Our analyst-acclaimed solutions are built on a world-class, compliant architecture that leverages the blue-chip technologies organizations already know and trust.
We deploy best-of-breed solutions including: Disaster Recovery, Contact Center, Unified Communications, DaaS, IaaS. Our services are analyst-acclaimed, vendor-validated, client recommended and award-winning.
Evolve IP is proud to have achieved the honor of being HITRUST CSF certified! Certification to the HITRUST Common Security Framework (CSF) affirms that all of Evolve IP’s cloud computing and cloud communications services adhere to the strictest security standards for electronic protected health information (PHI). The HITRUST security standard was developed by and for the healthcare industry as a means of going above and beyond the compliance requirements of HIPAA.
The HITRUST Common Security Framework (CSF) was developed to address the multitude of security, privacy and regulatory challenges facing healthcare organizations. The HITRUST CSF was developed by healthcare and IT professionals to provide an efficient and prescriptive framework for managing the security requirements inherent in HIPAA. HITRUST CSF rationalizes healthcare-relevant regulations and standards into a single overarching security framework. An important part of the “What is HITRUST” answer is understanding that the CSF is risk-based and compliance-based so that organizations can tailor the security control baselines and vendor management programs that they follow based on their specific organization type, size, systems, and regulatory requirements.
The Privacy regulations of the U.S. Health Insurance Portability and Accountability Act (HIPAA) require health care providers, organizations, and their business associates, develop and follow procedures that ensure the confidentiality and security of protected health information (PHI) when it is transferred, received, handled, or shared. This applies to all forms of PHI — paper, oral, and electronic, etc. Those who fail to adhere to HIPAA can suffer from huge fines climbing into the millions of dollars for major violations.
The Compliance Cloud™ fully enables covered entities and their business associates subject to HIPAA regulations to leverage a secure environment to process, maintain, and store protected health Information (PHI) featuring among other controls.
SSAW 16 Service Organization Control II (SOC 2)
Evolve IP has received an SSAE 16 SOC 2 Type II report on our internal controls relating to how we assess and address the potential risks associated with the security, availability, and confidentiality of not only the cloud-based services that we provide, but also our physical and logical infrastructure. Evolve IP utilizes the Certified Public Accounting firm of Grant Thornton to perform its annual audit and attestation in accordance with the Statements on Standards for Attestation Engagements No. 16 and the associated Trust Services Principles, as published by the AICPA, to evaluate the effectiveness of Evolve IP’s service organizations controls.
While Forbes regularly features coverage and recognition about Evolve IP, they've most recently recognized Evolve IP as being the "Best Cloud Computing Companies And CEOs To Work For In 2017". They've ranked Evolve IP in the Top 3 just behind Google and Microsoft in the Cloud Infrastructure classification. (Feb 2017). Forbes also recently recognizes Evolve IP for bringing Singer Equipment Corporation, a mainstream business based in PA, into the cloud by means of unified communication. (Sept 2017). Last year, Forbes recognized Evolve IP's survey of 1,080 executives citing that the number one reason to go to the cloud is the same reason that it is avoided. (Mar 2016).
Unified Communications Product of the Year
TMC and Internet Telephony Magazine have named Evolve IP’s unified communications platform as a 2017 Unified Communications Product of the Year Award winner. This marks the 6th time Evolve IP has been honored with this prestigious award and follows a series of product innovations that have allowed the company to rapidly expand its international coverage.
Evolve IP’s business collaboration tools and IP phone system dramatically improve employee productivity in the office and on the road with a Unified Communications as a Service (UCaaS) platform that fully integrates voice, video, instant messaging & presence (IM&P), desktop sharing, audio/web conferencing and more. The company also provides a sophisticated Web-based management portal, OSSmosis®, that allows administrators to easily configure system functions and quickly modify users without the need to reach out to a third party for changes.
Inc. magazine has recognized Evolve IP in the 34th annual Inc. 500|5000, an exclusive listing of the nation's fastest-growing private companies. The list will be unveiled in the September issue of Inc.
The story of this year's Inc. 5000 is the story of great leadership. In an incredibly competitive business landscape, it takes something extraordinary to take your company to the top," says Inc. President and Editor-In-Chief Eric Schurenberg. "You have to remember that the average company on the Inc. 5000 grew nearly six-fold since 2012. Business owners don't achieve that kind of success by accident.
Payment Card Industry Data Security Standard (PCI DSS)
Evolve IP has achieved Payment Card Industry (PCI) Data Security Standard (DSS) compliance covering all 12 sections of the PCI DSS. The PCI data security standard is a comprehensive set of standards that require merchants and service providers that store, process, or transmit customer payment card data to adhere to strict information security controls and processes. It was created by the founding brands of the PCI Security Standards Council, which includes American Express, Discover Financial, JCB International, MasterCard Worldwide, and Visa Inc.
Evolve IP is also a registered and participating member of the CSA Security, Trust & Assurance Registry (STAR). The CSA was formed to encourage transparency of security practices within cloud providers. It is a free, publicly accessible registry that documents the security controls provided by various cloud computing offerings, thereby helping users assess the security of cloud providers they currently use or are considering contracting with. CSA STAR is open to all cloud providers, and allows them to submit assessment reports that document compliance to CSA published best practices. The searchable registry will allow potential cloud customers to review the security practices of providers, accelerating their due diligence and leading to higher quality procurement experiences. CSA STAR represents a major leap forward in industry transparency, encouraging providers to make security capabilities a market differentiator.
Deloitte’s Technology Fast 500TM
Evolve IP has been ranked for the second consecutive year on Deloitte’s Technology Fast 500™, a ranking of the 500 fastest growing technology, media, telecommunications, life sciences and energy tech companies in North America – both public and private. Technology Fast 500 award winners are selected based on percentage fiscal year revenue growth from 2012 to 2015. The list is a veritable Who’s Who of technology that has included tech companies like Google, VMware and Facebook.
Technology Fast 500 provides a ranking of the fastest growing technology, life sciences, and energy the companies – both public and private – in North America. Technology Fast 500 award winners are selected based on percentage fiscal year revenue growth during the period from 2012 – 2016.
Red Herring has named Evolve IP as one of the Top 100 Companies in North America. Red Herring’s Top 100 recognizes the leading and most promising private companies from around the world. Among the over 20 criteria used to analyze companies for the award, Evolve IP was noted for its financial performance, technological innovation, customer footprint, the DNA of its founders and addressable market.
Red Herring selects the award winners for North America from approximately 1,200 privately financed companies each year in the US and Canada. Since 1996, Red Herring has kept tabs on these organizations and its editors were among the first to recognize that companies such as Facebook, Twitter, Google, Yahoo, Skype, Salesforce.com, YouTube, Palo Alto Networks and eBay would change the way we live and work.
Evolve IP has been recognized as one of the “Best Entrepreneurial Companies in America” in Entrepreneur magazine’s Entrepreneur360™ Performance Index, a study involving a comprehensive analysis of private companies in America. Based on this study forged by Entrepreneur, Evolve IP is recognized as a company that exemplifies growth, not just in top and bottom line, but in sustainability and the ability to achieve lasting success.
According to Entrepreneur, after evaluating approximately 10,000 U.S. based firms, the team of editors and researchers behind the E360 Performance Index collected more than 250 pieces of data from the finalists, focusing on growth drivers and challenges, goal setting, resource allocations, and reward systems. The analysis uncovered a class of leading companies, including Evolve IP, whose continued success is largely based on superior value creation for their customers, building an adaptive learning culture, and aggressive geographic expansion—placing them amongst the most dynamic firms in America today.
Evolve IP®, the world's leading provider of Work Anywhere™ solutions; today announced that it has integrated the omnichannel Evolve Contact Suite with Microsoft Teams and the company's virtual Workspaces...