How To Protect Your Website From Another Google Outage
What did the Google outage look like?
Yottaa actively monitors the performance of Google services across the 1500+ eCommerce sites that use our eCommerce acceleration cloud technology. We do this so we can quickly identify performance anomalies (like this one), alert customers, and ensure sequencing rules are working correctly so customers don’t experience an issue.
Figure 1: Last Byte Time (ms) – Google advertising services
Figure 2: Last Byte Time (ms) – Google API
Interestingly, we also saw an impact on one of the Google analytics tags (Classic). But it wasn’t what you might expect. Last byte times actually decreased during the outage.
Figure 3: Last Byte Time (ms) – apis.google.com, googleapis.com
Given that the Classic tag loaded faster during this window than it did the entire week prior, I imagine they had a “kill switch” in place. In other words, rather than letting a severity one problem slow websites when the tag doesn’t respond, Google probably responded immediately with a 500 error (i.e. a “fail fast” strategy).
Although website teams won’t see analytics data for that 4 hour window, their shoppers weren’t affected by the outage. Definitely a preferable outcome.
How often do 3rd party technology outages impact site performance?
Google reported that the outage was due to a planned configuration change in the Google network. Unfortunately, the change was applied to a larger number of regions than intended. This caused network congestion and delays when too much traffic was rerouted through the remaining servers.
What does this say about your site’s risk exposure from 3rd party vendors? Google has a top notch network team that was able to minimize the impact for many services (Shopify the big exception). But delays like this will often last for last hours and occur at the worst times. The IBM / Coremetrics issue on Black Friday in 2018 is a recent example. If this can happen to Google, IBM and Amazon all within the last two years, it can happen to any vendor service used on your site.
How to prevent 3rd party website technology problems
The best way to prevent performance problems from 3rd party services is to assume they could happen at any moment. Given that, here are steps you can take to ensure a similar outage doesn’t bring down your site this year.
1. Have a Network vs Single Provider Backup Plan – Relying on a single cloud provider is a mistake. Make sure your site and all your mission critical vendors take advantage of multiple networks. That way an outage in one won’t disrupt all traffic to your site.
4. Have an Anomaly Resolution Plan – Make sure you have clear action plan to resolve performance anomalies when they arise. Because even a series of small anomalies, when left unaddressed, can add up to impact your revenue. Over time, strive to detect and resolve smaller and smaller anomalies, so you can deliver a more consistent site experience for all your site visitors.
All eCommerce sites need to plan for performance anomalies
The Google outage exposes the interdependencies within the eCommerce ecosystem. A problem with one vendor can ripple through and impact sites that don’t even contract with that vendor directly. Make sure you have a plan and technology to manage 3rd party vendor anomalies before they take down your site. Because you don’t want to be that one eCommerce team sifting through performance alerts and log files on a Sunday afternoon while your better prepared competitors sail by you.