From Black Friday to Cyber Monday, the Thanksgiving weekend always poses big challenges for online retailers. How e-tailers cope with those challenges can have a major impact on their user experiences, online reputations, and ultimately sales, but every site deals with them differently.
To get a sense of how major sites did this year, we used New Relic Synthetics to run tests on 35 major Web sellers, pinging them from three different locations in the United States (San Francisco; Washington, DC; and Portland, Oregon) from the Wednesday before Thanksgiving right through Cyber Monday. We set up scripted browser tests that monitored the sign-in pages of all the retailers every 15 minutes from each location, collecting data points like load times, load sizes, and downtimes. Then we used a combination of New Relic Synthetics as well as New Relic Insights to drill down into the data to find out as much as we could about how the various sites did, what went wrong, and what went right.
We looked at the total number of outages and average page load times during the entire period, as well as several key data points. We assessed how page load size--particularly changes in page load size--affected performance. We looked at whether third-party page components were behind any of the slowdowns and other problems we uncovered. And we checked out how sites dealt with any outages they had.
Overall Performance
While most sites did relatively well, there were plenty of slowdowns and outages. Even Amazon popped up with 20 second delays for a while on Thanksgiving Day, while athletic shoe retailer Footlocker had response times approaching 30 seconds early on Cyber Monday.
The chart below shows the longest response times recorded for the 35 retailers on both Black Friday and Cyber Monday. These range all the way from 4 seconds to 54 seconds!
How much data are you sending?
Page load size can be a key factor when determining how fast a website will load. Pages heavy with large images and animations will likely be relatively slower to load. Best practices during peak traffic periods would be to limit the amount of data you're sending to provide as fast an experience as possible.
According to data from New Relic Synthetics, at least one giant retailer seems to have done just the opposite. Over the weekend, the site suffered from performance three to four times slower than normal, with spikes on Cyber Monday of up to 15 to 20 times normal wait times, along with some short outages. One possible reason for this could be that the site more than doubled its page load size from 1.8 MB to 3.9 MB.
The chart below shows how the vendor's page load size increased over the Thanksgiving weekend. On Wednesday, November 26, it started off with a load size of 1.8 MB and on both Black Friday and Cyber Monday, the load size was almost 4 MB.
Sure, it can be tempting to add a bunch of slick, high-resolution images to boost sale items, but the best-performing sites generally don't pump up their page load during the highest traffic periods. As an example of best practices, cookware retailer Williams-Sonoma improved site performance by cutting its page load size from 11 MB to a still-hefty 5 MB over the weekend by using fewer images. The key takeaway here? Less is more, especially on peak traffic days.
The chart below shows how Williams-Sonoma's page load dropped significantly during the Thanksgiving weekend, helping to ensure faster response times--below 6 seconds through most of the weekend. The first vendors' response times, meanwhile, topped 25 seconds.
Are third parties living up to their SLAs?
We put all the Synthetics data into New Relic Insights to understand how third-party services might be affecting the performance of the retailers, and here's what we uncovered:
The Average Response Time for all of the synthetic requests we made across all retailers was 75.8 ms.
The slowest response times relative to the average came from ad-tech and analytics companies:
- Google Tag Manager was used by about 25% of the retailers and had an average duration more than 7 times the average for all requests made
- Brightcove Video Player was nearly 8 times the average
- Google Maps API was 5 times the average
- Akamai was 4 times the average
This is a good time to check in on all those SLAs you have in place with third-party vendors and make sure they are complying--it would be a shame if third parties undo all the hard work you put into your website's performance. Of course, New Relic Synthetics is a great way to monitor those SLAs 24/7!
If you have to fall, fall with grace
The chart above captures all the errors that occurred across the multiple retailers we monitored through the Holiday weekend. And as you can see, bad stuff happens. When lags or downtimes cannot be avoided, you want to put on as good a face as possible. If your site goes down, are you displaying a graceful, branded, customer-centric error message or are you showing an ugly, generic "access denied" page?
New Relic Synthetics has the ability to capture the 'error screenshot' that customers see when they encounter a site that doesn't load--and those screenshots are all over the map. Despite the high-profile outages that Best Buy experienced during the long weekend, its failure screen was gracefully handled with a Holiday-themed "We're Sorry" message:
That's infinitely better than this screen, which is how one retailer greeted users who couldn't access the site:
As a best practice, be ready for anything so that if and when your site does go down, your customers see a message that acknowledges the problem, appreciates them, and lists a secondary call to action. Remember, you want your customers to return!
The list
Here's the complete list of the 35 sites we tracked. (Note that none of the companies were contacted for comment or participation.):
- Costco
- Amazon
- Apple
- Gap
- BestBuy
- Dell
- Macys
- Nordstrom
- HSN
- QVC
- PC Connection
- FootLocker
- Walmart
- Target
- Kohls
- Newegg
- Overstock
- Lowes
- Sears
- CDW
- Barnes and Noble
- Home Depot
- Uniqlo
- Zappos
- JCPenney
- Urban Outfitters
- Neiman Marcus
- Saks Fifth Avenue
- LL Bean
- Williams-Sonoma
- Crate and Barrel
- Toy R Us
- Staples
- Sony Electronics
- Etsy