Case Study

Foreo

Drives Millions of Dollars of Revenue During the Holiday Seasons with Tag1’s Technical Architecture and Leadership

Overview

Tag1 was engaged to provide emergency services for the Foreo.com website which was crashing under heavy load during Black Friday sales. Tag1 quickly got the site back online, performing faster than ever - by Cyber Monday the site could handle 3x the number of requests per second, with response times that were over 30x faster, improving page load times by over 3,000% yielding sub 1.5 second load times under peak traffic.

After addressing the emergency situation, Tag1 then worked with Foreo to improve infrastructure and application architecture, as well as the development processes, including automation of change control with DevOps to ensure problems didn’t resurface in the future.

“Working together non-stop over the next 48 hours, Tag1 collaborated closely with the Foreo team to quickly address the emergency situation, diagnosing and remediating all problems to ensure a successful Cyber Monday that would bring even more traffic.”

Situation

Triage for the short term and prevention for the long term.

Within hours of receiving the request for emergency help, Tag1 deployed a team of infrastructure and application experts to begin reviewing performance issues on Foreo.com. Working together non-stop over the next 48 hours, Tag1 collaborated closely with the Foreo team to quickly address the emergency situation, diagnosing and remediating all problems to ensure a successful Cyber Monday that would bring even more traffic.

On Black Friday, November 25, 2016, the Foreo.com website began crashing under heavy load at the peak of Black Friday sales, on the cusp of the most lucrative online shopping period of the year. Having a mere 48 hours to resolve the performance issues they were facing before Cyber Monday, Foreo was at a loss as to prevent a public relations nightmare and the potential evaporation of millions in revenue from holiday shoppers faced with prohibitively lengthy load times.

After the emergency situation was addressed, Foreo needed to position themselves for long term success and ensure their sites remained online and highly performant under extreme loads in the future.

Risk Mitigation

  • Outside of the automated per-release tests, even more aggressive and in depth load testing and tuning was scheduled leading up to the biggest sales events.
  • Readiness fire drills and chaos testing was performed to ensure the teams responsiveness to a variety of problems under pressure.
  • Advanced real-time monitoring and alerting systems were put in place to catch small problems before they become revenue impacting.
  • 24x7 on-call support teams were established around critical events, with staff ready to go at a moments notice to get ahead of any problem that might arise.

With Tag1’s guidance, and a proactive approach to performance, Foreo hasn't had problems since. Their infrastructure and web applications can handle massive traffic spikes, at any time, without any impact, ensuring maximum conversion rates on sales at all times.

Challenge

Surviving and Learning from an Even Larger Traffic Spike

Shortly after election day, on November 16, 2016, Andrew D. Romero, executive director of the ACLU, appeared on The Rachel Maddow Show, and donations began pouring in. As tens of thousands of users attempted to sign up for mailing lists, send letters to elected officials, and donate to the ACLU—at a rate of 500 new members per minute—database queries that would normally need milliseconds under typical traffic loads began to take upwards of 10 seconds or even more than a minute to complete. Many users faced ‘503 Service Unavailable’ and other 5xx errors as the website struggled to stay online. It was a significant missed opportunity to capture new members, volunteers, and donors. Recognizing the potential ramifications on the organization’s financial goals, the ACLU called in Tag1’s Technical Architecture and Leadership, a service Tag1 provides to help clients solve difficult problems, to perform emergency support and rescue work to get the ACLU Action website back online as quickly as possible.

Supporting an 85-fold increase in donations and user traffic required immediate and expert insights to remedy adverse financial impacts caused by site downtime. Within 48 hours, Tag1 rapidly audited and scaled the full technology stack and infrastructure to bring the ACLU Action platform back online and lay the foundation for long-term scalability, enabling the ACLU to continue supporting civil liberties at critical junctures in American history.

Solution

Triage for the short term and prevention for the long term

Within hours of receiving the request for emergency help, Tag1 deployed a team of infrastructure and application experts to begin reviewing performance issues on Foreo.com. Working together non-stop over the next 48 hours, Tag1 collaborated closely with the Foreo team to quickly address the emergency situation, diagnosing and remediating all problems to ensure a successful Cyber Monday that would bring even more traffic.

Performance tuning is a methodical process of triage and iteration. You find the biggest problem that you can quickly solve - the biggest bang for your buck - and fix it. Changing too much at once can cause more problems than it solves. Then you need to test the results of your work to confirm it had the impact you expected before you identify the next biggest problem (which may have changed from your previous assessment based on the change you just made). Knowing where to look and how to work across the entire stack (infrastructure, caching, databases, code, etc.) takes years and years of experience, and executing quickly under the stress and pressure of money being lost by the minute, on less and less sleep, while keeping a cool head is something few can do well.

After the emergency was resolved the teams worked together to ensure future success by auditing and improving application and infrastructure architecture, enhancing development processes & workflows, and implementing advanced DevOps change control that incorporated automated code reviews as well as load tests around each release. Code bases and websites are always under development, and Foreo needs to ensure their site is always ready for a spike in traffic, and that any update or change doesn’t impact performance.

In addition, risk mitigation strategies were established to minimize the impact of any problems that might surface. Outside of the automated per-release tests, even more aggressive and in depth load testing and tuning was scheduled leading up to the biggest sales events. Readiness fire drills and chaos testing was performed to ensure the teams responsiveness to a variety of problems under pressure. Advanced real-time monitoring and alerting systems were put in place to catch small problems before they become revenue impacting 24x7 on-call support teams were established around critical events, with staff ready to go at a moments notice to get ahead of any problem that might arise.

With Tag1’s guidance, and a proactive approach to performance, Foreo hasn't had problems since. Their infrastructure and web applications can handle massive traffic spikes, at any time, without any impact, ensuring maximum conversion rates on sales at all times.

We were able to mitigate the massive traffic spike originating from Anthony Romero’s interview thanks to a creative yet simple solution. Together with the ACLU, we set up a static HTML donation page and delivered it through a content delivery network (CDN), temporarily bypassing the ACLU’s servers. This helped alleviate the initial financial impact as donations were able to continue unhindered, allowing the team to shift focus to remedying the underlying foundational problems that caused the platform to fall over in the first place.

With the help of available data from New Relic, Pantheon, database logs, and other systems, Tag1 developed a strategy of quick query optimizations and indexing changes on the live database, adding in improved caching mechanisms, to reduce server load and bring the site back online. Once these optimizations were online, the initial page response time decreased by 57% from 1400 milliseconds to 650 milliseconds.

Foreo

Foreo Drives Millions of Dollars of Revenue

Foreo is a Swedish multinational beauty brand specializing in high-end beauty and wellbeing products such as the iconic Luna facial cleanser. In a mere six years, Foreo has grown from just two employees in Sweden to over 3,000 employees around the world, shipping over 20 million Foreo products in the process.

View Live Site.

Work With Tag1

Be in Capable Digital Hands

Gain confidence and clarity with expert guidance that turns complex technical decisions into clear, informed choices—without the uncertainty.