More LookBook Test

This was a phrase we heard from Helena, our test engineering intern, time and time again when Phoenix ticket were marked unreproducible. The developers thought they could use IE10 in IE8 mode to reproduce the exact behavior we were reporting.

It all began in July 2013, when a couple of developers began working on a performance improvement project. The goal was to reduce entry page load times from ~13 seconds to under 5. From day one it was obvious that the project be called 'Phoenix' as a new front end code was being built from scratch on the ashes of the previous codebase to make it more extensible.

Our challenge in establishing Phoenix's quality was that we had numerous OS/browser and template combinations on which we had to ensure every little quirk we had to work.

Initially, a couple of us started exploratory testing and were reporting about 20 issues a week mostly about missing styling. Later we pulled test engineers from other projects and all eight members of our team focused solely on Phoenix. We dove into all modules and adopted a more methodical divide and conquer testing approach. We were then opening about 65 new bug tickets a week. At this point more developers were added to catch up with our bug reports.

One of our favorite series of tickets was about reports related to missing custom banners. We had hundreds of those banners and many weren't showing up on Phoenix. Those tickets kept one developer busy for two weeks analyzing 10 large SQL queries that used to take 3 seconds without rendering the right data to 2 new queries that work accurately in 60ms.