Our users are very active on the site but they weren't using a key feature we had implemented: saving projects, teachers, and schools as favorites. We took our cue from popular apps like Instagram and decided to run a test that changed our favorites icon from a star to a heart. We developed a sprite with a heart icon that had three color variations and two sizes depending on what part of the site was displaying it. The shades of color depended on the user state, while the sizing was dependent on if it was aligned with headline text or not.
This change would affect most sections of the site, so we had to test extensively prior to releasing the changes into production. We had to examine several different states of users, from brand new visitors to those that were logged in and returning users. Each user state had several areas that could be changed, from their internal dashboard, to viewing project pages, to how the homepage was displayed. Since there was such a large swath of the site that was being affected, I decided to do a full audit of the site and created a checklist of tests to pass before launching the test.
A/B Testing, Data Analytics and Presentation, Blog write-up
It's exhilarating to implement a change site wide and see the number of user impressions that it's receiving. Even better is seeing the test variation blow away the results from the original. We actually doubled the amount of users saving projects as favorites in our hearts variation!