User blog:Rupert Giles/Product Testing on Wikia

Hello Wikians! My name is Chris, aka Rupert_Giles (hello Buffy fans!), and I'm a Product Manager at Wikia. I work with a group of designers and engineers whose primary goal is to improve the reading and navigation experience across all communities. This will be my first post on Community Central since joining Wikia a few months ago and I'm really excited to get to know everyone.

I'm here today to talk to you about A/B testing and why it can make things even better for you and your communities. A/B testing, a big initiative for the Wikia Product team in 2014, is a way of optimizing site performance by delivering alternate versions of the same web page to different users at the same time. The tests will track the interactions of each group and determine which versions perform better.

For example, let's say that we're trying to get more users to create a wikia. We have a hypothesis that, by drawing more attention to the Start a Wiki button, we can increase the amount of wikias created. So, we run an A/B test that displays the current button to half our users (the control group) and the new button to the other half (the variation group). After a period of time, we see that the new button increased clicks by 45%. Armed with this data, we can feel confident in rolling out this change on a permanent basis to all of our users.

That's obviously a very simple example but there's so much more we can do. We can change the size of images, reconfigure the navigation, or even change the entire layout of an article page. Put simply, pretty much anything on Wikia can be altered and tested in some way.

So what does this mean for your community? First and foremost, you might see updates on different pages of your community. Some of the changes might be obvious like updating the placement of a module, but some might be very subtle like changing the algorithm on search results. Most of these tests won't last longer than a week. Some might only run for a few days or even a few hours. The nice thing about having a large audience is that we can attain statistical significance pretty quickly.

That said, if you do not want your community to take part in testing, please let us know so we can work around that. If, after discussing the matter on your wikia, your community would like to opt out, just send us a message via Special:Contact to let us know.

It is our sincere intention to be transparent about the kind of tests we perform. We may not say exactly what we are looking for while a test is running (so as not to skew the results) but we will definitely report back when everything is finished&mdash;what worked, what didn't, and what our next steps are. In fact, I would love to have a regular blog series on Community Central where Product Managers report back on their results.

And last but certainly not least, this means that our product decisions will be based not on a whim but on real world user behaviors. Before we make any major (or even minor) updates to the site, we want to be absolutely confident that those changes will have a positive impact on our readers and our editors. We want your communities to have confidence that these decisions aren't taking place based on our opinion alone. We want to show our work and demonstrate to you that these new features can bring real value to your readers and editors.

If you're interested in learning more about A/B testing, here are a few resources to get you started:


 * Conversion Rate Optimism blog
 * A Beginner’s Guide To A/B Testing
 * Wired’s “The A/B Test: Inside the Technology That’s Changing the Rules of Business”
 * A/B Testing: The Most Powerful Way to Turn Clicks Into Customers

I'll be happy to answer any questions you might have in the comments section. And if you have any suggestions for new features or tests please feel free to reach out to me on my wall. I'd love to hear your thoughts!