Forum:A/B testing

Hi all, just a heads up for the regulars here. We are currently doing some A/B testing on some skin elements. As well as all the initial data and feedback from the beta stages of the skin, we are now getting a lot of larger scale live-use data. That's being used to look at specific areas where we think we can make changes to improve usability. For example, we are looking at what effect shrinking the top header has on the use of other links in that area -- including the wiki's navigation.

Each change will only be seen by a small percentage of visitors, and most of those will be casual readers who won't know that what they are seeing is not the usual. The testing is based on statistical analysis of the changes, so we will actually get the best results from those who don't notice a difference as such, just react to it naturally :) But, because the test group is selected randomly from logged out visitors, some of the regulars may get added to the group before they log in.

So if there are questions in the forums about differences in in the skin, please keep in mind that it may be part of this testing -- especially if you don't see the same change. Thanks -- Sannse (help forum | blog) 18:51, November 18, 2010 (UTC)


 * Somewhat heartening to hear, but it doesn't sound like the testing addresses any of the major complaints I've seen in the beta or afterwards. "Slow and steady wins the race..." except on the internet.
 * As a reminder, here are the high level things we keep hearing complaints about:
 * Inadequate help pages that have well under 50% the amount of info as Monaco (possibly around 10%).
 * Fixed width apparently optimized around a 1024 px width window.
 * Intrusive, ever-present side rail that, combined with fixed width, crowds many content pages.
 * Too limited navigation menus. Only 4 and only 7 deep with no sub-menus.
 * Wikia top bar with a fixed set of wiki categories that are often not appropriate to the wiki, but appear all the time.
 * Floating bottom tool bar that is mostly wasted space and is poorly documented.
 * Wiki activity button that can't be altered and goes to changes activity page that is much less useful to experienced users than Special:RecentChanges.
 * Misleading image attribution.
 * Answers wikis don't work with new skin, so transition from an Oasis-based wiki is strange in the least.
 * Missing functionality that was previously provided by Widgets.
 * Lack of access to history on many pages.
 * These are off the top of my head. Please reply, anyone with additional issues.
 * -- Fandyllic  (talk &middot; contr) 3:43 PM PST 18 Nov 2010


 * Agree Fandyllic, sound a fair list for starters of the main points that are 'annoying' experienced editors like me and a number of the points also appear to be annoying 'newer' editors sufficiently to bring then to central and/or leave wikia all together.


 * I'll add these
 * Generally poor access to editing tools that previously were ine click away in tabs or in the toolbox are niw hidden/not accessible from some pages.
 * Loss of the wikis individual identity due to loss of the logo to the dumbed down wordmark idea.
 * Themedesigner - (Colour selector in reality) is poor and misnamed calling it designer.
 * preview giving a squashed view rather than a WYSIWYG version (note i dont use the useless RTE that messes up templates Still)
 * Blog comments not previewing, and being hard to see when updated since last visit etc - Forum works better
 * Predictive search appears very glitchy & the lack if a red link to create a page from if nothing is found, thus makes adding pages hard work as some pages dont have a create a new page link !


 * For some wikis the layout does look ok but the 2/3 1/3 split does not work with tables and infoboxes on technical wikis as once you drop below the rail the huge blank space looks bad & info boes squash the content. I'm sure there are others to be add later.


 * We may be getting used to it but still dont like it, so am editing far less & mainly following the debate/complaints/blogs not adding new content and cleaning up the blank articles from the earlier update that makes adding articles easy but they have no content 9 times in 10. So ultimately Wikia is not growing at its previous growth as my wiki should be past 5000 articles by now but has stagnated since July & some users have left due to the new skins disruption, and i'm sure this is also the case with may other wikis. - BulldozerD11 01:19, November 19, 2010 (UTC)


 * We are currently doing some A/B testing on some skin elements.
 * Hum... A/B testing is not what is needed, beta testing is, or better, was needed.
 * My list:
 * Inconsistent add page. In some pages it's there, in some pages it's not. In some pages it has one location, in some has another location. It even changes name!
 * My tools is also another example of inconsistency making it useless. In many cases it is covering important content, for example the result of a search done in the browser (firefox in my case). During the beta testing didn't anyone use a browser and pressed CTRL+F to find something on a page!? And why isn't it on the editing page? I need it just as much when I'm editing an article.
 * The summary box at the bottom of the edit pane has varying/erratic behavior. It stays in place (like it should) during the first edit but it gets stuck to the bottom of the screen after the first time I use the preview feature. That makes it even more annoying and intrusive than the My tools.
 * Images box. A disaster from the beginning. In "my" wiki it started with 2 images of blobs of color although my wiki is not about abstract painting, which were actually the icons (that's all my wiki has) with dimensions 16x16 zoomed in to the point that they are unrecognizable blurs of colors. They "fixed" it and now there is an image with clouds and Anyone can help to make this wiki look amazing Add a photo!. /sarcasm It works great now. It's a great stamp marking it as a newbie wiki that has 125photos on this wiki above the clouds image. On top of that, I impatiently and eagerly wait for the first image that will be shown there (NOT!!!). /sigh I'm sure that image will be someone's avatar completely unrelated to the wiki.

See also Forum:Complaining_about_new_look for another reason.

Sorry for the slight rant tone.

12:10, November 19, 2010 (UTC)

I'll do some combining... I know I'm repeating stuff that you've already heard, but hopefully having it all together will help.


 * Help pages: These were not used much pre-Oasis (from visit numbers, click tracking) so haven't been a priority. That said, we definitely want to improve them so that they are used more. We worked on the most relevant pages first and are continuing down the line now.
 * Fixed width/right side rail/top bar/visability of the Wiki identity: these will all be part of the A/B testing (we already have the feedback and usage data to go with that)
 * Tool bar/theme designer/image box: the first versions of these have generally had good reviews, we can't please everyone, but hopefully as new versions come are developed they will please more people.
 * Variations in add page/history/toolbar display: This feedback is with the design team
 * Navigation menus/Wiki activity button/image attribution/removal of links: We've talked a lot about the reasons for these... but as with everything, we'll continue to review them.
 * Widgets: now work as embedable elements on the new look (including the shoutbox)
 * Answers wikis: I agree it's a strange transition, I believe we'll be looking at them again after the holidays
 * Preview issues/blog comments/predictive search/tables display improvements/toolbar over content/summary box behaviour - I believe all these are in the feedback and bugs lists, I'll get that checked.

I know it feels like a long process... but we are really still in the very early days of a skin that we want to last a long time. Lots of work still to do! -- Sannse (help forum | blog) 01:26, November 20, 2010 (UTC)


 * Thanks for the 'update' Sannse, but please remind your (the wikia) team that the longer these issues drag on the more seasoned editors will drift away if no visible changes occur to address some of these issues we have raised. Wikis will down grade if the founders / admins stop maintaining them & when new visitors visit an see article with patent nonsense, spam links in C&P articles and few active recent edits they will quickly leave and not return.


 * Help pages have always been an issue in my view but users often went to mediawiki or wikipedia help but as the new skin differs more than Monaco did from monobook they becume les use and the basiks need explaining for newby editors, especially now wikicode is hidden following the introduction of the RTE and Oasis has hidden a lot more of the editing tools used by editors so as users stop looking what buttons do ( as they are now options on my tools) they will not see the underling structure of a wiki & actualy get the concept of how it links together. How often do you see new pages by new editors with NO Wiki links or categories & often find edits that add no content - just have a summary with what may be content they thought they were adding, as occurrences with the Category button regularly. My Tools should be a list of options selected by a check box for each one, not having to copy them in to the active list as currently done.


 * Wikia started revamping the Hubs and some of the features/layout of the new look appeared on them last year, but after the 3 main ones and the new portals and 'community central' was launched the others were never updated, few users will find most of then as were are the links to then in the new skin ? This half done hub project does not bode well for future update happening to 'fix' the main skin. Help had a revamp that whent so far but then appeared to stagnate, as enthusiasm runs out or the next new thing appears and the team moves to that or the member working on it leaves. I'm the same and have many unfinished 'projects' on my wiki, as i have dont have any pressure / help to complete them when they hit a brick wall, so the next (big) idea takes over as the focus of attention. - BulldozerD11 02:26, November 20, 2010 (UTC)


 * I just realized... it's like in Inception. We thought we were out of the Beta, but there was a Beta within the Beta. We're now in the Wikia-wide Beta. -- Fandyllic  (talk &middot; contr) 8:53 AM PST 20 Nov 2010

Help:Smelly Users (talk) was last edited at 16:53, November 22, 2010 (UTC)
 * My two cents: speed up loading. You may have been working on this a lot, but every other day pages just stop loading and are left stuck in loading heck. Might I point out that until the rest is loaded, the toolbar never appears and the "reply" buttons never work? At least they don't send you to the top of the page. How about this? How about experienced users that are not Facebook-connected no longer need to see the Facebook buttons, and have a choice (hidden deep in the preferences where no-one will find it) not to see spotlights altogether. Some of us have lives, you know, we can't wait 'till the next century for a page to load! And I agree with everyone about the bars, space, etc. FIX IT!

So that's what's going on! Phew, I thought Oasis was glitching up again. Well, all I can say is I wish I was part of the group. White Star Line 17:57, November 24, 2010 (UTC)
 * So that's why I saw sidebar on the left in the morning. But after that, I think it looks better on the right side. Final Cannon 19:56, November 24, 2010 (UTC)
 * ...or it looks better nowhere, since most of the stuff on it is useless most of the time. -- Fandyllic  (talk &middot; contr) 8:40 AM PST 25 Nov 2010

User Testing
I'm all for user testing. But the A/B testing is getting a bit crazy!

The interface changes in major ways multiple times during a single edit session. And I don't know which of the changes are new skin changes and which are just a test. There's no good way for me to refresh a page and check how it will look to the majority of our users. (Speaking of which... that would be a good function, a "preview as un-logged in reader" or as "default login skin", especially since as so many people are unhappy with the current default and are changing what they can with CSS.) New features or moved menus will normally get extra clicks... simply because people are trying to figure out what's going on, not necessarily because they like it or would use it on an ongoing basis. You can also learn a great deal from looking at existing patterns, for instance, how many clicks on "wiki activity" go to Special:RecentChanges almost immediately after? Has the % of visitors that edit or create a log-in gone down?

But my biggest concern right now is that I can't tell the difference between the changes that are part of the base interface being in constant flux these days and the changes that are just a test. I know that part of the idea is to not let the users know, and to see what naturally occurs, but that doesn't work well when you start experimenting on the people who are helping to run the experiment. Could there be some way to check when a page looks odd whether it was part of an experiment and to provide feedback? As confounded as it is with the other changes, the A/B testing is probably not giving you good data.

-- CocoaZen 19:08, December 19, 2010 (UTC)


 * Hi CocoaZen, This does sould like something is off. The A/B testing should not be affecting you in this way. Are you still seeing these issues? If so, where, and do you have a screenshot or anything to share with me? I would love to help figure out what is going on for you, just let me know more details. Thanks, Sarah (Help Forum) (blog) 19:24, December 27, 2010 (UTC)