Forum:Site performance issues and improvements

Hi all,

I just wanted to give you a brief update regarding some of work Wikia is doing to increase site stability and performance. As mentioned here, there are a number of initiatives underway.

Here are some issues were working on:
 * ISP Replacement: The Internet connection in our San Jose, CA data center was unstable and had resulted in downtime of the site. Failing over to our new data center in Iowa will not be reliable until it is 100% complete. We recently replaced our single San Jose ISP and added a second connection to Internap.
 * Database Servers: As our traffic has grown, our database servers have occasionally been overwhelmed. While the traffic is great, MediaWiki doesn't handle the situation gracefully (i.e. using a less-busy database server), so the result is downtime or slowness. These database issues have also caused Wikia Stats and database dumps to fall out of date.  To resolve these issues, we are upgrading our existing servers (adding additional memory among other things), and installing a new Virident database server.
 * Cache Servers: Recently we have had a few situations when our cache servers ran out of memory. While this does not result in downtime, this occasionally causes blank pages to show up rather than the desired content, and sometimes changes to a wiki don't show up immediately. This is especially apparent when updating site CSS or Javascript. To prevent this, we are increasing the capacity of our Varnish servers, and we created a patch for Varnish which significantly improves performance.
 * Creating Redundancies: We are putting the finishing touches on our new Iowa data center and expect it to be fully online within the next three weeks. We have already been sending traffic (about 200Mb/sec. of 500Mb/sec. total) through Iowa for some time to improve page load times, but these final improvements will create complete redundancy in our infrastructure and eliminate the remaining single points of failure that can cause downtime.
 * Page Load Times: In addition to all of the above improvements, much of our engineering team will be spending the next few months optimizing both the MediaWiki and extension back-end code, as well as the Javascript that is executed in the browser on each page load. The combined impact of greater code efficiency should result in a noticeable reduction in page load times.

Some of the projects above may result in brief site issues. Please bear with us while we create a little dust. As always, don't hesitate to let us know if you think something is amiss, but also know that we're likely to already be working on a fix.