User blog:Crucially/Iowa datacenter test

'Update: The second part of this test will be tonight (Monday 23 November) at 11pm EST. This involves physically disconnecting the main datacenter, so there will be a short time of unavailability for some users until the Iowa datacenter takes over. Once Iowa is serving all pages the site will be read-only until the test is over. Thanks for your patience with this essential testing!'

Our Iowa datacenter is nearing completion for automatic failover. To test this ahead of time we are going to do a controlled failover to Iowa at 11:55 PM PST on Wednesday. Since Iowa is a backup, it is configured as read-only, meaning you will not be able to edit anything. All pages should however render correctly.

What we are going to do is, in a controlled environment, switch all Apache servers in San Jose to stop accepting traffic. Our front side caches will notice this, and redirect traffic to Iowa. Once the test is complete we will enable the Apache servers in San Jose and the traffic will flow back. During the switch over we will monitor Iowa and try to find any errors that might happen, this period of time should not last more than 30 minutes. If we discover any serious errors we will move traffic back faster.

The second step in our testing will be next week (we will announce ahead of time when we get closer to it) when we will actually pull the plug on the datacenter. This involves physically disconnecting the redundant connections to our upstream datacenter. When this happens our London and Iowa front end servers will immediately start sending traffic to Iowa. Anyone accessing San Jose directly will be temporarily be cut off while our DNS provider automatically changes which datacenter they are pointing to. This will last 30-60 seconds.

Initially I expect Iowa to be a bit slow while everything spins up and the caches start to fill, this will result in some slowness, but shouldn't last more than 60 seconds. Depending on the result of this, we will be looking for ways to get Iowa to have its caches full.

If you want to try out the Iowa datacenter, you can go to Special:Datacenter and choose Iowa. You will be in read-only mode, but you will have an entire datacenter for yourself.

If you want to watch what happens when we switch over, our Ganglia install can be found at http://ganglia.wikia.net/iowa/, the graphs should show a large spike in activity when the switch over happens.