The State of The Net

Over the last couple of months, you may have noticed that we've had some downtime due to scheduled maintenance - the first for a very long time. Unfortunately we've also had some unscheduled downtime.

Until recently, Phoenix (the server MUD runs on, not the mobile) had been up and running for over 500 days without a reboot. Its long uptime was finally brought to an end by a power failure at the data centre it was being hosted at - something which simply shouldn't happen!

The first scheduled maintenance we had planned actually involved moving Phoenix (and the entire network it lives in) to a different data centre. This should have been a relatively quick move, but as so often happens, things don't turn out the way you planned.

It so happened that the weekend we moved everything, the Microsoft "slammer" worm hit the net. While it didn't affect us directly, it left our ISP without management of their NOC on the day we moved. This unfortunately meant they weren't able to switch the routeing of our network to the new data centre. We were left with the choice of putting everything back, or moving it all and hoping they'd regain management in time.

We made the decision to move and thankfully, our ISP regained control of the network and everything was brought back up in the new data centre, albeit a few hours late.

The second scheduled maintenance we had planned was to upgrade from a 100Mbit to a gigabit port, and inserting a new router between the firewall Phoenix sits behind and the Internet. Thankfully, this went a lot more smoothly than the first change!

These changes are partly to accomodate a new site being hosted on the same network as MUD - formula1.com, the new official Formula 1 website. As I'm sure you can imagine, this is an enormous site - and needs some hefty hardware to support it.

Unfortunately, we experienced what can only be described as "network instability" in the early stages. Some quick redesigning and fast reconfiguration resolved the issues, and everything has been (touch wood) behaving since then.

I would love for MUD to stay up and running without interruption for 24 hours a day, 365 days a year. But there are times when changes have to be made and unfortunately with changes there are inevitably teething troubles, too.

So, I would like to take this opportunity to appologise to everyone who has been affected by these issues over the past couple of months. Hopefully, Phoenix is now in a good position to stay up for another 500 days without interruption.

Thanks for your patience.

Foddy