And then i completely spazzed out and lost track of this thread.
Blackbeard fell prey to a terrible fight amongst the big brother processes. Essentially, our servers are designed to be extremely fault tolerant. Pull one out of a shelf while running, and the players on that server will be thrown out of the game, but when they try to log back in, their characters will be automatically respawned up on another server. Total down time? 30 seconds.
Cool, right? Except what happens if the process that controls who's allocated to which server goes down? Well, another one spawns up. There's various races to get to see who wins, but one big brother becomes supreme, and we're all good again. The problem is that there was some logging that was causing a forty second delay on the big brother process (which is something we've never seen before) that was delaying each one long enough to be slayed by a rival. Which meant lots of processes were spawning up, and nobody was winning.
Please keep in mind that the following explanation comes second hand via my wife (she has first dibs on the VPN connection) and neither of us are devs. Well, she used to be a dev, and I got through K&R, but that's not who we are, if you catch my drift.
I'm going to have a shower and lie down.