Note from SOE’s CTO About AoD Delay

From Rich Lawrence on the EQ2Forums:

Hi folks, this is Rich Lawrence, CTO with SOE. Some of you probably remember me from original Planetside as “Twist”. TR rules, but I digress.

I must apologize for the extended down time of EQ2 while we get everything in order for free to play. Obviously we didn’t anticipate it being this long. I know how important it is for the game to be available when you want to play, and what it means to miss a raid you were looking forward to, or check out new features. In this case the issues that are keeping you from playing are under my watch, so I need to say it: I’m sorry. I value my game time as well when I can get it, and I know how you feel.

EQ2 is a great product with a long history for us, and part of the reason we are taking so long on this update is because of that. Let me explain a bit. In order for everyone that played in the past or currently to get the best experience when logging in on the first day of free to play, there were many changes that had to be made in data to represent all of the features those players (past or present) were entitled to, plus new goodies, and the merge of what amounts to two different products in a business sense (EQ2 and EQ2X). This is a lot of accounts – millions – and a lot of ‘business rule’ complexity. While we have top of the line hardware running our databases for this kind of thing, and folks dedicated just to keeping it working well, the result of all that took quite a bit longer than we thought it would.

I hear the engineers of you out there (and yes, I know we have many very capable developers in our audience) saying, “But you should have modeled it ahead of time and known how long it would take!”. You’re absolutely right. We did. Turns out our model was just a tad less than 1:1 scale in some respects. I know, I know, Captain Obvious calling.

In the end we realized two things as the maintenance progressed – that this was going to take much longer than we thought it would, and correspondingly the results would be not as well tested as they should be before release. For instance, there was a risk that some out of those millions might not end up with the right race packs or character slots. In the end, we decided it was best to extend the maintenance through the night so we could do a better job of checking that kind of thing.

Please understand that we don’t do this casually or think that it’s a lack of dedication. Most of the key people involved in this publish, on the game team and our platform side, have been here very long days and every day leading up to this. I just had to tell some folks that had been here for 30+ hours to go get some sleep. If there was any way I thought we could be certain you’d be able to play with everything correct tonight, we would have done that.

I’m committed to changing how we do this type of operation going forward so that it has much less visible impact on you the players – we will trade more back end prep time and resources against less publish downtime, basically. I do hope everybody enjoys the massive effort the game team has put into this release. I’ll see you online tomorrow.