A Tale of Two Servers: Freeport, Everfrost

Rothgar, Zoltaroth, and others on the EQ2 team, as well as OPS (Operations) were up until all hours trying to wrangle Everfrost and the Freeport [EQ2X] servers into behaving reliably. But first, a quick fix for Everfrost players who are missing characters:

Everfrost players who are still missing characters are encouraged to follow the instructions on the EQ2 Forums. In short, go to your EQ2 folder and delete your LoginCache and LoginCache-US folders, as well as any <Servername>_characters.ini files.

Everfrost Woes

The Everfrost server has had issues with characters disappearing. From Rothgar:

We’re experiencing issues on Everfrost currently and these problems may explain the missing characters. The characters aren’t really missing, the server is just timing out on your request for character data.

and:

I just wanted to post an update for Everfrost players. We have a new server build that we are preparing that should hopefully correct the problem that was occuring with your server today.

and as an attempt to resolve the problem, the Calendar was disabled on Everfrost as of 9:10pm pst:

Server is unlocked now. Please let us know asap if anything appears amiss.

Also, due to the new server build, that calendar information will temporarily be offline until the next hotfix. We apologize for this inconvenience but assume that missing calendar info would be better than an unplayable server.

as of 10:30pm pst:

We’re still seeing some excessive load times on startup probably due to the large number of people coming in at once. We’re going to lock the server occasionally to try to manage that load times.

The Freeport Report

The Freeport server has experienced several crashes (including one just at the time of this writing) during the day and into the night. Players are reporting absurdly long Zone Times. As we’d reported earlier, SOE instituted Queuing for the Freeport server to see if this helped with the issues:

Just an FYI, Gold and Platinum members log in and bypass the queue. Although, it seems we have a bug with the queue counter not cleaning itself up properly as people leave the queue.

We are still working on troubleshooting the zoning problems. We’ve seen a bit of an improvement in the past few minutes by disabling some processes but are still working on a permanent solution.

and:

We know without a doubt that the zoning issues are not caused by server population. Some of you have been asking to open a new server but this won’t fix the issue. When we initially opened the server with the login queue enabled, we were having the zoning delay even with a small number of people logged in.

and:

Ok, I just confirmed that there is indeed an issue with the queue where only Gold members can bypass it, and not Platinum. Although this is normally insignificant since we’ve never used the queue, we will be sure to get this fixed in the next hotfix.

Again, we’ve made every attempt to not have to use the login queue, its only there as a safety valve and to help with troubleshooting issues like we are having today. Bypassing the queue is a perk of membership just like many of the other perks.

and:

We had to enable the queue in order to confirm that its not population related, not vice versa.

We are confirming with current players that the zoning problems have improved and when we are confident that the server is in better shape, we will increase the population numbers.

FYI, I did confirm there is an issue with the queue number not counting down properly when people leave it early. It will eventually clean up the numbers as more people are put into the game, but with the max pop set so low right now, new people aren’t getting in very quickly.

and:

We’ve brought the server back up on a new set of hardware. We’ve enabled the login queue once again to make sure we don’t break something right out of the gate. If the new hardware is handling the load fine after a few minutes I’ll continue to increase the population numbers until we get the queue turned all the way off.

The ultimate goal is to see if this fixes the zoning problems.

and:

I appreciate the patience and understanding that many of you are showing through this. We are still continuing to track down the zoning issues. We’re slowly increasing the population numbers since it doesn’t appear that lower pop has any bearing on the zone times.

To those asking why Velious went live considering the issues over the weekend:

This problem doesn’t occur for us internally, in fact, it only appears to happen on Freeport and none of the other live servers. So before we could troubleshoot it further, we needed to get all existing changes out to the server. It had nothing to do with the expansion being a higher priority than server stability. Server stability is our highest priority, but the changes we made were coupled with the expansion code.

Now that those changes are live, we can get a clearer picture of where the problems are occuring. We are working on that now.

as of 9:00pm pst:

I just wanted to update everyone that we are creating a special build of the server that will provide some additional info to help us troubleshoot the problem. We hope to have this build ready to go within the hour.

as of 10:00pm pst:

We’re not going to take the server down anymore this evening but we understand that the zoning issues are still happening and we are still working on a solution for that.

and:

We have to collect more data and continue to look for the issue. There doesn’t seem to be much sense to leave the servers down while we do that. While the zoning problem is annoying, at least you should be able to get some playing time in the new zones.

As far as sleep, most of us have been up since 6am and being mere mortals at some point we do have to sleep, yes.

Could Have Been Avoided?

A curious footnote to the 7.5 hours of server downtime some players experienced on Velious Launch Day is this comment from SmokeJumper:

Yeah, well…I’m not going to hedge about that at all. It sucks. I’m very disappointed because I personally think a good chunk of these issues were avoidable with just a bit of clear thinking ahead of time. But we’re fixing them as quickly as possible now and hope to have things on a stable keel soon.

It’s hard not to interpret that as an indictment of some of the decision-makers at SOE.