Feldon From the EQ2 Forums:
Crushbone, Befallen, and Permafrost will be brought down Friday morning for some upgrades to address the lag issues.
The estimated downtime is four hours.
We’ll be bringing down the servers at 6am PST and will update this thread with details as needed.
Rothgar has been communicating what steps the server team have been taking with regards to this issue. You may be interested in some of those details he’s posted, after the jump.
We seem to be experiencing some issues at the DB level and multiple servers are being affected at the moment. Everyone is looking into it and I’ll post updates when I hear something back.
…
We’ve determined the cause of the [database] latency tonight.
As you might know, we’ve been purging some old data, moving DB tables and rebuilding indexes to try to improve performance across all of the servers. Apparently the database ended up in a state where it wasn’t using the correct indexes due to some of the changes that were made. This had a severe impact on query performance.
The DBA’s have updated the indexes for the tables that were having issues and are now seeing an overall improvement in database performance. Right now things look to be performing very well. We’re at the beginning of our peak time and my eyes will be glued to our performance data over the next couple of hours to see if any more problems occur. I’ll be paying close attention to Crushbone and AB.
…
The problem seems to have subsided for now. Our DBAs and Operations department are still looking into the cause of the issue. It was certainly something that was across multiple servers, but not all servers.
And yes, we are aware that Crushbone and [Antonia Bayle] have been experiencing similar problems for a long time. Permanent fixes n the way of hardware upgrades are on the way.
Sounds cool. If they’re using oracle, this phenomena of queries not using the proper indexes has a devestating effect, and it’s hard to determine the cause fast.
Come on now,the lag problem is not just isolated to these particular servers. lets see this on all servers please
Rothgar has been on the Crushbone Servers’ 70-79 chat the last two nights for a brief time. Tuesday night he took a bit of a pounding, the players were really torqued at the lack of response from SoE (rightly so) … last night the tone was better … especially as there had been a minor fix and zoning times had reduced from an average of 20 minutes to 8 or less (for most of my raid group and my guildies).
On another note, one that I am following as I am effected. LU52 (November 23) brought massive problems to frame rates and memory crashes for lots of players. Someone asked Rothgar about it and he responded that FPS was a client side problem and the LU should not have effected it. That being the case, why did he respond to an SoE thread 25 times that they were working on it ?
http://forums.station.sony.com/eq2/posts/list.m?start=75&topic_id=459319
Note: Autenil (1), Rothgar (25), TsrKenC and Tsr-DanielH (4) as well as Act and Feldon all chimed in and have responded. So SoE thought that they had caused a problem and they WERE working on it, that they had a few small fixes, and that they were still working on a solution.
From Rothgars’ comments last night, to me it seems that they can not wholly fix the problem and are now pretending that it never happened and are just going to walk away from it.