A growing number of players on the Splitpaw, Valor, and Storms servers (located in Amsterdam and serving the UK and English-speaking Europe) have been reporting unplayable gameplay lag and connection delays.  Some players are reporting no problems while others are experiencing 5-6 second cast delays. Further confusing matters, there seems to be no geographic pattern to the problems, as some living in close proximity to Amsterdam are reporting the biggest problems while those far away are getting fast responses.

UPDATE: We’ve updated this article to address the Valor and Storms servers.

In the absence of further information, this would suggest a Routing problem of some sort from the Amsterdam data center.

More information as we have it…

This information is not exactly a secret, but it’s a question that comes up often. Where is your EQ2 server located? Thanks to Rothgar for the list which we supplemented.

Servers on the west coast [San Diego] include:

• Guk
• Everfrost
• Antonia Bayle (RP)
  
• Unrest
• Butcherblock
• Nagafen (PvP)
  
• Vox (PvP RG)
  
• The Bazaar (RG)
  
• Freeport (EQ2X)
  
• Sebilis (JP)

East Coast includes:

• Permafrost
• Oasis
• Crushbone

Our European [Amsterdam] data center includes:

• Valor (German)
  
• Storms (French)
  
• Splitpaw (UK)
  
• Barren Sky (Russian)
  
• Harla Dar (Russian)
  

Raid Zone Lag can add randomness to an already challenging fight. There are many causes such as how an encounter is designed, unintended interactions in the zone code itself, etc. This issue is of particular importance in Velious because the Tower of Frozen Shadow x2 raid zone drops a Clicky item which is required to take on most of the Velious x4 raid content. Players receive an item which must be clicked by certain classes at certain times in certain named fights in Velious hard mode raid zones in order to make them defeatable.

Although some guilds have been unable to progress due to lag specific to certain zones/encounters, compounding a consistent level of lag already present on their server (Unrest, etc.) I have received word that guilds on every server have managed to get these items, albeit with a lot of extra runs and some luck.

That said, while we wait for new server hardware, Rothgar and his team have been working on fixing specific issues within the raid zones to reduce lag:

All Velious raid zones should be running much better from now on. We’ve also found a few other performance improvements today that should hopefully make it out next week.

We are focusing very hard on performance along with the features for GU60.

added another quote:

Please do try the zones again and let us know how its working for you. We do have plans in place to resolve issues like these and we were able to make some changes last night without taking the servers down that should have helped tremendously.

We have additional plans in place to improve server performance even more over the next few weeks and months.

EQ2 Designer Windslasher is working on the lag issues in the Storm Gorge public quests, but apparently it’s not an easy problem to solve:

I am working on changes to reduce lag. I’m very aware of how bad it is, and I consider that the #1 problem to fix. It also happens to be the hardest, which is why the going is slow. That is all.

I’ve finally managed to escape Charles de Gaulle airport which completely ground to a halt due to 3 inches (7cm) of snow. Meanwhile, our SAS pilot made a perfect landing in near whiteout conditions with heavy snowfall on a runway and at an airport positively blanketed with snow. Ah… Copenhagen… 😉

Anyway, ’bout those server merges…

Today’s hotfix included some additional lag fixes. Here’s Rothgar:

With this morning’s update we made a fundamental change to how our zone servers throttle their frame rate. Our hope is that this change will allow busier processes to consume more of the server resources and increase performance (reduce lag). Whether or not this change has a noticeable impact will depend on many variables that can only be observed in the live environment and under a significant load.

We’ll be watching things on our end as we ramp up to primetime tonight but I’m also looking for your feedback as to whether or not you’ve noticed an improvement after this morning’s changes.

Thanks!

From Rothgar on the EQ2 forums:

We’ve located the issue that was causing the slow performance last Tuesday. This problem had ZERO to do with battlegrounds but they were taken down as a precaution anyway. I know that battlegrounds tends to get the bad rap because it’s new, but they are not always the cause of problems. In fact, we’ve got the character loading issues under control now so that there are no performance spikes as a result of characters going to and from the BG world.

We’ve been making some general performance improvements that will help lower the game server lag (exhibited as spell lag) which will be going into testing soon. When we are confident that these changes are working correctly, we will get them out to live servers.

We have several engineers working on performance issues and this is currently our top priority.

After the widespread outages of the EQ2 servers the night prior, a thread was created on the EQ2 Forums to solicit an explanation from the EQ2 Developers, and/or new EQ2 Senior Producer Dave Georgeson, on the cause, and to seek reassurances that the issues are being identified and on track to be resolved.

As they tend to do, the thread branched into speculation on the origins of Dave Georgeson’s forum handle — SmokeJumper. Skywarrior posted his theory:

Hmmmm.  Smokejumpers are extremely brave (or crazy) and well-trained individuals who jump out of airplanes in order to reach remote areas involved in forest or brush fires, with the express purpose of helping extinguish the flames.  It seems to me that Dave’s handle is quite appropriate to the situation he finds himself in.

Dave “SmokeJumper” Georgeson, the new EQ2 Senior Producer responded with:

Ding, ding, ding. You have it in one, sir.

And yes, I still answer to “Qix” or “Qixter” also. (I’ll bore you with the story of where those names came from some other time.)

Regarding the performance issues: Yes, we’re completely aware of it, and yes, it’s our highest (and only) focus for the programming team currently.

It’s a two-pronged issue that affects multiple aspects of our architecture, and neither of the issues is a “quick fix”.

In other words, we have to gather more information and study that data to find out exactly where the problems are before we can create solutions.

That’s in progress now and we’ll get solutions made ASAP. But currently, I don’t know an exact date when “ASAP” will be completed…mostly because we don’t have the solutions figured out yet.

But as I said in the other thread…we *will* kill them.

In a break from the stock answer that has been given in regard to server merges for the last few years, Rothgar says server merges are something they are considering a little down the road.  The EQ2 world isn’t exactly in the best shape it’s ever been in at the moment with zoning issues, instance lag, contested named fight lag, and broker lag so the dev team has a few issues to take care of before they can move forward with this idea.  But this tells us that the idea is at least back on the table.

Rothgar writes:

We are considering server mergers but they would be a little down the road.  Before we consider any more mergers, databases need to be upgraded and general server performance needs to be addressed.  These are things we are working on and when we feel that we’ve achieved some of these goals we will seriously look at what is best for the EQII population.

And in response to someone claiming that Server merges = Dieing population, Rothgar responded with:

This is most definitely not the case.  Our subscription numbers are on the rise.

The addition of easier character transfers through the Station Marketplace has made population numbers more liquid than in the past.  We are seeing players migrate towards higher population servers like Antonia Bayle and Crushbone.  In fact I’d be willing to bet that AB currently has a higher population than it has ever had in the history of EQII.

In order to stabilize the population, we’d like to increase the numbers across all servers and the easiest way to do that would be to merge some of the lower population servers making more “super servers” like Antonia Bayle.

This would also make better use of our server resources and allow us to re-purpose some of that hardware to better serve the larger worlds.  When you think about the minimum requirements for each world, we need all static zones up and running at all times.  This includes all of the city zones and most of the overland zones.  On some worlds maybe those zones have < 20 people in them when they can easily support much more.  By combining worlds we lower the total number of zones that need to be running by putting more people in them.  So, the hardware that is freed up can be put to use servicing instanced zones which is where players are currently seeing the most server lag.

It’s a win-win for us and for players.  We just can’t do it until we can be sure that your experience will be improved by the process.

As always, EQ2Wire will stay on top of this story and let you know as new developments occur.

Rothgar has been working closely with the Database folks trying to solve the Server Performance Issues, specifically Server-side Lag, and extremely long Zoning times which have become a major problem facing EQ2 since Game Update 55, the Sentinel’s Fate expansion, and the introduction of Battlegrounds. These Database and Server Lag issues have been so disruptive, that they have largely drowned out discussion on other issues facing EQ2.

Fortunately, Rothgar brings us an update on the situation regarding some long-needed caching being added to how character data is transmitted and stored:

Right now we are addressing the database-related issues and then we will be moving on to the game server performance issues that you’re speaking of.

We have several very positive changes in the works to improving database performance. Today’s hotfix contained a small change that should help and we have several other changes going onto the Test Server. One of the changes I’m most excited about is something that Zoltaroth just finished working on. Character data will no longer need to be loaded from the database when you move between zones. This should cut out a lot of traffic from the database and even make zoning a little faster. If we ever experience a problem with the database like we’ve had in the past, this new caching scheme should keep the database issues from affecting your zoning times.

If this round of changes makes the impact we are hoping for, we will move on to other types of performance issues.

Tomorrow’s Hotfix will include some changes targeted at server performance/lag. These changes will affect the number of spells and combat arts which can be linked together in 1 macro. We are trying to determine the maximum number of spells or combat arts which it will be possible to put on a single macro.

Thursday, January 28th Hotfix