Feldon 
Greg “Rothgar” Spence, Nandy “Zoltaroth” Szots, Alex “GermInUSA“, Josh “Autenil” Kriegshauser
This is not an EQ2 panel specifically, but a panel about how MMOs like EQ2 are written, the behind the scenes tech stuff, etc.
* Devs talking about how they became part of the EQ2 team and some of their past.
EverQuest II
- It’s a pretty big project.
- 12 years in development.
- 250+ man years prior to launch.
- 400 physical zones (800 virtual — reused zones with different population)
- 500,000 art and sound assets
- 716,496 data files
- 65,305 characters/NPCs
- 160,000 items
- 11,408 quests
- 1500+ Achievements
- 583 collections
EverQuest II
80 Projects
- 2,116,388 lines of game specific code
- 804,334 lines o tools/utilities
- Full Build Time: 3 min 51 sec with Incredibuild
Tools
* Microsoft Visual Studio
* Game code and some tools written in C++
** Placer – used by artists and designers to populate game world
DPOs – designer placed objects
written in C++
** Composer — used to build characters in the game. apply sounds and particle effects to an NPC or character
** UIBuilder — written in C++
Designer tools written in C#
** Sequence — designers use it. glorified XML editor. quests, characters all created in here with all different custom forms
** Designer Dashboard — fairly new tool. Allows more control over populating zones in the client. Sort of like a house decoration tool but for placing and moving NPCs and spawn points.
Web based Tools
* Zoltaroth showing some of his Perl scripts.
* They get 20 GB of logs per server per day. They log everything.
* If we find an exploit, we write a Perl script to scan the logs for CS to deal with it.
* Heirloom items were changed by a Perl script that modified 18,000 files in one shot.
Incredibuild
* Network distributed compiler. Everyone’s computer does work on compiling of EQ2.
SOEMON
* Written by SOE.
* Dashboard for maintaining live games. If you tour SOE headquarters you’ll see this tool monitoring server health, # of players per server, frame times, server performance.
* EQ2 devs generally have SOEMON running in the background to check on server status.
Recap
* View auto-generated client crash logs which are automatically uploaded to the server when your EQ2 client crashes.
* Recap can tell which line of code the crash occured on, and is aware of different builds of the game.
* Recap has helped reduce client crashes substantially.
Telemetry
by Rad Game Tools
* Helping us fight the Lag Beast!
* Rad Game Tools writes a lot of middleware that SOE games use.
* Profiling tool. Look at the game’s running performance during a lag spike to figure out what is causing lag.
* Devs can play test content, connect to Telemetry, and see what happened during lag spikes.
* View which function calls were running during a specific time slice.
* Can break down to which part of the UI might be taking too long to redraw.
* We really want to use this to figure out what’s going on in the client or server as far as lag. For instance a dev could join a PQ, run Telemetry, and then have a very detailed analysis of client and server lag.
Q&A Begins
Q: Can you tell us about Database servers vs. Zone servers?
A: We have a World Server, Multiple Login Servers..
Q: How much is EQ2 influenced by the original MUDs?
A: That would have been a great question 12 years ago. 😉
Q: How do you maintain the game on so many different computers and rendering?
A: Talking about Shaders and code that runs on your graphics card. It’s very hard for us to find these things without reports from players. We’ve given better tools to our artists to find Shader problems.
Q: Can you track the performance of past events like a Storm Gorge PQ from hours/days ago?
A: We have to turn on Telemetry when we are capturing data. If we left it on, the servers would get a lot slower. We don’t run it passively. But if the server detects a long ‘frame’ or update cycle, we do write extra logging to disk.
Q: When a character dies, what actually happens on the back end? Is there a flag that gets set? We’ve had some problems with Call of the Veteran.
A: Yeah we talked about this a bit with players being unable to use CoV cause it reports that one player is dead. When you’re dead, there is a flag on your character, and it’s playing a certain animation state. CoV should be using the same function call checking whether you’re dead, whether you’re able to cast a spell. It’s curious why CoV would fail while other checks don’t have this problem.
Q: Can you share with us any details about the Server Upgrades?
A: We have a number of entity servers that run individual zones. A single piece of hardware can run multiple zones. A lot of the server hardware was launch hardware. High end Intel Xeon at the time. But we’re asking the server pool to do more work with all the zones we’ve added. The new servers are something like six core machines with HyperThreading can run 24 processes. We’re actually able to cut down the number of ‘boxes’ from 32 to 10 machines and still have plenty of overhead. Less power requirements, less rack space. It was a big budget hit to do, but we’re very happy to be doing that.
Q: Talking about caching data without hitting the database all the time.
A: A few months ago, we made the Zoning process entirely in memory instead of hitting the database. Every time you zoned, it used to hit the database like you were logging in for the first time. At first we had problems with hanging at Receiving Zone Info. We fixed that.
Q: Do you cache database updates?
A: We don’t cache them but they go into a pool. They go into a queue. Sometimes if a server crashes, sometimes a few get missed and it looks like a ‘server rollback’ until the server catches up.
Q: What’s the biggest character you’ve seen in terms of data?
A: Character Transfer System is hitting 5-8 MB limit of compressed character data. So individual characters are getting quite big. Typically as a rule, we don’t delete characters. Even a ‘deleted’ character is just flagged.
Q: Any movement on using multiple cores on the client?
A: Even the servers are pretty linear as far as threading. As far as multicore support in the client, Alex is working on that next. What’s currently happening is the game updates the frame, sends all the data it needs to the graphics card, then the CPU has to wait for the graphics card to finish the frame before it can proceed. We’re going to move graphics updates to a thread on the second core so the game can be working on building the next frame while the graphics card is busy. Hopefully coming soon!
PANEL END
This was a Live Blog by EQ2Wire.com from SOE Fan Faire 2011 at Bally’s in Las Vegas, NV.
Only the last question im interested in. So many mmos have recieved graphical facelifts over the years and eq2 still can’t manage proper support for multi core cpus. The whole “its too much work” to re-write engine kicks in. I would happily fork out a premium price for an expansion where i know that it will improve only upon the graphical and performance aspects of the game.
I have been reading and checking your site as it seems to be very up to date on all things eq2, but I can’t find out about the new housing “Tenebrous Island Refuge and Felwithe Mansion”. Do you know how we can own these or what they are for? Guild halls? Build your own home?
Juanita,
The new houses are now on Test and should be available soon in-game. The Felwithe Mansion is just another style of house based on either New Tunaria or the Throne Room.
Meanwhile the Tenebrous Island Refuge is a floating island with a small pond and waterfall which is otherwise a blank slate for players to build their houses from scratch. With the variety of dividers, floor tiles, and other building supplies available within EQ2, players have actually been requesting a blank slate or empty lot for some time now.