SOE: Why Do We Patch When We Do?
Ever wondered why SOE patches their games at 7:00am Pacific time? Planetside 2 producer David Carey took to the Planetside 2 reddit to answer this question. Keep in mind that his comments relate to Planetside 2, which patches its US and EU servers simultaneously. As we all know, EverQuest II either patches its EU servers simultaneously with US servers or on an ~18 hour delay, depending on whether the update includes new zones, crucial exploit fixes, or sometimes just the direction of the wind, etc.
I’ve seen this asked several times, and I’m not sure if we’ve ever given an official response. I know some the European players feel our downtimes are unfair (6 am local for us) and wonder why we do it. Trust us, the last thing we want is the game down during anyone’s prime playing times. But we have a few reasons for this time slot, and we actually spent a lot of time thinking about it. I doubt it’ll take the sting out of having the servers down, but maybe knowing why will help a little.
The first reason is that we are a worldwide game, and our population numbers have several spikes, so we’re always going to be near someone’s prime hours.
“Why don’t you just bring European servers down at a different time than the others?” Not technically possible with how our system is set up. We can’t have multiple client or server versions out in the wild at the same time. I mean, nothing is impossible to our Coders, because they are awesome. Think about it – a dude fires a bullet at me from his desk IN BERLIN and my char dies at my desk IN SAN DIEGO in real time…that’s just amazing when you consider it, right? But this change would mean tearing down PS2 and starting the network code all over in some key areas. I can get our Tech Dir if you need more details there.
Another reason is that we update once a week or more on average (even though we’ve stopped the official weekly publishes, we’re doing our third one of this week due to hotfixes etc). We need people to run the patches, so we really can’t just say “Hey you guys are working overnight tonight and maybe two other nights this week, we’ll let you know.” I mean, they have families and that’s just not fair. So that rules out some time slots like 2am local other than for emergencies.
The main reason we picked this time slot is we’ve learned over the years, not just on PS2 but previous games as well, that bringing servers down always has inherent risk. Long time PS2 players may notice that we’ve gotten better about the longer downtimes over the months. I know we just had one the other day for the merge, but I think overall the downtimes have gotten much better. BUT, when something does go wrong, we need all hands on deck. Art, Code, Design, build process itself…almost anything can be the bug that stops us unlocking. So we need everyone in house. If we start at 6am, folks are showing up just as problems start appearing. If we start any earlier, the turnaround time on calling people, waking them up, etc just means wasted time. Trust me, I’ve made those phone calls often, and waking people up to come into work is never cool.
In fact, you may not know this but probably should have guessed, game teams are late risers. We’re playing games until 3am sometimes just like you guys (just one more turn, my archers are about to take Constantinople and it has the Hanging Gardens!). So if we didn’t have European players we would probably pick 9 am local as our update time. But that would be way worse for Europe, so we factored that in.
Lastly, I see some people saying “SOE hates/doesn’t care/ignores Europe.” Nothing could be further form the truth. This is the most multicultural company I’ve ever worked at, and lots of our employees are from Europe (including myself, Forza Italia!). We love and appreciate our European heritage, and would never intentionally do something to alienate our customers over there. You guys are factored into every decision we make, no matter where you play from. (Random aside – thank god for Google translate, Germans are kickass bug at finding bugs)
Anyway, like I said, when servers are down and you want to play, it sucks. But hopefully this post will let you guys know why we do what we do and when.
PS If you guys have any more unanswered questions we’ve never answered, hit me up.
Trackback from your site.
Comments (9)
SekretInfoz
| #
He had me til this:
and that malarkey about it was a Sony Pictures deal is B.S. SOE picked PSS1 as an EU partner all by their lonesome.
Reply
Pipsissiwa
| #
Yeah that sentence gave my a wry smile of ‘yeah.. right’ too.
Reply
Anaogi
| #
Ahem: The parent company, Sony Computer Entertainment America, made that deal. SOE just had to deal with it (and I remain convinced they weren’t any happier than we were and breathed a sigh of relief when it was put out of its misery).
Reply
Kinya
| #
Feldon – this post is written by PlanetSide 2 producer David Carey not Smedley 😀
Reply
Hoot
| #
I’ll never understand why people get upset about patch updates being targeted at a designated hour. Do people realize that they need to shut the servers down at one point in order to update them?
The only real solution would be to have European servers updated at a different time to accommodate their timezones. Than again people will then bring up the issue of “oh boohoo you gave them early access to the patch/update/expansion you surely don’t care about us!”
Reply
Feldon
| #
I do wish they’d stop calling the process of taking down a server and changing its software a “hotfix”. This word. I do not think it means what you think it means.
Reply
LSD
| #
The term hotfix has never exclusively implied that it can be applied without a restart of some kind. Any hotfix that changes in-memory data (i.e. executables, DLLs, etc.), will require said data to be flushed, and the easiest way to do that is restart the service, which necessitates downtime. More recently with widespread cloud computing it’s become a lot easier to propagate updates over distributed services with minimal end-user impact, but there are still plenty of times when it is simply impractical to do so.
If they’re using the term hotfix to simply mean their scheduled weekly patch that goes through normal internal QA, then sure, that’s an incorrect use of the term. But saying it should only ever be used when it can patch running systems without a restart is misleading.
Source: A guy who’s worked in IT for over a decade and has had to apply numerous “hotfixes” that required a server restart.
Reply
griffonlady
| #
I worked nights for a few years (2008 through just this year, actually 😉 ) and I can say with certainty, having the servers go down just as I was able to log in was annoying, but not any where NEAR as bad as the servers crashing unexpectedly due to not getting maintenance. 😆
Not to mention they are usually only down for about half an hour, if it’s just standard maintenance…. and if you sit there staring at the “server Locked” character screen like i do.. waiting…. 🙄
>.> Also… if you manage to log in juuuust as the server is unlocked.. you are much more likely to get an elite mercenary, if you don’t have them/ do want them. ^_^ (I fiiiiiinaly got Kenny Tuesday. ^_^ Not so easy to be awake/home for the end of maintenance now that I’m a day time student again not a night time nurses assistant. o.0)
Reply
Striinger
| #
I was a little surprised that soe didn’t go with true ‘hot’ patching like GW2 (or 2 it NW?) does (and some other semi-recent games). They patch in parallel to live then cut over area by area. When your area is cut over its seamless inks you need a client patch (in which case you bounce your client).
I get that it may not be worth a change for older games, but what about the new stuff?
Reply