Feldon If there is any doubt that EQ2 devs are working as hard as they can to resolve the persistent Receiving Zone Info issues affecting Freeport and (to some extent) other servers, know that Zoltaroth and Rothgar put in 16 hours on Tuesday, and I know that Zoltaroth worked again until midnight this morning. Sleep is overrated!
From Rothgar:
The good news is that we’ve found and fixed the source of the crash that both Butcherblock and Permafrost experienced today. This crash fix will go out with tomorrow morning’s hotfix.
and:
To update further, we have a patching coming tomorrow morning with several changes as well as a large amount of additional logging that will help us troubleshoot the issue even further. This is still our top priority and our lack of updates today has only been due to the fact that we’ve been very busy behind the scenes working towards a solution for this problem.
We will update more later in the day tomorrow.
From SmokeJumper:
1) Long zoning times on Freeport : We’re still tracking this down. We thought we had a fix that went into yesterday’s update, but it only did a little bit to improve things. There’s definitely still a problem. We were here very late last night trying to track it down, but we finally had to settle for putting tons of reporting into the code. We’ll be watching those reports today to try to track down the source of the issue…and then kill it dead, dead, dead.
In response to players seeing long delays with Receiving Zone Info, Rothgar continued:
It’s certainly possible and probable that the same issue exists on live servers but for whatever reason, they are not experiencing it to the extent that Freeport is. There have been a few other assorted performance issues in the past couple of days so its always possible a different problem is exhibiting the same symptoms.
I assure you we are just as concerned with the impact that these performance problems are having on potential new players, but honestly I’m more concerned about the impact its having on you guys that have been suffering through it for a couple of weeks.
In response to some serious trolling, Rothgar had this to say:
First and foremost, if you’ve worked in any kind of engineering or development company it wouldn’t be news that 90% of the time can be spent just finding the cause of a problem. Often times the fix is simple and can be done in minutes. So, in my opinion as a software engineer with 15+ years of programming experience, it’s perfectly fine to say that you don’t know what the problem is as long as you have a clear plan to figure it out. (Which we do)
Now myself and Zoltaroth were here when those guys left almost 5 years ago. We’ve both been with EQ2 for almost as long as the game has been live. No one knows the in’s and out’s of this software better than us and if anyone can fix it, we can. We understand that the issue is frustrating and while it seems we should just be able to take down the server, press a button and fix it, sometimes its not that simple. Insulting our intelligence, calling for firings and otherwise taking your anger out on our team isn’t going to help the situation.
This is a tough issue, we will admit that, but we’re making progress. The same code is running on all EQ2 servers yet Freeport is the only one with this problem. You might think it was population, yet we’ve reproduced this problem on the server when only 100 people were logged in. You might think its a database issue, but we’ve also reproduced this problem in situations where the data is 100% cached and the database was never accessed. It’s one of those issues where every theory you have can be disproved by another test. But like I said, we are making progress and we understand where the issue is occuring but need to narrow it down further.
Tomorrow’s hotfix has a few changes as well as additional logging that will help us narrow that down. The best case scenario is that tomorrow’s hotfix solves the problem. The worst case scenario is that we gather more information and get that much closer to a solution.
Again, accept our apologies for the problems that you’re experiencing. We are doing everything we can at this point to solve it in a timely manner.
Unfortunately I cannot answer questions about extending Velious rewards or other types of compensation but I will forward that on to Smokejumper.
We really don’t need to hear how many hours you guys put in, I personally couldn’t care less. What do you people expect a pat on the back for doing you job? Not from me that is what you get a paycheck for. SOE released a product that was broke all to hell still on the day that it was released, had our server down for nearly 12 hours more than all of the others. Some things were broken moreso than they were during the last week of beta, heroic zones broken at the bottom of the chain that keep you locked from the second zone in that series which you can’t unlock becuase the first one is broken.
I don’t think you guys should be looking for any pats on the back here and I hope like hell you all were not running around the office high fiving each other when you finally managed to get the damn servers up for everfrost THE DAY AFTER THE LAUNCH.
/rant
it’s all well and good that they are NOW FINALLY looking into the problem, which they acknowledge has been going on for weeks – and they’ve also acknowledged that they couldn’t replicate in-house. now they’re going to add lines of code to help trouble shoot – usually writing a message into a file or a queue – so this will make matters worse on freeport.
i hope that when they have an idea, they TAKE THE SERVER DOWN in a timely manner to install a hotfix, and not make use wait through the weekend.
hopefully they are completely embarrassed by this xpac installation – i didn’t hear a lot about wow people being unable to get on with cataclysm, and i don’t expect the same thing when rift starts up today.
hopefully it will lead to some sort of improvement, maybe with some added resources.
Its depressing to read these “sack them” or other insulting comments on the forums, its pretty much a given that these types of morons aren’t going to ever personally meet a dev, or another player. If we were able to give them the cold shoulder in the game it might make them think before posting, not being able to is certainly a downside of anonymity.
I don’t blame the devs. It’s pretty obvious that if this team is putting in 16+ hour days, they don’t have the resources and manpower they *should* have, but why would Sony spend money to improve the quality of their service for its customers? That would imply they give a damn.
I have to agree with Arcturys on this one. It is not the devs’ faults. They are doing the best they can to fix the mishandling of this release by Sony. Being an IT Director for some years, I suspect resources are a big issue. As every company cuts back, this is where it hurts the most. Clearly the signs are there: server merges galore, release of xpacs before they are fully tested and ready, equipment woes, etc. The list goes on and on.
It’s certainly frustrating to the end-user, those who fuel Sony’s engine with money. BTW, I was one of the “unfortunate” veterans who had a problem with my Velious purchase. So what did I do, I gave them another $30 to get the “in-game” rewards I was supposed to get for $60 along with a game credit. Now if I could just get into the game and get the rewards!
I work as an IT Manager, and I know the pain in the ass that is the corporate penny pusher. While I don’t hold the dev’s soley responsible for this Lemon Launch, I do hold Sony responsible.
I am also very annoyed that they’d try to reach out for a pat on the back or handouts, the only reward they should be getting is the pride of FINALLY fixing something. Not the fact that they spent 16 hours… Big freakin whoop, I’ve done months of 18+ hours just to fix a network that I had taken over. I didn’t go around to the customers and employees asking for a medal, pat on the back, or any form of recognition, because that was my job.
Oh BOO freakin HOO…I work 12-16 hours everyday…My only joy when i get off is to come home to play EQ for a couple of hours. So when i get home and the servers are down AGAIN, and i have to hear a couple of whiney devs ( who took 2 xtra weeks to develop, and how is that working out for ya) whine, ” Please don’t pick on us we are working hard”. Freakin amazes me!!! You know what happens when people in the real world continue to screw up…we get canned! Yes fired,sacked,axed, or whatever you want to call it.For someone who is providing a service for money aka the customer service industry… you have no clue what makes your customers(your money) happy. Here is a hint…It’s not excuses and whining its results and service….whew I’m done I think
The grumpy ones like Jojo are experiencing withdrawals. It’s the same sort of irritability and whining that an alcoholic or meth addict would go through when they can’t get their fix. Wa wa waaaa. Go get some sleep so you won’t be so negative.
Feldon needs to put an “add cheese” button to go with all of this whine. If the same problem was on every server I might agree with some of the premature launch criticisms. It’s not, so anyone with a brain and a job can empathize with the developers here.