Dethdlr 
Ok, maybe not “another weekend” but it seemed to fit in with the theme from our last article.
From SOE’s Facebook page as of 9:13 pm EDT:
We wanted to let you know that our games will not be up this weekend. We are working around the clock to get our services back up and running soon. Thank you for your continued patience and support.
Would be interesting to know how much all this downtime is costing Sony, all future class action lawsuits aside.
How much? is a great question!
How many players are going to other games?
thats gotta pull on the pocket threads….
I hope they do some work on the game with all this downtime they are having. We shouldnt need to have a HotFix for a couple months. I know..I know..but I like to dream.
I don’t know or care how much it is costing them but I am starting to care about the lack of a time frame for getting this thing back up and running, we have had what 4 days of “not tonight” before they decide “not this weekend” how about tentative guess for a change maybe a “all going well late next week”.
I don’t expect exact dates and times but a rough ETA I think would be in order.
also now they changed from “not today” to “not this weekend”… a maybe monday to “not this week” ?:(
well next week i expect more of the same for 5 days. Not today I begining to think the end of may.
It is getting to the point that if the servers were up I would cancel my account till they get there crap together. 🙁
It’s hilarious how they are saying the exact same thing near verbatim and still thinking they are keeping us updated.
IT IS NOT HARD FOR A MULTINATIONAL CORPORATION TO WORK AROUND THE CLOCK, they do it anyways.
Hell, corporation I work for that is NA only only has 3-4 hours a day when no one is on-the clock.
Of course the severs sent going to be up. It’s the weekend, and no one works on the weekend. They’ll be back to work on the issue Monday morning.
Having an ETA is BASIC to project management. It’s perfectly fine to not hit an ETA due to unforeseen difficulties, but not to publish an ETA indicates to me at least, that something other than letting the engineers do their job is happening (ie something political).
Publish an ETA Sony! If you don’t hit it, you don’t hit it, but it looks bad not to have one.
This isn’t “Basic Project Management” though,this is fire-fighting.
I’d imagine some delays are due to ordering / configuring new hardware (a decent harware firewall to start), nevermind training their staff on how to upgrade existing software and finally, and most importantly, testing how it all works together. There is no point in rushing out a solution only to see it either crash and burn or get re-hacked.
I’m as annoyed by the delays as everyone else, but in reality they have ONE opportunity to get things up and running right; if it comes back up with the same problems then their reputation would take an even bigger hit.
There really is no sun here right now!
It is good that they told us that it will not be up this weekend, though. It stops me from checking every 5 minutes. I will go see some friends 🙂
I’m just hoping the (two?) devs are hard at work, and not hanging around the watercooler gossiping /whip
I’ve been browsing through my screenshot folder to answer the question on another website (guildm8’s) about “how many people will quit over this”, I know I’m a big fan, but I’m also just not seeing anything else close to EQ2 worth playing (AoC is the only one that tempts graphically for me). I.e. somewhere else that has Frogloks, Ogres that look big, Trolls that look mean and storylines that someone appears to have cared about at somepoint 😉
Too many of the new ones seem to have the accounts department involved in the design process, or at least thats the only reason I can see for the lack of vision/imagination.
Plus nowhere else (apart from EQ1) has Frogloks…
Of course this is basic project management even if initially it was on “on the fly” from the moment they began to rectify the problem someone must have been developing the project so even if on day one or two they did not have a rough idea of the return to service date they must by now have one.
Most people I know are not going to hang around for a month of “maybe tomorrows”.
How could they possibly give an ETA when they don’t even know the extent of the security issues. They’re still in the investigatory phase as far as the public can tell from their statements, they obviously won’t know how long it’ll take to fix until they know what exactly needs fixing. If your car breaks down, you can’t expect an ETA on the repair time until you know why it broke down in the first place.
Of course, the “not today”, “not today”, “not today”, stuff is kind of dumb – but then again, there’s not much alternative. If they said “at least a week”, a lot of people would misinterpret that as “everything will be back up in 1 week” and it’d just create even more forum drama if they aren’t up by then. I guess they could do the Scotty approach and say something like “3 months”, then seem like miracle workers when they get it back up in 2 weeks…
Play http://www.project1999.org/
I don’t expect an ETA. There’s no way to know when it WILL be done. But they do know when it WON’T be done and they’ve been piss poor at telling us that.
Im alive got a suntan read most of my books and been cycling lost a few pounds now for my dollars worth on this situation. Hindsight is a wonderful thing basic understanding is a requirement having not invested heavily in the architecture of the SOE PSP PSN networks security or servers this is what you get.The corporation cutting so much profit and investing so little back into its products. If SOE had done the job properly this would never have happed cheap products break well made products don’t.
WoW gets attacked constantly every day hundreds of times per day they have never been breached.SOE will now be rewriting the hole script at a huge cost getting external teams in to fix this. This is typical corporate mentality showing SOE products are not well maintained after release. If you buy a washing machine TV VCR or any other product in the UK it has a 5 years warranty by law you can get it repaired at anytime and if they cant the have to replace the product with a new one or cash refund.
In this case we have bought the product but also pay a sub to keep the game in good health from a security stand point and additional content. I personally think that they are suable for the loss of our personal information i have had my details used illegally.Do i have a right to be angry ? well if i pay for a service and the company is not using some of that money to keep its products up-to date and safe then i think i have a legitimate complaint.
I was a COBOL programmer for 24 years. During that time I did my share of fighting fires. I can tell you from an it perspective you had 2 days to get yous stuff back up. Having your CICS down for more than 24 hours got folks fired. Having your info security down for more than a few hours, got folks fired. I can say from an it perspective this kind of outage seams than more of a hack. I think something else might be going on.
To me it is starting to look like some restores of files that might have been deleted. It takes a while to restore a database if it get destroyed. That depends on how often your database is backed up, and how often you set the check point restores. Or some hardware rebuilds might be taking place.
It should not take this long to get firewall upgrades, and come up with new security to put in place. Hardware should have taken hours as you have the new stuff in place ready to go to swap out.
All I can say is we keep getting the same thing, not today. It not good enough at this point.
I would just like them to come clean. What are the problem, why is it taking so long, it just all speaks of incompetence.
I’ve estimated that this outage is costing them in revenue somewhere around 250K per day for their core products alone. It is certainly more though. I estimate that EQ2 and EQ have a combined population of 400K. Vanguard and SWG probably add another 100. These figures don’t account for PoTC, Freeworld, DC universe, nor there secondary products like Pox Nora.
My guess is they there are legal considerations and I’ll also bet they have an insurance policy to cover at least a portion of this loss. Thus, they will need to do as much research and documentation as possible to make the claim. I am certain their in house counsel (and I wouldn’t surprised if they brought in some outside counsel as well to deal with this) are advising them on what to do and most likely preparing for a regulatory smack down.
There is only so much cash a company can loose before it crosses the line. I think the 250k per day is short a good deal just for the PC side. I bet they lost millions.
The real issue now becomes how long of a sustained outage can they suffer before they call it and turn the servers off.
Not only are they going to have to deal with the loss of customers, the lawsuits, they are going to HAVE to throw those of us who DO return a bone, hell a whole SKELETON.
If they just turn on the switch when everything’s hunky/dorey and say: “Ok, we’re back up, sorry ’bout dat, here ya go”. I know I’ll leave. There better be some serious compensation in the works (aside from the Month free and 1 day per downtime free – and no bonus XP weekends!)
This is what was posted last night: MAY 7, 2011, 1:39 A.M. ET
Sony Delays Game Network Restart
TOKYO — Sony Corp. is unlikely to meet the deadline it had set to restore the PlayStation Network, keeping millions of irritated videogamers offline for another weekend as the company continues to conduct internal safety checks to defend a new system against hackers.
A Sony spokesman said Saturday the PlayStation Network will most likely not be back online over the weekend, a step back from its announcement May 1 when executives said they were aiming to get parts of the service back up within the week, while a full system reboot would take until the end of the month.
If you look now its has changed : http://online.wsj.com/article/SB10001424052748703859304576308260949123474.html?mod=WSJ_Tech_LEFTTopNews
My comment above is daily revenue lost, and almost certainly underestimated. 250K burn rate is a non-trivial amount even for a very large corporation, let along a small unit within a large corporation. In four days that’s annual salary for more than 10 people. Considering the margins are rather slim, I am sure they are there are a lot of white knuckles.
There are other loses here as well, such as damage to their brand, and damage to their core business model. I think it’s a safe conclusion that something less than 100% of the players will return at this point.
At a minimum certain other players will pick up other games in the interim and decide they prefer those games.
I think for games on their last legs, such as Vanguard and SWG, I think this could be the nail in the coffin. They will bring up the servers, but I think this could shatter an already fracturing community.
I don’t think they will ever get to a point where they say “no, we aren’t going to bring them online”. That is rather silly. They might try to sell off either the entire SOE unit or certain IPs to other competitors. In fact, I think this is highly likely in the next few months after things come back online. The parent (Sony) may decide that the the online business is a distraction from their core business of selling electronics and sell the property to another firm.
I could easily see them looking for a partner to purchase Vanguard or SWG and possibly retain production rights or something.
Whats interesting is the very last part” A full system reboot would take to the end of the month”. This BS is just completely out of control…just sell the EQ2 franchise to a well established Company that can and does care about the customer. A company that will communicate a company that will say:
Due to our ongoing investigation we cannot set a actual time frame, But the online services will not be up for approximately a month.
The lack of communication is nothing new. Look back a bit to the eartquake and its related issues and how that was handled on a communicationing to citizens level.
I wouldn’t expect a fixed time table for the return of services as that would be rather suicidal for SOE however I can not see why some sort of general expectation can not be given ie end of next week, end of next month or end of next year all going well but I am certain that this one day at a time thing they seem to be trying is going annoy people enough to leave.
At this point the biggest problem is people people picking up F2P games like LotRO, the longer these ones are off line the more people are going to invest time into toons there and they will realise that not needing a CC to play is a far safer option than hoping like hell Sony can sort this out as there has been a sizable amount of trust lost through this.
In short time is not on SOE’s side and the longer the less information they give us the less time they will have to keep people.
Lets think about this for a minute.
Security Audit 2 days for a complete network and server scan + compiling the reports.
half Day to a Day going tho the reports and getting the tasks handed out.
Two Days working out solutions and making sure they won’t effect normal traffic.
Network Software upgrades, large network 24/7 2-4 days.
Network Load and Secuirty Testing.. 2 days.
Database Upgrades and Reencrypting at a higher Encryption Level or with new Keys. 4-6 days.
Installing new equipment to deal witht the higher encryption level if applied 2-3 days.
Database load and Security Testing… 2 days.
Applying secuirty Updates and Testing.. 4 days.
Creating new firewall rulebases and testing… 3 days.
Rewriting Software to deal with new keys or new encryption settings… (1-2 days per game)
Testing the new Patches (per game 1-2 days each)
even doing these all at the same time, it all addes up and if a test fails u got to go to step #1 and start again.
Facebook links on last two stories to include them are missing http:// prefixes making them relative and linking to the wrong place
Finally got “The Beast” running and ready to FPS the hell out of SG or anything SOE can dream up! Looks like SOE has fabricated a story about hackers rather than face my rig!
…but how many days does it take to reticulate splines?
I completely blame the hackers for this mess. However, the communication to (us) the consumers from Sony has been horrible and I’m starting to really get upset about it. At the very lest they cold bring a test server online or something so we could make alts and play them temporarily.
“This isn’t “Basic Project Management” though,this is fire-fighting.
I’d imagine some delays are due to ordering / configuring new hardware (a decent harware firewall to start), nevermind training their staff on how to upgrade existing software and finally, and most importantly, testing how it all works together. There is no point in rushing out a solution only to see it either crash and burn or get re-hacked.
I’m as annoyed by the delays as everyone else, but in reality they have ONE opportunity to get things up and running right; if it comes back up with the same problems then their reputation would take an even bigger hit.”
can you say, “Fukushima Maru”?
I think most are missing the point, it doesn’t matter how long it takes to fix as much as it matters that they wont give some idea when they expect it to back on line.
I have no issues with them coming back with something like “we expect services will be returned by the end of next week (month, year, century insert what ever is realistic) all going well” but I object to being strung along with “maybe tomorrow” when they know that it will take far longer.