GoDaddy Outage, Anonymous Member Claims Responsibility

Lazybones · September 10, 2012, 02:49:03 PM

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/

If you where wondering why the forums where having issues.

Mr. Analog · September 10, 2012, 03:14:08 PM

Ah, that explains it

Darren Dirt · September 10, 2012, 03:36:56 PM

Quote from: Lazybones on September 10, 2012, 02:49:03 PM
http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/

also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago but not your OWN "news releases" page?

"@film_girl I'm taking godaddy down bacause well i'd like to test how the cyber security is safe and for more reasons that i can not talk now"
- http://twitter.com/AnonymousOwn3r/statuses/245234582205652992

Lee Nicholson ‏@filmoreclark
"@AnonymousOwn3r Please elaborate on how this is not totally self serving on your part?"

Mr. Analog · September 10, 2012, 03:51:36 PM

Quick, someone e-mail them and let them know their network is down

Tom · September 10, 2012, 03:52:28 PM

Hah. Man I'm glad I switched away.

Melbosa · September 10, 2012, 04:40:06 PM

Quote from: Darren Dirt on September 10, 2012, 03:36:56 PM
also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago but not your OWN "news releases" page?

Wow, when was the last time you worked on an IT disaster at a company? Facebook/Twitter was GoDaddy's only communication medium all day, and I'm sure even 33 minutes after they said it was all good on Facebook, they were still working with companies and calls that it hasn't been addressed. I feel for their IT department!

Darren Dirt · September 10, 2012, 05:09:05 PM

Quote from: Melbosa on September 10, 2012, 04:40:06 PM
Quote from: Darren Dirt on September 10, 2012, 03:36:56 PM
also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago but not your OWN "news releases" page?

Wow, when was the last time you worked on an IT disaster at a company? Facebook/Twitter was GoDaddy's only communication medium all day, and I'm sure even 33 minutes after they said it was all good on Facebook, they were still working with companies and calls that it hasn't been addressed. I feel for their IT department!

Just think it's funny that on GoDaddy's main page AND on the news page they had no mention of CURRENT issues their PAYING CLIENTS were experiencing.

And yet they weren't playing "say nothing and it won't be that big of a problem", instead they DID mention it on their Facebook page. Neither or both, just sayin'.

Thorin · September 10, 2012, 05:34:26 PM

Really? Neither or both? How about they're trying to update people as quickly as possible? For all we know it takes longer to update their own news feed, or maybe the Facebook page updater doesn't have the access required to update their own news feed... I dunno, that seemed like a pretty flimsy thing to complain about. Now, if you complained about how easily their name resolution servers got taken out, well, that's something worth complaining about.

Mr. Analog · September 10, 2012, 05:39:42 PM

GoDaddy employs press agents, they could have had the word out as soon as they knew what was up, my suspicion is that they've already lost a lot of trust with the whole SOPA thing, having Anon hate show them as a weakness would push a lot of customers over the edge so they kept it on the QT.

Melbosa · September 10, 2012, 05:53:40 PM

I suspect this won't be kept on the down low or low key. It was pretty covered by all news sites and I suspect possible new cast coverage as well.

I suspect Thorin is more correct with the access problems or just not there yet.

Tom · September 10, 2012, 05:59:19 PM

Their entire media/relations team could be too busy to update the site.. But you'd think they'd have a decent news feed setup where you could just make the same post you made to facebook saying "Something is up, we're working on it, we'll keep everyone updated" rather than silence on their own official site.

Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.

Lazybones · September 10, 2012, 06:11:45 PM

http://www.wired.com/wiredenterprise/2012/09/godaddy-moves-to-verisign/

Apparently the fix was to redirect dns.

Melbosa · September 10, 2012, 09:02:53 PM

Quote from: Tom on September 10, 2012, 05:59:19 PM
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.

This is true, but again I think Thorin hit in on the noise. It probably came down to timing, access or just plan missed. Use what was working and fix the fires until no fires left.

I'll tell you if this happened at my office, my media people would be not so much worried about our online feeds, and neither would I be. Our first response would be to inform our client base what is going on the best way we could think of, then work on a solution. Once the solution to the major problem was fixed, we would informed using the same channels we did in the first place as people would be expecting that is how we would update them. Next we would make sure our support channels had been updated (and this isn't our news feeds or what's happening here today sites) to say what happened, and what to expect with our services - so read this as our internal ticketing software and phone support lines - the two things we will need to better assess the damages and respond to our client base's needs.

Next our media people would be working on how best to counter the bad press, ensure our corporate image is looked after and our customer relations are taking into account - that is contacting our major clients, then mediums, then minors to better work with them to make sure they get what they need out of this time of turmoil.

Our IT people would be working on the other services that would be down or the other fires that would have started due to one piece of infrastructure missing. Could take hours, could take days, but as soon as we were live and mostly working to the world, we would focus inward until everything was sorted and all client facing services were running properly, then we would deal with any "how, where, why, and story telling elements".

So being that I've been through 5 unexpected power disruptions to our core data centres in the last 3 years, I can tell you this is how we react at the IT level at NAIT. While we may not be as large as GoDaddy, or as funded, we still operate at a large company style business. Even though the General Public has a right to know why, especially that we are publicly funded, they usually are the last to know or be informed.

I guessed that the DNS redirect was going to be the solution. Now the question is... Verisign next?

Lazybones · September 10, 2012, 09:10:19 PM

Verisign runs there own DDOS mitigation service, the redirect might be due to that as DNS is usually redirected as part of those services.

Tom · September 10, 2012, 09:25:00 PM

Quote from: Melbosa on September 10, 2012, 09:02:53 PM
Quote from: Tom on September 10, 2012, 05:59:19 PM
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.

This is true, but again I think Thorin hit in on the noise. It probably came down to timing, access or just plan missed. Use what was working and fix the fires until no fires left.

I'll tell you if this happened at my office, my media people would be not so much worried about our online feeds, and neither would I be. Our first response would be to inform our client base what is going on the best way we could think of, then work on a solution. Once the solution to the major problem was fixed, we would informed using the same channels we did in the first place as people would be expecting that is how we would update them. Next we would make sure our support channels had been updated (and this isn't our news feeds or what's happening here today sites) to say what happened, and what to expect with our services - so read this as our internal ticketing software and phone support lines - the two things we will need to better assess the damages and respond to our client base's needs.

Next our media people would be working on how best to counter the bad press, ensure our corporate image is looked after and our customer relations are taking into account - that is contacting our major clients, then mediums, then minors to better work with them to make sure they get what they need out of this time of turmoil.

Our IT people would be working on the other services that would be down or the other fires that would have started due to one piece of infrastructure missing. Could take hours, could take days, but as soon as we were live and mostly working to the world, we would focus inward until everything was sorted and all client facing services were running properly, then we would deal with any "how, where, why, and story telling elements".

So being that I've been through 5 unexpected power disruptions to our core data centres in the last 3 years, I can tell you this is how we react at the IT level at NAIT. While we may not be as large as GoDaddy, or as funded, we still operate at a large company style business. Even though the General Public has a right to know why, especially that we are publicly funded, they usually are the last to know or be informed.

I guessed that the DNS redirect was going to be the solution. Now the question is... Verisign next?

I can understand the entire technical and most/all of the management being completely focused on getting things fixed, but what does the marketing and customer service leads have to do /other/ than keep in contact with the customers when there's a large problem like this? If your main site is still up, that is the /first/ place you should post information as a service provider. That is where everyone is going to look. If your site is down, you need to contact media to get the news out. Putting it up on one site isn't the greatest idea. Though at least it was a site with a fair number of users. But I never saw anything regarding godaddy on facebook and I was a customer, and have a ton of people I follow who are in IT. It should only take one customer relations staffer to call up and/or email various media outlets giving them a heads up. The rest can deal with contacting customers directly (as they likely contacted all of their largest/most-important customers first).

Or at least that's how I see it. I assume NAIT doesn't have anywhere near as large of a customer relations team /per user/ as godaddy should. So its understandable that pretty much everyone on NAITs staff would be swamped with actually working on the problem, and putting out direct fires (ie: upper management).

And there's really no need for "Story telling". What I'm saying is they should have at least acknowledged the problem and communicated to their customers that they were working on it. And all anyone got for hours was a facebook posting.