http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
If you where wondering why the forums where having issues.
Ah, that explains it
Quote from: Lazybones on September 10, 2012, 02:49:03 PM
http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
(http://24.media.tumblr.com/tumblr_ljtxocThnE1qa1saxo1_500.jpg)
also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago (http://www.facebook.com/GoDaddy/posts/10151014860750686) but not your OWN "news releases (http://www.godaddy.com/newscenter/releases.aspx?ci=9081)" page?
"@film_girl I'm taking godaddy down bacause well i'd like to test how the cyber security is safe and for more reasons that i can not talk now"
- http://twitter.com/AnonymousOwn3r/statuses/245234582205652992
Lee Nicholson @filmoreclark
"@AnonymousOwn3r Please elaborate on how this is not totally self serving on your part?"
Quick, someone e-mail them and let them know their network is down :)
Hah. Man I'm glad I switched away.
Quote from: Darren Dirt on September 10, 2012, 03:36:56 PM
also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago (http://www.facebook.com/GoDaddy/posts/10151014860750686) but not your OWN "news releases (http://www.godaddy.com/newscenter/releases.aspx?ci=9081)" page?
Wow, when was the last time you worked on an IT disaster at a company? Facebook/Twitter was GoDaddy's only communication medium all day, and I'm sure even 33 minutes after they said it was all good on Facebook, they were still working with companies and calls that it hasn't been addressed. I feel for their IT department!
Quote from: Melbosa on September 10, 2012, 04:40:06 PM
Quote from: Darren Dirt on September 10, 2012, 03:36:56 PM
also, nice going GoDaddy! You update your FACEBOOK 33 minutes ago (http://www.facebook.com/GoDaddy/posts/10151014860750686) but not your OWN "news releases (http://www.godaddy.com/newscenter/releases.aspx?ci=9081)" page?
Wow, when was the last time you worked on an IT disaster at a company? Facebook/Twitter was GoDaddy's only communication medium all day, and I'm sure even 33 minutes after they said it was all good on Facebook, they were still working with companies and calls that it hasn't been addressed. I feel for their IT department!
Just think it's funny that on GoDaddy's main page AND on the news page they had no mention of CURRENT issues their PAYING CLIENTS were experiencing.
And yet they weren't playing "say nothing and it won't be that big of a problem", instead they DID mention it on their Facebook page. Neither or both, just sayin'.
Really? Neither or both? How about they're trying to update people as quickly as possible? For all we know it takes longer to update their own news feed, or maybe the Facebook page updater doesn't have the access required to update their own news feed... I dunno, that seemed like a pretty flimsy thing to complain about. Now, if you complained about how easily their name resolution servers got taken out, well, that's something worth complaining about.
GoDaddy employs press agents, they could have had the word out as soon as they knew what was up, my suspicion is that they've already lost a lot of trust with the whole SOPA thing, having Anon hate show them as a weakness would push a lot of customers over the edge so they kept it on the QT.
I suspect this won't be kept on the down low or low key. It was pretty covered by all news sites and I suspect possible new cast coverage as well.
I suspect Thorin is more correct with the access problems or just not there yet.
Their entire media/relations team could be too busy to update the site.. But you'd think they'd have a decent news feed setup where you could just make the same post you made to facebook saying "Something is up, we're working on it, we'll keep everyone updated" rather than silence on their own official site.
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.
http://www.wired.com/wiredenterprise/2012/09/godaddy-moves-to-verisign/
Apparently the fix was to redirect dns.
Quote from: Tom on September 10, 2012, 05:59:19 PM
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.
This is true, but again I think Thorin hit in on the noise. It probably came down to timing, access or just plan missed. Use what was working and fix the fires until no fires left.
I'll tell you if this happened at my office, my media people would be not so much worried about our online feeds, and neither would I be. Our first response would be to inform our client base what is going on the best way we could think of, then work on a solution. Once the solution to the major problem was fixed, we would informed using the same channels we did in the first place as people would be expecting that is how we would update them. Next we would make sure our support channels had been updated (and this isn't our news feeds or what's happening here today sites) to say what happened, and what to expect with our services - so read this as our internal ticketing software and phone support lines - the two things we will need to better assess the damages and respond to our client base's needs.
Next our media people would be working on how best to counter the bad press, ensure our corporate image is looked after and our customer relations are taking into account - that is contacting our major clients, then mediums, then minors to better work with them to make sure they get what they need out of this time of turmoil.
Our IT people would be working on the other services that would be down or the other fires that would have started due to one piece of infrastructure missing. Could take hours, could take days, but as soon as we were live and mostly working to the world, we would focus inward until everything was sorted and all client facing services were running properly, then we would deal with any "how, where, why, and story telling elements".
So being that I've been through 5 unexpected power disruptions to our core data centres in the last 3 years, I can tell you this is how we react at the IT level at NAIT. While we may not be as large as GoDaddy, or as funded, we still operate at a large company style business. Even though the General Public has a right to know why, especially that we are publicly funded, they usually are the last to know or be informed.
I guessed that the DNS redirect was going to be the solution. Now the question is... Verisign next?
Verisign runs there own DDOS mitigation service, the redirect might be due to that as DNS is usually redirected as part of those services.
Quote from: Melbosa on September 10, 2012, 09:02:53 PM
Quote from: Tom on September 10, 2012, 05:59:19 PM
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.
This is true, but again I think Thorin hit in on the noise. It probably came down to timing, access or just plan missed. Use what was working and fix the fires until no fires left.
I'll tell you if this happened at my office, my media people would be not so much worried about our online feeds, and neither would I be. Our first response would be to inform our client base what is going on the best way we could think of, then work on a solution. Once the solution to the major problem was fixed, we would informed using the same channels we did in the first place as people would be expecting that is how we would update them. Next we would make sure our support channels had been updated (and this isn't our news feeds or what's happening here today sites) to say what happened, and what to expect with our services - so read this as our internal ticketing software and phone support lines - the two things we will need to better assess the damages and respond to our client base's needs.
Next our media people would be working on how best to counter the bad press, ensure our corporate image is looked after and our customer relations are taking into account - that is contacting our major clients, then mediums, then minors to better work with them to make sure they get what they need out of this time of turmoil.
Our IT people would be working on the other services that would be down or the other fires that would have started due to one piece of infrastructure missing. Could take hours, could take days, but as soon as we were live and mostly working to the world, we would focus inward until everything was sorted and all client facing services were running properly, then we would deal with any "how, where, why, and story telling elements".
So being that I've been through 5 unexpected power disruptions to our core data centres in the last 3 years, I can tell you this is how we react at the IT level at NAIT. While we may not be as large as GoDaddy, or as funded, we still operate at a large company style business. Even though the General Public has a right to know why, especially that we are publicly funded, they usually are the last to know or be informed.
I guessed that the DNS redirect was going to be the solution. Now the question is... Verisign next?
I can understand the entire technical and most/all of the management being completely focused on getting things fixed, but what does the marketing and customer service leads have to do /other/ than keep in contact with the customers when there's a large problem like this? If your main site is still up, that is the /first/ place you should post information as a service provider. That is where everyone is going to look. If your site is down, you need to contact media to get the news out. Putting it up on one site isn't the greatest idea. Though at least it was a site with a fair number of users. But I never saw anything regarding godaddy on facebook and I was a customer, and have a ton of people I follow who are in IT. It should only take one customer relations staffer to call up and/or email various media outlets giving them a heads up. The rest can deal with contacting customers directly (as they likely contacted all of their largest/most-important customers first).
Or at least that's how I see it. I assume NAIT doesn't have anywhere near as large of a customer relations team /per user/ as godaddy should. So its understandable that pretty much everyone on NAITs staff would be swamped with actually working on the problem, and putting out direct fires (ie: upper management).
And there's really no need for "Story telling". What I'm saying is they should have at least acknowledged the problem and communicated to their customers that they were working on it. And all anyone got for hours was a facebook posting.
Quote from: Tom on September 10, 2012, 05:59:19 PM
Their entire media/relations team could be too busy to update the site.. But you'd think they'd have a decent news feed setup where you could just make the same post you made to facebook saying "Something is up, we're working on it, we'll keep everyone updated" rather than silence on their own official site.
Not everyone has facebook. Nor get godaddy's posts in their feed. So over all its a pretty poor place to post news as the only source.
this.
the text already existed shared with the public. posted on Facebook. Not anywhere on their main page (which was serving out to the public -- if it was actually DOWN for the public, fine, I understand not pasting in that same text).
Not complaining btw, just observing the chaos. and the continual loss of a business' remaining good faith.
Quote from: Melbosa on September 10, 2012, 09:02:53 PM
This is true, but again I think Thorin hit in on the noise. It probably came down to timing, access or just plan missed. Use what was working and fix the fires until no fires left.
I'll tell you if this happened at my office, my media people would be not so much worried about our online feeds, and neither would I be.
btw I wasn't using some news FEED reader, I just went to "godaddy.com" and saw nothing there, and clicked the "NEWS" linkand saw nothing there more recent than September 1st. Then I commented on it. #nothingtoseeheremovealong
Quote from: Melbosa on September 10, 2012, 09:02:53 PM
So being that I've been through 5 unexpected power disruptions to our core data centres in the last 3 years, I can tell you this is how we react at the IT level at NAIT. While we may not be as large as GoDaddy, or as funded, we still operate at a large company style business. Even though the General Public has a right to know why, especially that we are publicly funded, they usually are the last to know or be informed.
Forget power disruptions. What if your DB went down (or the website's connection to it). Would you not think the visitors to your still-up website expect some kind of text at the top, or on your "news" page, to at least say "we are experiencing technical issues, working on resolving as quickly as possible" type of thing.
OTHERWISE (here's the motivation for my sharing of my observation) ... Jo Public thinks the problem is on HIS/HER end, not on the business' side. And later finding out it wasn't, well there goes some warm and fuzzy feelings for the business for not keeping the CLIENTS (paying and non) informed asap. Even if not intentionally keeping the public in the dark, due to some kind of fear or whatever.
Quote from: Tom on September 10, 2012, 09:25:00 PM
And there's really no need for "Story telling". What I'm saying is they should have at least acknowledged the problem and communicated to their customers that they were working on it. And all anyone got for hours was a facebook posting.
Yeah,
this. There were surely MANY customers (and potential customers) who visited the GoDaddy website and were expecting to see either a DOWN website, or something like this:
(http://www.wired.com/wiredenterprise/wp-content/uploads//2012/09/godaddy-down-cropped2.png) (http://www.wired.com/wiredenterprise/wp-content/uploads//2012/09/godaddy-down-cropped2.png)
Instead they saw NOTHING acknowledging an issue. For way too long a time.
maybe here's the REAL reason for the attack against GoDaddy...
http://www.google.com/search?q=elephant+Bob+Parsons
Yeah, we'll see if this "AnonymousOwn3r" decides to share his real reasons for the attack with the rest of the world. Seemed kinda out of left field, to me. GoDaddy hasn't made the news for anything terrible recently (in Internet time, their support for SOPA was eons ago!)
Quote from: Melbosa on September 10, 2012, 09:02:53 PM
It probably came down to timing, access or just plan missed.
mighta been technical issues, but then again sometimes big companies just make BAD DECISIONS:
http://www.imdb.com/help/show_leaf?facebookcomments
So I got this email today:
QuoteDear Customer,
We owe you a big apology for the intermittent service outages we experienced on September 10 that may have impacted your website, your email and other Go Daddy services.
We let you down and we know it. We take our responsibilities ? and the trust you place in us ? very seriously. I cannot express how sorry I am to those of you who were inconvenienced.
The service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented a series of immediate measures to fix the problem.
At no time was any sensitive customer information, including credit card data, passwords or names and addresses, compromised.
Throughout our history, we have provided 99.999% uptime in our DNS infrastructure. This is the level of performance we expect from ourselves. Monday, we fell short of these expectations. We have learned from this event and will use it to drive improvement in our services.
As a result of this disruption, your account will be credited for the value of 1-month of service for each of your active/published sites.* This credit will be available to you for the next 7 days. Please click the button below to redeem your credit.
[ Button Would Be Here ]
It's an honor to serve you. Thank you for the opportunity to re-earn your business and trust.
As always, please call us 24/7 at 1-480-505-8877 ? anytime, for any reason.
Sincerely,
Scott Wagner
CEO
GoDaddy.com
First time that has happened due to a DDOS attack.
Wasn't there a press release yesterday claiming that it was an internal networking issue?
Edit: There was http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
Yep even in that email it says internal error. But I haven't seen a retraction by the Anonymous guy yet, so maybe its a cover up.
I doubt very much the guy would post a retraction as it works in his favor to cast that shadow of doubt
Maybe he did it maybe he didn't either way you slice it it's bad news for GoDaddy (either it's a lack of DDOS protection or internal incompetence, not sure which is worse)
:(
I'm inclined to believe its incompetence. And I think it actually was good of Godaddy to admit it. Own up to your issues and solve em. Good on em.
Anonymous0wn3r has released what he claims to be logs that show SQL injection vulnerabilities, put up on pastebin and similar. Apparently he's also distributed a couple of screen shots of him apparently putting messages up on the GoDaddy main page, although that appears to have gone through a querystring injection and then it only shows up for him anyway. At least, that's what the screenshots showed. Running a sql injection vulnerability finder script and posting the output and injecting some text into a web page via a querystring smacks of script-kiddie-ness, and is nowhere the same as orchestrating a massive DDoS attack that takes out all three of a registrar/hoster's root DNS servers. This is why I have trouble believing it was a hacker over believing it was internal problems.
Interestingly, the first tweet from Anonymous0wn3r on this subject (claiming responsibility) was about an hour and a half after the first tweet from GoDaddy on this subject (telling clients they knew something was wrong). Whereas Anonymous always seemed to take credit before the results of their actions were reported anywhere.
Anyway, we'll see if this truly was a hack or just internal problems - Anonymous0wn3r tweeted that he'd have to do it again just to prove it was him.
also notice that in this case, Anonymous0wn3r made it very clear that it was a solo project, not in conjunction with "Anonymous" as a whole (not that there is any kind of central planning anyway)
Quote from: Thorin on September 12, 2012, 02:04:43 PM
Anonymous0wn3r has released what he claims to be logs that show SQL injection vulnerabilities, put up on pastebin and similar. Apparently he's also distributed a couple of screen shots of him apparently putting messages up on the GoDaddy main page, although that appears to have gone through a querystring injection and then it only shows up for him anyway. At least, that's what the screenshots showed. Running a sql injection vulnerability finder script and posting the output and injecting some text into a web page via a querystring smacks of script-kiddie-ness, and is nowhere the same as orchestrating a massive DDoS attack that takes out all three of a registrar/hoster's root DNS servers. This is why I have trouble believing it was a hacker over believing it was internal problems.
Interestingly, the first tweet from Anonymous0wn3r on this subject (claiming responsibility) was about an hour and a half after the first tweet from GoDaddy on this subject (telling clients they knew something was wrong). Whereas Anonymous always seemed to take credit before the results of their actions were reported anywhere.
Anyway, we'll see if this truly was a hack or just internal problems - Anonymous0wn3r tweeted that he'd have to do it again just to prove it was him.
Any script kiddy can do a DDoS. Just fire up the Low Orbit ION Cannon (http://en.wikipedia.org/wiki/LOIC) and you have an instant DDoS.
But your evidence makes me lean harder into thinking it was incompetence.
Also, this kid said this was him alone and not Anonymous, so it makes sense that it wouldn't look like a standard Anonymous op if he indeed did do something.
Have to wonder though, whats worse PR wise, a one time snafu, or a massive DDoS and "hack". A mistake is human, but an ongoing DDoS is forever ;)
That's exactly what I was thinking, kind of a "damned if you do, damned if you don't" kinda deal really.
A single script kiddie running a LOIC wouldn't take out all three of their name servers. At least, I certainly hope not. Anyway, someone who brags about being able to put some text on a web page but only on his copy of it? Not a particularly "leet" hacker in my books. That was something we would try back in '95.
Quote from: Thorin on September 12, 2012, 03:02:42 PM
A single script kiddie running a LOIC wouldn't take out all three of their name servers. At least, I certainly hope not.
Thats the fun part about LOIC. It commands /many/ machines that also happen to be running LOIC. One computer can't take out anything, but the LOIC network sure as heck can.
Quote from: Thorin on September 12, 2012, 03:02:42 PM
Anyway, someone who brags about being able to put some text on a web page but only on his copy of it? Not a particularly "leet" hacker in my books. That was something we would try back in '95.
Yeah he doesn't sound all that impressive.