Critical Drobo Alert
Your Drobo: "Drobo1" has reported the following critical alert.
Red Alert. Drobo detected a hard drive failure. Replace the hard drive indicated by the blinking red light.
-----
That's the email I received today. Turns out one of the drives I bought eight days ago has already bitten the bullet. I wonder what MemEx's return policy is? If it's 7 days like BCOM used to be, then I better start me an RMA process.
<sigh> technology
I believe its 14 days. If you bought the insta RMA thing for $7-10 (I always do when I Get from memex) you get a new drive right away and they take care of the rest. Otherwise they will take your drive and tell you to come back to pick up a new drive after they've tested the one you gave them. Either way though its a max of a couple/few days without a drive.
Memex has been quite good to me, I expect they are still quite good about DOA hard drives.
I once had to RMA a hard drive /almost/ a couple weeks after I got it, and then RMA its replacement /almost/ a couple weeks later. The replacement's replacement is doing fine :)
7 days for returns.
30 days for replacements.
1 year for in-store warranty (where they'll RMA / advance-RMA for you).
I brought it in and said my Drobo says it's dead. They hooked it up to their test rig and it failed SeaTools in about a minute and a half. So I got a new drive on the spot.
So much easier than anything I've had to return to Future Shop and BCOM (god, remember the lack of customer service at that place?).
Drobo's happy again, busy adding data to the new drive.
Quote from: Thorin on July 07, 2012, 04:06:57 PM
7 days for returns.
30 days for replacements.
1 year for in-store warranty (where they'll RMA / advance-RMA for you).
I brought it in and said my Drobo says it's dead. They hooked it up to their test rig and it failed SeaTools in about a minute and a half. So I got a new drive on the spot.
So much easier than anything I've had to return to Future Shop and BCOM (god, remember the lack of customer service at that place?).
Drobo's happy again, busy adding data to the new drive.
Yeah, MemEx is quite good about RMA. I think the reason they didn't want to do my second replacement right away is cause it seemed a bit suspicious. And I agree with that. Don't have much problem with them taking the extra step of testing it fully before giving me a /third/ drive.
I think maybe its their "advanced replacement" service that is 14 days. That warantee they like to push on you ;D normally without it, they won't just give you a new drive on the spot. Maybe they will if the tech team in back isn't busy. I dunno.
Actually, their IPR plan has a _lot_ of advantages if you have problems, such as replacement with new (instead of refurbished), new matching parts if needed (new mobo doesn't support old CPU? Here's a new CPU too), much shorter turnaround time, and stuff like that.
I don't mind doing RMAs for drives when needed, so I don't pay the extra "lazy tax". But the extra support _almost_ makes me pay for it (but then, I remember I'm Dutch and don't spend money freely).
Quote from: Thorin on July 07, 2012, 05:08:52 PM
Actually, their IPR plan has a _lot_ of advantages if you have problems, such as replacement with new (instead of refurbished), new matching parts if needed (new mobo doesn't support old CPU? Here's a new CPU too), much shorter turnaround time, and stuff like that.
I don't mind doing RMAs for drives when needed, so I don't pay the extra "lazy tax". But the extra support _almost_ makes me pay for it (but then, I remember I'm Dutch and don't spend money freely).
Heh. Even when I was dead broke I paid for even NCIX's version of the IPR. It wasn't worth the extra hassle later on. Also ncix pays for return shipping on RMAs with their thing, so that helps.
Awesome! I'm glad everything worked out for you.
Oddly enough I have to to replace a drive on my Drobo, somehow filled up 2 TB already...
Quote from: Mr. Analog on July 08, 2012, 11:00:04 AM
Awesome! I'm glad everything worked out for you.
Oddly enough I have to to replace a drive on my Drobo, somehow filled up 2 TB already...
That's nothin.
Filesystem Size Used Avail Use% Mounted on
/dev/md2 5.5T 4.9T 647G 89% /mnt/boris/data
Anime:
2.3T video/Anime/
254G BitTorrent/00 Anime/
196G BitTorrent/
2.8T total
Bahaha.
Nice, Tom. Now, have you seen it and memorized it all?
Yeah, Mr. A, I'm over a gig now myself. I've 4x1GB Western Digital Caviar Blacks (5 year warranty instead of the more common 3 year, that's why) in there, plus I turned on dual disk redundancy, so my 4TB turns into 2TB. I expect it to fill up in another two years, at which time I don't know what I'll do - switch out two of the drives to 2TB drives, or find another file storage solution, or <gasp!> delete something.
Delete! what a horrible word lol
Quote from: Thorin on July 08, 2012, 01:50:55 PM
Nice, Tom. Now, have you seen it and memorized it all?
Seen 90%+ of it. Probably closer to 99%.
Quote from: Thorin on July 08, 2012, 01:50:55 PM
Yeah, Mr. A, I'm over a gig now myself. I've 4x1GB Western Digital Caviar Blacks (5 year warranty instead of the more common 3 year, that's why) in there, plus I turned on dual disk redundancy, so my 4TB turns into 2TB. I expect it to fill up in another two years, at which time I don't know what I'll do - switch out two of the drives to 2TB drives, or find another file storage solution, or <gasp!> delete something.
I delete regular tv shows. But since I haven't watched most of this seasons shows yet, I haven't deleted them yet :-x I'll eventually get around to it. With the tv stuff deleted I sit at around 1TB free.
I also have 220G of music :D
I dare you to put that on shuffle for an entire month...
Quote from: Thorin on July 08, 2012, 07:08:36 PM
I dare you to put that on shuffle for an entire month...
Heh. There's a lot of it that I got from big collections, or other people. I still have yet to go through it all and delete the @%&# I really don't like. There is some there.
Also there's a fair amount of FLAC, and 320kbps+ files.
I wish my Drobo only had a dead drive... turns out mine has a corrupt partition structure. I have to run FSCK against it after I buy 5x2TB drives and clone them first. I haven't been able to get at my 5.5TB of data now since last week when I rebuilt my backup and then my Drobo (primary source) died right after I formatted my backup drives. GREAT TIMING!
I really am only scared at loosing my 300GB of Documents, and 2TBs of Photos/Home Videos. I have no stateful backup of them since I reorged it all about 2 weeks ago.
Hm, if it were a normal NAS device I'd suggest gparted to attempt to repair the partition structure (it has a repair option, of course you still want to make a backup/clone before trying it). But afaik the drobo uses its own Logical Volume Management, soft RAID, and filesystem. so parted is unlikely to be able to find filesystems to mark as potential partitions. :(
Similar issues make me avoid hardware raid cards.
:( I hope you get that sorted out quick, Mel.
Eh, I need someone to give me a run up to MemEx sometime this week (maybe the weekend?)
Gotta buy a bigger HDD
Quote from: Tom on July 09, 2012, 08:56:24 AM
Similar issues make me avoid hardware raid cards.
True hardware raid cards, consumer or business grade, have very rarely produced any problem like this in the 20+ years of using them in my experience. I however have had to repair more than I would like to say NAS appliances in the past 4 years for faults that I wouldn't see at all in a True hardware raid solution.
Its the black-box NAS appliances that you never know if they are using a hybrid software/hardware raid (like the onboard raids on most mobos) or just a software raid of some type with a beefy sata controller and bus.
I might go to the NL40 from HP for my next NAS (or something like it if this Drobo is history or if I find myself with extra cash), which is a full computer at NAS appliance pricing and scale, with the ability to run a True hardware raid and windows OS. Knowing HPs hardware from my experiences at work, I'll be very confident in how it will work and what to do when it goes wrong.
I've had two hardware RAIDs and both of them failed in an non-recoverable way. One was an on board RAID controller that just melted down and the other was an actual RAID card that just decided to erase partition info one day. Sort of soured me on the whole thing (as well I make periodic backups to ROM for really important stuff)
In both situations I had no other backup and lost quite a bit of stuff...
Well yeah, RAID is not backup, which sales people still push - was in MemEx the other day and heard a sales person refer to RAID as a full backup of your system right in your computer.... which really it is only fault-tolerance for hardware failures, not data corruption like we've experienced.
Damn I wish I hadn't just wiped my backup to start fresh..... grrrrrrrrrrrrr
Any clone drive suggestions out there? I have access to ghost, but curious recommendations from other sources (please include the live cd or distro you are quoting when replying), thanks.
Quote from: Melbosa on July 09, 2012, 10:36:54 AM
Well yeah, RAID is not backup, which sales people still push - was in MemEx the other day and heard a sales person refer to RAID as a full backup of your system right in your computer.... which really it is only fault-tolerance for hardware failures, not data corruption like we've experienced.
Damn I wish I hadn't just wiped my backup to start fresh..... grrrrrrrrrrrrr
Any clone drive suggestions out there? I have access to ghost, but curious recommendations from other sources (please include the live cd or distro you are quoting when replying), thanks.
Clonezilla Live (http://clonezilla.org/clonezilla-live.php) may be all you need.
It supports local, and remote disk imaging.
My main reason for avoiding hw raid is just that if you don't have an identical spare card and yours fails, you're hosed. Nothing short of thousands in data recovery (or lots of time spent trying to find a replacement card that is identical) will get your data back.
Yep, I've never had a drive fail before a hardware RAID...
Quote from: Mr. Analog on July 09, 2012, 10:58:13 AM
Yep, I've never had a drive fail before a hardware RAID...
I'm not entirely certain what you had /was/ hardware raid mind you. Those onboard things are just software raid with some BIOS firmware allowing the system to boot off it. Many cheaper RAID cards are as well. You really only start seeing hw raid cards once you hit the $300 level.
I avoid fakeraid cards/chips like the plague. They are just plain useless. If you do have one, you're better off using it as a dumb SATA controller, and using windows's Dynamic Disks, or linux's mdraid system.
Quote from: Tom on July 09, 2012, 10:57:07 AM
Clonezilla Live (http://clonezilla.org/clonezilla-live.php) may be all you need.
It supports local, and remote disk imaging.
Thanks budy!
Quote from: Tom on July 09, 2012, 10:57:07 AM
My main reason for avoiding hw raid is just that if you don't have an identical spare card and yours fails, you're hosed. Nothing short of thousands in data recovery (or lots of time spent trying to find a replacement card that is identical) will get your data back.
Yes and no... its the raid chip that matters. But I see your point there... same can be said though about a NAS appliance or storage appliance, at least at the consumer level where most are not redundant parts inside.
Cheapest solution for most consumers is cloud storage, where it isn't their problem. Next to that is a NAS appliance with own backup (burns, usb sticks, ec), and then NAS appliance with cloud backup. Then finally is your own server with redundant parts, and backup to boot.
Quote from: Tom on July 09, 2012, 11:00:22 AM
If you do have one, you're better off using it as a dumb SATA controller, and using windows's Dynamic Disks, or linux's mdraid system.
Windows users rejoice... Dynamic Disks as we know them a really not used anymore in 2012. Microsoft has adopted the models of linux raid software and now your storage options in Windows 2012 will allow for many many many fully fault tolerant and resilient OS solutions. Took 'em long enough... now just to wait for M$ to release Windows 2012!
Hmm, does anyone here have any experience with FlexRAID?
http://wiki.flexraid.com/2011/09/02/what-is-flexraid/
http://wiki.flexraid.com/2011/03/26/beginners-guide-to-flexraid-2-0/
Sounds like an awesome piece of software, but who knows if it truly works, right?
Quote from: Tom on July 09, 2012, 10:57:07 AM
Clonezilla Live (http://clonezilla.org/clonezilla-live.php) may be all you need.
Have you used this before on 2TB drives? Any idea how long a clone on a regular SATA channel would take?
Quote from: Tom on July 09, 2012, 11:00:22 AM
Quote from: Mr. Analog on July 09, 2012, 10:58:13 AM
Yep, I've never had a drive fail before a hardware RAID...
I'm not entirely certain what you had /was/ hardware raid mind you. Those onboard things are just software raid with some BIOS firmware allowing the system to boot off it. Many cheaper RAID cards are as well. You really only start seeing hw raid cards once you hit the $300 level.
I avoid fakeraid cards/chips like the plague. They are just plain useless. If you do have one, you're better off using it as a dumb SATA controller, and using windows's Dynamic Disks, or linux's mdraid system.
Ah the first one was an onboard RAID controller built into the motherboard.
The second one was an expansion card with its own power.
Either way, both failed due to RAID corrupting and NOT the drives.
I also had a situation where I borked a software RAID back in '08 but that was PEBKAC and not the controllers fault.
Quote from: Thorin on July 09, 2012, 11:10:52 AM
Hmm, does anyone here have any experience with FlexRAID?
http://wiki.flexraid.com/2011/09/02/what-is-flexraid/
http://wiki.flexraid.com/2011/03/26/beginners-guide-to-flexraid-2-0/
Sounds like an awesome piece of software, but who knows if it truly works, right?
Hehe, welcome to native support for above in Window 2012. And RC is free to download right now if you want to check out.
We tested FlexRAID at work and found it had quite a performance hit when doing JBODs under the covers. While it worked and fully supported and never lost I/O its IOPS was not very pleasing. Not sure if you could say Transcode from a read of this source in real time.
Oh, and CloneZilla works nicely. Or you could just ddrescue them? Either or, both are free :)
Quote from: Melbosa on July 09, 2012, 11:01:15 AM
Quote from: Tom on July 09, 2012, 10:57:07 AM
Clonezilla Live (http://clonezilla.org/clonezilla-live.php) may be all you need.
It supports local, and remote disk imaging.
Thanks budy!
No problem :)
Quote from: Melbosa on July 09, 2012, 11:01:15 AM
Quote from: Tom on July 09, 2012, 10:57:07 AM
My main reason for avoiding hw raid is just that if you don't have an identical spare card and yours fails, you're hosed. Nothing short of thousands in data recovery (or lots of time spent trying to find a replacement card that is identical) will get your data back.
Yes and no... its the raid chip that matters. But I see your point there... same can be said though about a NAS appliance or storage appliance, at least at the consumer level where most are not redundant parts inside.
Cheapest solution for most consumers is cloud storage, where it isn't their problem. Next to that is a NAS appliance with own backup (burns, usb sticks, ec), and then NAS appliance with cloud backup. Then finally is your own server with redundant parts, and backup to boot.
Actually, the raid firmware, and the hw revision also matter. Anything can change the on disk format, and I've seen plenty of references showing that some firmware upgrades on true hardware raid cards can and will change the on disk format.
Quote from: Melbosa on July 09, 2012, 11:03:09 AM
Quote from: Tom on July 09, 2012, 11:00:22 AM
If you do have one, you're better off using it as a dumb SATA controller, and using windows's Dynamic Disks, or linux's mdraid system.
Windows users rejoice... Dynamic Disks as we know them a really not used anymore in 2012. Microsoft has adopted the models of linux raid software and now your storage options in Windows 2012 will allow for many many many fully fault tolerant and resilient OS solutions. Took 'em long enough... now just to wait for M$ to release Windows 2012!
Heh, I was reading about the new stuff. It does look quite nice.
Quote from: Melbosa on July 09, 2012, 11:11:25 AM
Quote from: Tom on July 09, 2012, 10:57:07 AM
Clonezilla Live (http://clonezilla.org/clonezilla-live.php) may be all you need.
Have you used this before on 2TB drives? Any idea how long a clone on a regular SATA channel would take?
I haven't tried it, but partclone doesn't clone the entire disk. It clones the filesystem, skipping unused sectors.
Quote from: Mr. Analog on July 09, 2012, 11:12:55 AM
Quote from: Tom on July 09, 2012, 11:00:22 AM
Quote from: Mr. Analog on July 09, 2012, 10:58:13 AM
Yep, I've never had a drive fail before a hardware RAID...
I'm not entirely certain what you had /was/ hardware raid mind you. Those onboard things are just software raid with some BIOS firmware allowing the system to boot off it. Many cheaper RAID cards are as well. You really only start seeing hw raid cards once you hit the $300 level.
I avoid fakeraid cards/chips like the plague. They are just plain useless. If you do have one, you're better off using it as a dumb SATA controller, and using windows's Dynamic Disks, or linux's mdraid system.
Ah the first one was an onboard RAID controller built into the motherboard.
The second one was an expansion card with its own power.
Either way, both failed due to RAID corrupting and NOT the drives.
I also had a situation where I borked a software RAID back in '08 but that was PEBKAC and not the controllers fault.
You /really really/ want ECC ram with software raid of any kind. As well as a UPS, and a very stable motherboard. If you're missing any one of those you'll see corruption sooner or later. (or so I've found)
I actually have plenty of corrupted files on my raid. Luckily the fs itself is fine, but I bet that's due to XFS being extremely resilient to FS damage.
When I finally bit the bullet and upgraded my home server, I got a bunch of ECC ram, and a proper server board. Any corruption I see from now on will likely be due to pebkac or an outright hardware failure (rather than just random corruption appearing over time).
Quote from: Thorin on July 09, 2012, 11:14:10 AM
Oh, and CloneZilla works nicely. Or you could just ddrescue them? Either or, both are free :)
ddrescue is mostly for rescuing data off damaged disks. If the disk is fine, and the fs isn't completely hosed, you want something like CloneZilla.
If the fs can't be properly mounted, you then want gparted-live (http://gparted.sourceforge.net/livecd.php/) to try and rebuild the parition table and fix the filesystems. Though Clonezilla may contain gparted, I think it might.
Yeah, at the time I was using all second hand RAID gear so, you get what you pay for too... ;)
Quote from: Mr. Analog on July 09, 2012, 11:21:25 AM
Yeah, at the time I was using all second hand RAID gear so, you get what you pay for too... ;)
Hey, my RAID stuff is all second hand. I have a used IBM M1015 8 port SAS/SATA card, and I use mdraid ;D
Quote from: Melbosa on July 09, 2012, 11:13:50 AM
We tested FlexRAID at work and found it had quite a performance hit when doing JBODs under the covers. While it worked and fully supported and never lost I/O its IOPS was not very pleasing. Not sure if you could say Transcode from a read of this source in real time.
Was that disks hooked up to the local machine, or accessed over a network? Could the network be bottle-necking it? I know that read/write from my Drobo FS on the network is pretty slow (my old machines only have 100mb network cards), but I'm able to read files from it and transcode them in real time.
Quote from: Thorin on July 09, 2012, 11:23:29 AM
Quote from: Melbosa on July 09, 2012, 11:13:50 AM
We tested FlexRAID at work and found it had quite a performance hit when doing JBODs under the covers. While it worked and fully supported and never lost I/O its IOPS was not very pleasing. Not sure if you could say Transcode from a read of this source in real time.
Was that disks hooked up to the local machine, or accessed over a network? Could the network be bottle-necking it? I know that read/write from my Drobo FS on the network is pretty slow (my old machines only have 100mb network cards), but I'm able to read files from it and transcode them in real time.
It was local attached in machine storage on the mobo bus. My Drobo FS is on a gigabyte network with Jumbo frames and it is blazingly fast - but it also failed :P
Quote from: Tom on July 09, 2012, 11:20:26 AM
If the fs can't be properly mounted, you then want gparted-live (http://gparted.sourceforge.net/livecd.php/) to try and rebuild the parition table and fix the filesystems. Though Clonezilla may contain gparted, I think it might.
I need block level clone as the File System will not mount on the Drobo FS on these disks. I need to clone each disk to a new disk and execute the built-in FSCK on the Drobo itself, but they and I highly recommend having the sources not be the ones executed against encase it wipes or breaks the File System completely.
So I guess I am at ddrescue time then for the block clone? Or does Clonezilla give you the option to do that instead of the Partition read stuff? Anyone?
Quote from: Tom on July 09, 2012, 11:22:44 AM
Quote from: Mr. Analog on July 09, 2012, 11:21:25 AM
Yeah, at the time I was using all second hand RAID gear so, you get what you pay for too... ;)
Hey, my RAID stuff is all second hand. I have a used IBM M1015 8 port SAS/SATA card, and I use mdraid ;D
*slow clap*
Good luck!
Quote from: Melbosa on July 09, 2012, 11:30:10 AM
Quote from: Tom on July 09, 2012, 11:20:26 AM
If the fs can't be properly mounted, you then want gparted-live (http://gparted.sourceforge.net/livecd.php/) to try and rebuild the parition table and fix the filesystems. Though Clonezilla may contain gparted, I think it might.
I need block level clone as the File System will not mount on the Drobo FS on these disks. I need to clone each disk to a new disk and execute the built-in FSCK on the Drobo itself, but they and I highly recommend having the sources not be the ones executed against encase it wipes or breaks the File System completely.
So I guess I am at ddrescue time then for the block clone? Or does Clonezilla give you the option to do that instead of the Partition read stuff? Anyone?
Pretty much. I doubt Clonezilla supports DroboFS. And I'm not sure parted knows the format of a Drobo volume group either. TBH I can't remember what it uses. Last I heard it was some custom logical volume management setup.
Quote from: Mr. Analog on July 09, 2012, 11:33:27 AM
Quote from: Tom on July 09, 2012, 11:22:44 AM
Quote from: Mr. Analog on July 09, 2012, 11:21:25 AM
Yeah, at the time I was using all second hand RAID gear so, you get what you pay for too... ;)
Hey, my RAID stuff is all second hand. I have a used IBM M1015 8 port SAS/SATA card, and I use mdraid ;D
*slow clap*
Good luck!
:P
These are originally $300 cards ;) I got mine for $90 including shipping. They are rock solid. Based on LSI MegaRaid 9240 cards. The only difference is IBM "differentiated" them by making the sw raid5 support an optional addon that costs $100+ for a silly little jumper stick. So far its far and away better than the old card I had. That thing didn't even have a working driver >:(
I've only used Clonezilla a couple of times, and I don't remember it doing a block-by-block clone of a disk.
Quote from: Thorin on July 09, 2012, 11:40:23 AM
I've only used Clonezilla a couple of times, and I don't remember it doing a block-by-block clone of a disk.
You can likely use dd/ddrescue from clonezilla. but its main function is to copy filesystems at a block level, only copying the used blocks.
I used dd_rescue from the Knoppix Live distro, that worked like a charm. I think I've written about it in another thread on here.
edit: no, that was ddrescue, not dd_rescue.
Quote from: Thorin on July 09, 2012, 01:14:46 PM
I used dd_rescue from the Knoppix Live distro, that worked like a charm. I think I've written about it in another thread on here.
edit: no, that was ddrescue, not dd_rescue.
Yeah I quoted you in the sticky about ddrescue: http://forums.righteouswrath.com/index.php/topic,8737.0.html