Data loss on the Drobo 5D

I bought a Drobo 5D on the 29th of December, 2012, after experiencing catastrophic data loss with the 1st Gen and 2nd Gen Drobo. During multiple phone conversations with Data Robotics’ CEO at the time, Tom Buiocchi, he convinced me that they were much better-engineered than previous-generation Drobos and they had built-in batteries and circuits that would automatically shut them down safely in case of power loss. I was also told the new firmware running inside them would be checking the data constantly to guard against file corruption or data loss. All of these were problems that I’d experienced with my existing Drobos, so even though I was exhausted after my ordeal and so weary of storage technology, I went ahead and purchased the new model and also agreed not to publish an account of what the Drobos had done to my data at the time. I want to make it clear that I paid for my Drobos, so I didn’t feel that I owed him anything, but I did want their company to do well, because back then they were new and deserved a second chance. Now though, there is no excuse for the multiple times their Drobos have lost my precious data. They’ve been around for 13 years and they’ve had plenty of time to make their technology stable.

What’s probably kept them on the market is the willingness of paying customers like me to take a chance on the uniqueness of their proprietary RAID: as far as I know, they are (unfortunately) the only RAID array that lets you store a large amount of data on a single volume that grows automatically as you add drives and also protects (except when it doesn’t) against hard drive failures.

However, after five years of using my Drobo 5D on a daily basis, I can tell you without any doubt that the Drobo 5D does not keep your data safe. Look elsewhere for safe data storage devices. I certainly cannot trust it with my data anymore, so I’ve elected to publish my account of data loss from 2012, as well as an account of my present data loss. Caveat emptor, lest you also lose your data. I’ve also had multiple problems with my two Drobo 5Ns. Because of these problems, some of which have led to significant data loss and to significant time and effort expended in order to restore my data from on-site and off-site backups, I cannot place my trust the new Drobo models that are available, either: I’m talking about the 5D3, 5N2 and the 8D. I see no reason at all to spend more money on more empty promises from Data Robotics.

I can’t say exactly what happened with my Drobo 5D. Drobo Support could not or did not choose to tell me, even though I sent them multiple diagnostic logs from the Drobo and I asked them to tell me what happened. My best guess is that the 5D kept “healing” a 6TB WD drive with bad sectors instead of asking me to replace it. Then a different drive from the array failed. Once the Drobo told me to replace that drive, I did. But during the process of rebuilding the data set, the Drobo 5D decided it didn’t like the 6TB WD drive it had been healing, and chose to tell me that I needed to replace it now. When I did, it would tell me I shouldn’t have taken it out and I should put it back in. I’d put it back in, and then it’d go through its internal processes, only to find out that it wouldn’t like that drive again, but it didn’t stop there. It’d reboot. At first this reboot cycle would take 10-15 minutes, allowing me to copy some data off it, but then it began doing it every 5 minutes. Since it takes a good 3-4 minutes to boot up, this meant I’d have only 1-2 minutes to copy data off before it’d reboot again. This was not workable. After opening a case with Drobo Support, they told me to put it in Read Only mode. You press Ctrl+Opt+Shift+R while you’re in Drobo Dashboard and this reboots it in that mode, which means it’s not going to try and rebuild any data internally, it’s simply going to present the volume to you as it is. This also turned sour quickly, because after allowing me to copy a small amount of data for a few hours, it began that same 5-minute reboot cycle. So I had no way of getting the data off the damned thing unless I mounted it through Disk Warrior and used the Preview app built into that software, which meant having to put up with USB 1.1 transfer speeds. More on the reduced transfer speeds below.

My take on the situation is that it’s a failure in Drobo’s firmware design. It should have asked me to replace the 6TB WD drive instead of working around its bad sectors. Because it didn’t ask me to replace it in time, it then failed when rebuilding its data set after the second drive went bad. That’s not two drives going bad at the same time, that’s a drive going bad and a few weeks later another drive going bad. The Drobo had plenty of time to fix the ongoing situation if its internals had been programmed correctly, but it didn’t, because of inadequate firmware running the device. That’s bad technology at work, causing me repeated data losses.

The 6TB WD drive had 9 bad sectors, but the Drobo 5D kept insisting on healing it until it was too late.

Here’s another example of Drobo’s crappy firmware: for the past three years, I have had to force my iMac not to go to sleep, because every single time I’d wake it up, the Drobo 5D would refuse to mount, forcing me to reboot the iMac and/or the Drobo and also disconnect/connect the Thunderbolt cable from my iMac in order to get the computer to see it. Data Robotics tried to fix this horribly annoying problem (which can also cause data loss) through multiple firmware fixes, but I can safely tell you that they still haven’t fixed it. Before I stopped using my Drobo 5D, I was on Drobo Dashboard 3.3.0 and Drobo 5D firmware 4.1.2, while my iMac is on MacOS Mojave 10.14.2, and this problem still very much occurred. Well, it definitely occurred before my 5D crapped itself. Oh, how I’d like to be back to those simpler times when all I had to deal with was keeping my iMac from going to sleep! But no, now I have to deal with massive data loss, once again. For the goddamned umpteenth time, Drobo!

Except I didn’t cause the disconnect, Drobo’s crappy firmware would cause it to happen, over and over and over…

What do you think would happen after enough improper disconnects? That’s right, volume corruption!

Look at the difference in size between the two directories. The first is a backup, the second is supposed to be the working directory. That’s the Drobo 5D, actively losing your data…
Only 54,890 missing photographs? Thanks, Data Robotics! (This was in 2017.)

Here’s more proof of data loss from 2017. Those yellow triangles are indications of missing video clips.
Here’s proof of more data loss from 2015, when even Disk Warrior couldn’t get my data back.

Data Robotics marketing speak tells you of super-fast transfer speeds and protection against data loss when you buy their devices. The things you most need to remember are that you will experience data loss (that’s a given) and you won’t be recovering your data at super-fast transfer speeds such as USB 3.0/USB-C or Thunderbolt 1/2/3. You will instead be forced to use Disk Warrior’s Preview application (if you’re on a Mac) and you will be recovering your data at USB 1.1 speeds. That’s right, take a moment to think about that! In order to recover data from my Drobo 5D, I have to use Disk Warrior (a Mac app known for its ability to recover failed volumes), because it won’t mount any other way. It certainly doesn’t mount through the Finder or through Disk Utility.

“Work” is the Drobo 5D volume that refuses to mount

The size of my Drobo volume is about 12 TB. Thank God some of it was backed up locally with Resilio Sync and some of it I recovered from online backups with Backblaze, so I only needed to recover part of my data from the Drobo 5D itself, but for a single 3 TB Final Cut Pro library, it took roughly 300 hours to transfer it to another drive! 300 HOURS! It might be somewhat tolerable (in some masochistic sort of way) if I knew the damned thing would stay on for all that time, allowing me to copy the data in one go, but you never know when it’s going to restart. It can go at any time.

I’ve posted illustrative screenshots below. Feel free to do the math yourselves as well. Also think about how much worse this problem gets when your Drobo volume is much, much bigger. The Drobo 5D and 5D3 can go to 64 TB, while the Drobo 8D can go to 128 TB. Do you really want to be stuck copying 50-60 TB of data at USB 1.1 speeds? How about 100 TB of data? Think about that before you click on the Buy button and get one of those shiny black boxes.

The hard truth is that you can’t put all your eggs in one basket. Having all your important data on a single volume, which is what the Drobo lets you do, is a dumb idea. If that volume goes, all your data goes along with it, and it can take MONTHS to recover it from backups.

After more than 12 hours…
Transfer speeds vary between 100 kB/s – 4 MB/s but mostly stay around 100 kB-200 kB/s
Two days for 350 GB… Great…

I don’t know about you, but I don’t want to spend anywhere between $5,000-$10,000 with a data recovery firm every few years, when a Drobo unit decides to fail and lose my data. But that’s what Drobo Support advises you to do from the very beginning. They advise against trying to recover the data yourselves, even while working with them, and instead try to convince you to send your Drobo into a drive recovery firm. For a device that’s supposed to protect your data and a company that brags about “protecting what matters”, that’s disgusting. How exactly are they protecting your data when their devices fail miserably every few years? I guess “protecting what matters” really means “protecting their bottom line” by ensuring suckers keep buying their product.

Do you want to know what happens when I try to open a ticket on the Drobo Support website? This:

For months now, all I get is this grey page with a rotating status ball when I visit Drobo Support

If a company can’t even fix its support page so that it loads up for its customers, that’s cause for worry. I tried explaining the issue to them, I even sent them screenshots, but the techs I spoke with seemed unable to comprehend why the website wouldn’t load. My guess is they’ve got a geofence that stops visits from Romanian IP addresses.

When it comes to their marketing people, I cannot describe them as anything but a bunch of slime buckets. Twice now they’ve ignored me in dire situations when I reached out to them for help, hoping they would redirect my messages internally and get someone to pay proper attention to my case. Back in 2012, back when my two DAS Drobo units lost over 30,000 files, they began ignoring me after I told them what had happened. Now, in 2018, they pretended to help, just so it looked good at first glance on social media, but they didn’t follow through on their promise and ignored me afterward. See my comments on their tweet here. I also wrote a message to some guy who calls himself the Drobo CTO but gives no real name, asking him to have a look at my case, but he ignored me. If it’s a fictitious account then it’s understandable, but if it isn’t, then he’s a turd as well for ignoring a legitimate and polite request for help from a customer.

I also reached out to Data Robotics’ current CEO, Mihir Shah, to ask for his help, and after an initial reply that said, “I am looking into this case for you. We will get back to you. Thanks.” — his line went dead. I am left to conclude that he’s of the same breed as his marketing people, who did the same thing when I reached out to them.

You might be asking yourselves why I chose to reach out to these people when I had already opened a case with Drobo Support? Because I felt the technician handling my case wasn’t doing a good job. This case wasn’t a freebie, I paid for it, and I wanted to get real help, not bullshit. Here are a few examples of the kind of “support” I received:

  • He told me to clone a drive using Data Rescue (a Mac app). This was supposed to fix the situation, but it didn’t, because he told me to clone the wrong drive. That’s right, he had me waste an entire day cloning the wrong drive, before I pointed it out and he came back with sheepish excuses, asking me to clone another.
  • He let me buy two new 8 TB hard drives in order to go through the data recovery process, without saying a damned thing, before another tech stepped in to tell me I should have bought a 6 TB hard drive instead, because the cloning process needed for my scenario requires a drive of the exact same size as the 6 TB WD drive that the Drobo 5D refused to use anymore. Why couldn’t he tell me that from the start? Besides, isn’t the whole point of BeyondRAID, which is Drobo’s proprietary technology, to let people use drives of varying sizes? There goes that advantage when you need it…
  • Having worked in IT for a long time, and having worked my way up from the help desk, I could tell the technician didn’t give a crap about the case. It was clear to me that he wasn’t interested in helping me or solving the case, he was simply posting daily case updates of 1-2 sentences with incomplete and unclear replies to my questions on how to proceed and dragging the case on, probably hoping I’d give up.

This time around, I’m faring better when it comes to data recovery than I fared in 2012. Unfortunately, I don’t have a full local backup of my data, even though the Resilio Sync software was supposed to mirror it from my Drobo 5D to my Drobo 5N. I do have a full online backup with Backblaze, but getting back about 12 TB of data through online downloads will prove to be a cumbersome and slow affair. They only let you download up to 500 GB at a time, which means that if I want to download all of it, I’d have to create at least 24 restore jobs on their servers, each of which would generate a ZIP file that would need to be downloaded and then unzipped locally. I could also choose their HDD recovery option which ships your data to you on 3.5TB drives, but I’d need 4 of them, which means it would cost $189*4 or $756. It would probably work out to something like $850-1,000 for me in the end, because I would incur a high shipping cost from the US to Romania and I would also be responsible for customs fees. Theoretically, Backblaze offers a money back guarantee if the drives are returned within 30 days, but I doubt I could return them in that time span, given they’d have to make it to Romania and back. They’re supposed to be working on a European data center and it might open this year, and while that’s going to be nice in the future, it’s not going to work for my situation at this time. No, the much more workable solution would have been for me to have a full local backup of the files on the Drobo 5D. But thanks to Data Robotics, that option got shot down and I didn’t even find out until it was too late…

You see, the Resilio Sync software was set to mirror my data from the Drobo 5D to a Drobo 5N. On my gigabit network, that would have worked just fine. The app was running on my iMac and it was also running on the 5N through the DroboApps platform. The software had a few months to do a proper job syncing my data between the two devices and it indicated to me that it had done it. Unfortunately, after my Drobo crashed, I found out that it hadn’t. The reasons offered by Resilio Support were the following:

  • The Drobo 5N’s processor wasn’t powerful enough
  • The Drobo 5N’s RAM wasn’t enough
  • The Drobo 5N ended up using swap memory and somehow it messed up the sync logs, which caused it to think it had finished the sync when it hadn’t

If I’m to take them at their word, the Drobo 5N is too underpowered to handle Resilio Sync. That’s not to say there isn’t plenty of blame to be assigned to the Resilio Sync software as well. After all, it should have kept track of the data sync accurately. What good are they if they can’t perform at their main advertised task?

If I am to look at what Drobo is saying about the successor to the 5N, the 5N2, namely that it “provides up to 2x performance boost with an upgraded processor and port bonding option”, that would certainly mean the 1st gen 5N is underpowered. Finally, who do you think recommended Resilio Sync to me? It was none other than Data Robotics, whom I called to ask for details on DroboDR, their own data sync application. They said it was too barebones of an app for my needs and suggested I look into Resilio Sync, which was much more robust and full-featured. Thanks once again to my trust in the Drobo, my backup plans got sabotaged, forcing me to waste weeks of my time recovering data from a failed Drobo volume.

But enough about the Drobo 5N, I’ll have a separate post where I’ll talk about how that device has also lost all of my data recently… Back to the Drobo 5D.

To recap, the Drobo 5D lost my data in 2015, 2017 and 2018. That’s three separate, significant, data loss events, the last of which wasted more than a month of time till I was back on my feet.

I am back on my feet now. It’s the 26th of January 2019 and I finally got the last of my data recovered. I’ve been down, unable to do my work, since the beginning of December 2018. I can’t even remember what day it was the Drobo 5D failed. It’s a blur now.

I ended up buying a third new hard drive, specifically a 6TB hard drive, to match the size of the drive that the Drobo didn’t want to use any more, and I cloned that drive, which turned out to have 5 clusters of bad sectors, onto the new drive, using Data Rescue software for the Mac. I had to use its “segment” cloning mode, which attempts to get around the bad sectors, and this meant the cloning operation took more than 120 hours!

A screenshot taken after 117 hours…

Around hour 122, my iMac crashed and I had to reboot it, so I don’t know if the cloning operation was completed or not. My screen went black and stayed that way. It could be that the cloning ended and then my iMac crashed, or simply that my iMac had had enough of staying up to work through a lengthy cloning process and went poof. I don’t know, and I wasn’t about to start cloning the damn drive again. I took my Drobo 5D out of Read-Only Mode, stuck the cloned drive into the Drobo 5D and booted it up, expecting some data loss. Besides, even if the cloning operation had completed successfully, I’d have still lost some data. If you look above, you’ll see that there were 298,2 MB of bad, unrecovered data. That wasn’t from one portion of the drive, that was from five different portions, because there were five significant slow-downs in the cloning process.

The 5D booted up and started its data protection protocol

Once it booted up, I started copying my data off it, not knowing how long it might stay on. For all I knew, it could go into a vicious reboot cycle at any moment. I’d already recovered all I could from Backblaze and from my local backups, and I needed about 6-7 TB of data from the 5D before I was back up to normal… “normal” being a loose term given how unreliable the Drobo is and how much data I’d already lost.

Sure enough, data loss soon reared its ugly head. Thanks, Drobo, you bunch of slime buckets!

One of the damaged FCPX libraries with lots of missing video clips

So far, two FCPX libraries containing the videos I’d made in 2017 and 2018, are damaged. I don’t know if I’ll be able to get those videos back.

I consoled myself with the idea that at least I was able to recover most of my data at Thunderbolt speeds, not at USB 1.1 speeds, so that saved me about 2-3 more weeks of painful waiting.

All of my data is now off the Drobo 5D, and I don’t plan on using it again. I’ve put it in a storage closet. I’m done with it, and I recommend you also stop using your Drobos, if you are using one, because it’s only a matter of time before they will lose your data. In my opinion, they offer no more protection than using individual drives. They just cost more, force you to buy more hard drives and they make more noise. My office is so much quieter now that the Drobo 5D is gone. I’m using individual drives which I plan on cloning locally. I will continue to use Backblaze for cloud backups, and I may also do some periodic local backups to a NAS of some sort. I have found out during these past few months that I also cannot rely on the Drobo 5N NAS to store my data, because one of my Drobo 5N units has just lost all of the data I’d entrusted to it. I am now in the process of recovering that… It’s non-stop torture, time and data loss with a Drobo!

I have been a Drobo customer since December 2007, when I bought my first Drobo. I was also among the first Drobo “Evangelists”, as they called their enthusiastic customers back then. It’s now January 2019, and I am done with Drobo. I was a loyal customer for 12 years and I stuck with them through an incredibly disheartening amount of data loss, problematic units and buggy firmware. That’s enough of that. Caveat emptor. Drobo no more.