Reviews

Data loss on the Drobo 5D

I bought a Drobo 5D on the 29th of December, 2012, after experiencing catastrophic data loss with the 1st Gen and 2nd Gen Drobo. During multiple phone conversations with Data Robotics’ CEO at the time, Tom Buiocchi, he convinced me that they were much better-engineered than previous-generation Drobos and they had built-in batteries and circuits that would automatically shut them down safely in case of power loss. I was also told the new firmware running inside them would be checking the data constantly to guard against file corruption or data loss. All of these were problems that I’d experienced with my existing Drobos, so even though I was exhausted after my ordeal and so weary of storage technology, I went ahead and purchased the new model and also agreed not to publish an account of what the Drobos had done to my data at the time. I want to make it clear that I paid for my Drobos, so I didn’t feel that I owed him anything, but I did want their company to do well, because back then they were new and deserved a second chance. Now though, there is no excuse for the multiple times their Drobos have lost my precious data. They’ve been around for 13 years and they’ve had plenty of time to make their technology stable.

What’s probably kept them on the market is the willingness of paying customers like me to take a chance on the uniqueness of their proprietary RAID: as far as I know, they are (unfortunately) the only RAID array that lets you store a large amount of data on a single volume that grows automatically as you add drives and also protects (except when it doesn’t) against hard drive failures.

However, after five years of using my Drobo 5D on a daily basis, I can tell you without any doubt that the Drobo 5D does not keep your data safe. Look elsewhere for safe data storage devices. I certainly cannot trust it with my data anymore, so I’ve elected to publish my account of data loss from 2012, as well as an account of my present data loss. Caveat emptor, lest you also lose your data. I’ve also had multiple problems with my two Drobo 5Ns. Because of these problems, some of which have led to significant data loss and to significant time and effort expended in order to restore my data from on-site and off-site backups, I cannot place my trust the new Drobo models that are available, either: I’m talking about the 5D3, 5N2 and the 8D. I see no reason at all to spend more money on more empty promises from Data Robotics.

I can’t say exactly what happened with my Drobo 5D. Drobo Support could not or did not choose to tell me, even though I sent them multiple diagnostic logs from the Drobo and I asked them to tell me what happened. My best guess is that the 5D kept “healing” a 6TB WD drive with bad sectors instead of asking me to replace it. Then a different drive from the array failed. Once the Drobo told me to replace that drive, I did. But during the process of rebuilding the data set, the Drobo 5D decided it didn’t like the 6TB WD drive it had been healing, and chose to tell me that I needed to replace it now. When I did, it would tell me I shouldn’t have taken it out and I should put it back in. I’d put it back in, and then it’d go through its internal processes, only to find out that it wouldn’t like that drive again, but it didn’t stop there. It’d reboot. At first this reboot cycle would take 10-15 minutes, allowing me to copy some data off it, but then it began doing it every 5 minutes. Since it takes a good 3-4 minutes to boot up, this meant I’d have only 1-2 minutes to copy data off before it’d reboot again. This was not workable. After opening a case with Drobo Support, they told me to put it in Read Only mode. You press Ctrl+Opt+Shift+R while you’re in Drobo Dashboard and this reboots it in that mode, which means it’s not going to try and rebuild any data internally, it’s simply going to present the volume to you as it is. This also turned sour quickly, because after allowing me to copy a small amount of data for a few hours, it began that same 5-minute reboot cycle. So I had no way of getting the data off the damned thing unless I mounted it through Disk Warrior and used the Preview app built into that software, which meant having to put up with USB 1.1 transfer speeds. More on the reduced transfer speeds below.

My take on the situation is that it’s a failure in Drobo’s firmware design. It should have asked me to replace the 6TB WD drive instead of working around its bad sectors. Because it didn’t ask me to replace it in time, it then failed when rebuilding its data set after the second drive went bad. That’s not two drives going bad at the same time, that’s a drive going bad and a few weeks later another drive going bad. The Drobo had plenty of time to fix the ongoing situation if its internals had been programmed correctly, but it didn’t, because of inadequate firmware running the device. That’s bad technology at work, causing me repeated data losses.

The 6TB WD drive had 9 bad sectors, but the Drobo 5D kept insisting on healing it until it was too late.

Here’s another example of Drobo’s crappy firmware: for the past three years, I have had to force my iMac not to go to sleep, because every single time I’d wake it up, the Drobo 5D would refuse to mount, forcing me to reboot the iMac and/or the Drobo and also disconnect/connect the Thunderbolt cable from my iMac in order to get the computer to see it. Data Robotics tried to fix this horribly annoying problem (which can also cause data loss) through multiple firmware fixes, but I can safely tell you that they still haven’t fixed it. Before I stopped using my Drobo 5D, I was on Drobo Dashboard 3.3.0 and Drobo 5D firmware 4.1.2, while my iMac is on MacOS Mojave 10.14.2, and this problem still very much occurred. Well, it definitely occurred before my 5D crapped itself. Oh, how I’d like to be back to those simpler times when all I had to deal with was keeping my iMac from going to sleep! But no, now I have to deal with massive data loss, once again. For the goddamned umpteenth time, Drobo!

Except I didn’t cause the disconnect, Drobo’s crappy firmware would cause it to happen, over and over and over…

What do you think would happen after enough improper disconnects? That’s right, volume corruption!

Look at the difference in size between the two directories. The first is a backup, the second is supposed to be the working directory. That’s the Drobo 5D, actively losing your data…
Only 54,890 missing photographs? Thanks, Data Robotics! (This was in 2017.)

Here’s more proof of data loss from 2017. Those yellow triangles are indications of missing video clips.
Here’s proof of more data loss from 2015, when even Disk Warrior couldn’t get my data back.

Data Robotics marketing speak tells you of super-fast transfer speeds and protection against data loss when you buy their devices. The things you most need to remember are that you will experience data loss (that’s a given) and you won’t be recovering your data at super-fast transfer speeds such as USB 3.0/USB-C or Thunderbolt 1/2/3. You will instead be forced to use Disk Warrior’s Preview application (if you’re on a Mac) and you will be recovering your data at USB 1.1 speeds. That’s right, take a moment to think about that! In order to recover data from my Drobo 5D, I have to use Disk Warrior (a Mac app known for its ability to recover failed volumes), because it won’t mount any other way. It certainly doesn’t mount through the Finder or through Disk Utility.

“Work” is the Drobo 5D volume that refuses to mount

The size of my Drobo volume is about 12 TB. Thank God some of it was backed up locally with Resilio Sync and some of it I recovered from online backups with Backblaze, so I only needed to recover part of my data from the Drobo 5D itself, but for a single 3 TB Final Cut Pro library, it took roughly 300 hours to transfer it to another drive! 300 HOURS! It might be somewhat tolerable (in some masochistic sort of way) if I knew the damned thing would stay on for all that time, allowing me to copy the data in one go, but you never know when it’s going to restart. It can go at any time.

I’ve posted illustrative screenshots below. Feel free to do the math yourselves as well. Also think about how much worse this problem gets when your Drobo volume is much, much bigger. The Drobo 5D and 5D3 can go to 64 TB, while the Drobo 8D can go to 128 TB. Do you really want to be stuck copying 50-60 TB of data at USB 1.1 speeds? How about 100 TB of data? Think about that before you click on the Buy button and get one of those shiny black boxes.

The hard truth is that you can’t put all your eggs in one basket. Having all your important data on a single volume, which is what the Drobo lets you do, is a dumb idea. If that volume goes, all your data goes along with it, and it can take MONTHS to recover it from backups.

After more than 12 hours…
Transfer speeds vary between 100 kB/s – 4 MB/s but mostly stay around 100 kB-200 kB/s
Two days for 350 GB… Great…

I don’t know about you, but I don’t want to spend anywhere between $5,000-$10,000 with a data recovery firm every few years, when a Drobo unit decides to fail and lose my data. But that’s what Drobo Support advises you to do from the very beginning. They advise against trying to recover the data yourselves, even while working with them, and instead try to convince you to send your Drobo into a drive recovery firm. For a device that’s supposed to protect your data and a company that brags about “protecting what matters”, that’s disgusting. How exactly are they protecting your data when their devices fail miserably every few years? I guess “protecting what matters” really means “protecting their bottom line” by ensuring suckers keep buying their product.

Do you want to know what happens when I try to open a ticket on the Drobo Support website? This:

For months now, all I get is this grey page with a rotating status ball when I visit Drobo Support

If a company can’t even fix its support page so that it loads up for its customers, that’s cause for worry. I tried explaining the issue to them, I even sent them screenshots, but the techs I spoke with seemed unable to comprehend why the website wouldn’t load. My guess is they’ve got a geofence that stops visits from Romanian IP addresses.

When it comes to their marketing people, I cannot describe them as anything but a bunch of slime buckets. Twice now they’ve ignored me in dire situations when I reached out to them for help, hoping they would redirect my messages internally and get someone to pay proper attention to my case. Back in 2012, back when my two DAS Drobo units lost over 30,000 files, they began ignoring me after I told them what had happened. Now, in 2018, they pretended to help, just so it looked good at first glance on social media, but they didn’t follow through on their promise and ignored me afterward. See my comments on their tweet here. I also wrote a message to some guy who calls himself the Drobo CTO but gives no real name, asking him to have a look at my case, but he ignored me. If it’s a fictitious account then it’s understandable, but if it isn’t, then he’s a turd as well for ignoring a legitimate and polite request for help from a customer.

I also reached out to Data Robotics’ current CEO, Mihir Shah, to ask for his help, and after an initial reply that said, “I am looking into this case for you. We will get back to you. Thanks.” — his line went dead. I am left to conclude that he’s of the same breed as his marketing people, who did the same thing when I reached out to them.

You might be asking yourselves why I chose to reach out to these people when I had already opened a case with Drobo Support? Because I felt the technician handling my case wasn’t doing a good job. This case wasn’t a freebie, I paid for it, and I wanted to get real help, not bullshit. Here are a few examples of the kind of “support” I received:

  • He told me to clone a drive using Data Rescue (a Mac app). This was supposed to fix the situation, but it didn’t, because he told me to clone the wrong drive. That’s right, he had me waste an entire day cloning the wrong drive, before I pointed it out and he came back with sheepish excuses, asking me to clone another.
  • He let me buy two new 8 TB hard drives in order to go through the data recovery process, without saying a damned thing, before another tech stepped in to tell me I should have bought a 6 TB hard drive instead, because the cloning process needed for my scenario requires a drive of the exact same size as the 6 TB WD drive that the Drobo 5D refused to use anymore. Why couldn’t he tell me that from the start? Besides, isn’t the whole point of BeyondRAID, which is Drobo’s proprietary technology, to let people use drives of varying sizes? There goes that advantage when you need it…
  • Having worked in IT for a long time, and having worked my way up from the help desk, I could tell the technician didn’t give a crap about the case. It was clear to me that he wasn’t interested in helping me or solving the case, he was simply posting daily case updates of 1-2 sentences with incomplete and unclear replies to my questions on how to proceed and dragging the case on, probably hoping I’d give up.

This time around, I’m faring better when it comes to data recovery than I fared in 2012. Unfortunately, I don’t have a full local backup of my data, even though the Resilio Sync software was supposed to mirror it from my Drobo 5D to my Drobo 5N. I do have a full online backup with Backblaze, but getting back about 12 TB of data through online downloads will prove to be a cumbersome and slow affair. They only let you download up to 500 GB at a time, which means that if I want to download all of it, I’d have to create at least 24 restore jobs on their servers, each of which would generate a ZIP file that would need to be downloaded and then unzipped locally. I could also choose their HDD recovery option which ships your data to you on 3.5TB drives, but I’d need 4 of them, which means it would cost $189*4 or $756. It would probably work out to something like $850-1,000 for me in the end, because I would incur a high shipping cost from the US to Romania and I would also be responsible for customs fees. Theoretically, Backblaze offers a money back guarantee if the drives are returned within 30 days, but I doubt I could return them in that time span, given they’d have to make it to Romania and back. They’re supposed to be working on a European data center and it might open this year, and while that’s going to be nice in the future, it’s not going to work for my situation at this time. No, the much more workable solution would have been for me to have a full local backup of the files on the Drobo 5D. But thanks to Data Robotics, that option got shot down and I didn’t even find out until it was too late…

You see, the Resilio Sync software was set to mirror my data from the Drobo 5D to a Drobo 5N. On my gigabit network, that would have worked just fine. The app was running on my iMac and it was also running on the 5N through the DroboApps platform. The software had a few months to do a proper job syncing my data between the two devices and it indicated to me that it had done it. Unfortunately, after my Drobo crashed, I found out that it hadn’t. The reasons offered by Resilio Support were the following:

  • The Drobo 5N’s processor wasn’t powerful enough
  • The Drobo 5N’s RAM wasn’t enough
  • The Drobo 5N ended up using swap memory and somehow it messed up the sync logs, which caused it to think it had finished the sync when it hadn’t

If I’m to take them at their word, the Drobo 5N is too underpowered to handle Resilio Sync. That’s not to say there isn’t plenty of blame to be assigned to the Resilio Sync software as well. After all, it should have kept track of the data sync accurately. What good are they if they can’t perform at their main advertised task?

If I am to look at what Drobo is saying about the successor to the 5N, the 5N2, namely that it “provides up to 2x performance boost with an upgraded processor and port bonding option”, that would certainly mean the 1st gen 5N is underpowered. Finally, who do you think recommended Resilio Sync to me? It was none other than Data Robotics, whom I called to ask for details on DroboDR, their own data sync application. They said it was too barebones of an app for my needs and suggested I look into Resilio Sync, which was much more robust and full-featured. Thanks once again to my trust in the Drobo, my backup plans got sabotaged, forcing me to waste weeks of my time recovering data from a failed Drobo volume.

But enough about the Drobo 5N, I’ll have a separate post where I’ll talk about how that device has also lost all of my data recently… Back to the Drobo 5D.

To recap, the Drobo 5D lost my data in 2015, 2017 and 2018. That’s three separate, significant, data loss events, the last of which wasted more than a month of time till I was back on my feet.

I am back on my feet now. It’s the 26th of January 2019 and I finally got the last of my data recovered. I’ve been down, unable to do my work, since the beginning of December 2018. I can’t even remember what day it was the Drobo 5D failed. It’s a blur now.

I ended up buying a third new hard drive, specifically a 6TB hard drive, to match the size of the drive that the Drobo didn’t want to use any more, and I cloned that drive, which turned out to have 5 clusters of bad sectors, onto the new drive, using Data Rescue software for the Mac. I had to use its “segment” cloning mode, which attempts to get around the bad sectors, and this meant the cloning operation took more than 120 hours!

A screenshot taken after 117 hours…

Around hour 122, my iMac crashed and I had to reboot it, so I don’t know if the cloning operation was completed or not. My screen went black and stayed that way. It could be that the cloning ended and then my iMac crashed, or simply that my iMac had had enough of staying up to work through a lengthy cloning process and went poof. I don’t know, and I wasn’t about to start cloning the damn drive again. I took my Drobo 5D out of Read-Only Mode, stuck the cloned drive into the Drobo 5D and booted it up, expecting some data loss. Besides, even if the cloning operation had completed successfully, I’d have still lost some data. If you look above, you’ll see that there were 298,2 MB of bad, unrecovered data. That wasn’t from one portion of the drive, that was from five different portions, because there were five significant slow-downs in the cloning process.

The 5D booted up and started its data protection protocol

Once it booted up, I started copying my data off it, not knowing how long it might stay on. For all I knew, it could go into a vicious reboot cycle at any moment. I’d already recovered all I could from Backblaze and from my local backups, and I needed about 6-7 TB of data from the 5D before I was back up to normal… “normal” being a loose term given how unreliable the Drobo is and how much data I’d already lost.

Sure enough, data loss soon reared its ugly head. Thanks, Drobo, you bunch of slime buckets!

One of the damaged FCPX libraries with lots of missing video clips

So far, two FCPX libraries containing the videos I’d made in 2017 and 2018, are damaged. I don’t know if I’ll be able to get those videos back.

I consoled myself with the idea that at least I was able to recover most of my data at Thunderbolt speeds, not at USB 1.1 speeds, so that saved me about 2-3 more weeks of painful waiting.

All of my data is now off the Drobo 5D, and I don’t plan on using it again. I’ve put it in a storage closet. I’m done with it, and I recommend you also stop using your Drobos, if you are using one, because it’s only a matter of time before they will lose your data. In my opinion, they offer no more protection than using individual drives. They just cost more, force you to buy more hard drives and they make more noise. My office is so much quieter now that the Drobo 5D is gone. I’m using individual drives which I plan on cloning locally. I will continue to use Backblaze for cloud backups, and I may also do some periodic local backups to a NAS of some sort. I have found out during these past few months that I also cannot rely on the Drobo 5N NAS to store my data, because one of my Drobo 5N units has just lost all of the data I’d entrusted to it. I am now in the process of recovering that… It’s non-stop torture, time and data loss with a Drobo!

I have been a Drobo customer since December 2007, when I bought my first Drobo. I was also among the first Drobo “Evangelists”, as they called their enthusiastic customers back then. It’s now January 2019, and I am done with Drobo. I was a loyal customer for 12 years and I stuck with them through an incredibly disheartening amount of data loss, problematic units and buggy firmware. That’s enough of that. Caveat emptor. Drobo no more.

Standard
Events

The time a Drobo lost over 30,000 of my photos and videos

The following is an account of what happened to me in late 2012, when two of my Drobo units malfunctioned and corrupted or erased over 30,000 of my photos and videos, rendering them either completely unusable or making them vanish into digital ether.

I have been a Drobo customer since late 2007, when I bought my first USB Drobo. I was glad to take advantage of the new storage possibilities it offered. Since then, I’ve bought one or two Firewire Drobos, two Drobo 5Ns and a Drobo 5D. After relying on my Drobos for several years, I experienced catastrophic data corruption on both of my Drobos at the time (a USB Drobo and a Firewire Drobo). That incident still haunts me to this day.

If you look through my past posts about the Drobo, you’ll see that I wrote about them and I defended them (speaking from my own experience at the time) when others said, among other things, that they “can’t be relied upon” and they often turn into “bricks” or lose people’s data. From my own experience, I had only good things to say about them. They hadn’t failed me for years, I hadn’t yet lost any data from them, the support was great and their design is beautiful. They are a pleasure to use when they work. Until recently, few things in hardware got me more excited than buying a new, bigger hard drive for my Drobo and sticking it in, then watching the capacity gauge show the new, larger volume and hearing it start to sync the data to the new drive…

Which brings me neatly right to the crux of the problem. I had bought a new 3TB hard drive for my Firewire Drobo and stuck it in, replacing an existing 2TB drive. The data sync had gone on for close to 30 hours and was getting close to completion (the Dashboard software was telling me there were only 5-6 hours left). All of a sudden, a red light next to the new 3TB hard drive went off. Before I had a chance to assess the situation, the Drobo crashed and rebooted. I thought, okay, it’s resetting itself and will begin to work once more. I waited for the reboot to complete, only to discover, to my dismay, that it rebooted again… and again… and again… During those brief moments when it was up, the red light switched places, moving from the 3TB hard drive to one of the existing 2TB hard drives. This wasn’t good. I decided to shut it off completely and unplug it. I filed a ticket with Drobo Support and waited.

This horrific experience came after I experienced catastrophic data loss on my USB Drobo just a couple of days earlier. That Drobo was and had been plugged into a surge protector (unlike my Firewire Drobo, which had been plugged into a UPS). The electricity had been cut off and when it came back up, the USB Drobo wouldn’t mount to the desktop. No big deal, I thought. The electricity had gone off before and when the Drobo wouldn’t mount, I would run Disk Utility on it and fix it right up. That’s what I did this time, only Disk Utility couldn’t repair it. Oh boy… I filed a ticket with Drobo Support. They told me they couldn’t help and that I should buy Disk Warrior or send it to a data recovery firm. I called the firm to get a quote, which they said would be anywhere from $1,200 to $7,900, depending on how much data they recovered. They told me that realistically, it would cost at least $4,000 if they recovered any data at all, and that it’s difficult to get the data off a corrupt Drobo volume, because of their complicated storage algorithms. They said it’s much easier with traditional RAID systems.

So I bought Disk Warrior and tried my luck. It was able to mount the Drobo temporarily, but couldn’t fix it. The error it gave me was that Disk Utility had messed up the catalog file when it tried to fix it and they couldn’t recreate it. I called their Tech Support and one of their technicians stayed on the phone with me for over an hour. We did a Screen Sharing session in iChat, where he kept running scans on the volume and checking the diagnostic files. Same result. Couldn’t be fixed. He suggested I get whatever data I could get off it directly through the Disk Warrior’s Preview feature. I started to copy off the data and got about 70GB off (onto the Firewire Drobo) which hadn’t crashed yet. Then the USB Drobo decided to call it quits and became completely unusable.

Out of over 3TB of data, I was able to salvage temporarily about 70GB. I lost the rest: over 15 years of personal and family information, photos, personal videos, letters, our movie collection (which I had painstakingly digitized and edited from VHS tapes and TV recordings over several years), our cartoon collection (also painstakingly digitized and edited from VHS tapes and TV over several years, and which I was looking forward to watching with our unborn baby — my wife was pregnant at the time; our daughter is now almost six years old) and other various files.

I decided not to send the Drobo to data recovery because the price to recover the data would have been higher than the price to buy all of the movies and cartoons on DVD once more, and because I don’t feel comfortable with other people rifling through my personal files. I didn’t know how much they’d be able to recover, if anything, and I consoled myself with the notion that at least my Firewire Drobo was still fine and I had my work files (my photo library of over 100,000 photos, the footage and FCP and iMovie events and projects from our shows (Romania Through Their Eyes and Ligia’s Kitchen), various other personal and published videos, etc. — basically, my creative work over the past 6-7 years.

Then, my Firewire Drobo went AWOL. As mentioned above, I filed a ticket with Drobo Support and waited to see what they would say. Their advice: buy Data Rescue 3 and run it on the 2TB drive that showed up with a red light next to it. Apparently the drive went bad during the data sync for the new 3TB drive. With Data Rescue 3, I could in theory clone it, then insert the cloned drive into the Drobo and recover my data. Well, I couldn’t clone the whole drive. I tried it a couple of times, but whenever the head came to a certain point on the platters, the drive would disconnect itself from the computer. So I did a reverse clone (where the cloning process begins at the end, not the start of the drive) to try and recover the rest of the data after the bad section on the drive. After that completed and with the blessing of Drobo Support, I called that a “best try” clone, stuck the new drive into the Drobo, put it in Read Only Mode and through Data Rescue 3, began to copy off my data.

Things looked good at the time. The software seemed to see all my files and although the copying process was slow, I had access to my data, which was the important thing. After 4-5 days of data retrieval, I was done and at Drobo Support’s advice, I reset the Drobo and formatted a new volume onto it, then began to copy my data back onto it. It was deemed healthy once again and I was sent a new power supply. I’m still not sure why the new power supply was needed, but alright.

After all my data was back onto the newly formatted Firewire Drobo, I began to access it. Only then did I discover data corruption at the file level. Photos, precious memories which I’d trusted my Drobo to keep were now forever lost. Photos of when I met Ligia (my wife), photos from the wedding, from our first years together… they’re gone now. Videos have turned into audio files (the video track can no longer be seen by video players or video editing programs. Just from the year 2007, I lost well over 17,000 photos (about 80% of them). The same percentage applied to the years 2000-2006, when I took less photos but lost the same proportion of files (about 80%). I still don’t know the exact number of photos I lost, but I estimate it to be well over 30,000. I still dread to open older video libraries in Final Cut Pro, because when I access my projects, I always find video clips that are either missing or completely unusable.

When I updated my ticket with Drobo Support to let them know about this, data recovery services were once more suggested. Now I’m not a data recovery expert, but after you’ve formatted a volume and written a LOT of data to it, doesn’t that make it harder, if not impossible, to get the old data back?

If you’d like to see what file-level corruption looks like, it’s not pretty. Here are a few screenshots. Those thumbnails are all I have left of photos I wanted to keep forever.

Let’s look at this thing from both sides.

The pros:

  • These two Drobos worked for five years, flawlessly;
  • My experience with Drobo Support was good from 2007 till 2012; even though the warranty had expired on both of these Drobos, they took up my cases and tried to help me;
  • I should have had my data backed in other places, I know, I know; That’s exactly what I as doing as the Drobos failed. I was reorganizing files and reshuffling my archives when it happened. If you were to ask me, as an IT professional, what I would recommend to others, it would have been the following: have the data in its primary location on a reliable and fast device, back it up to a server or NAS on the local network and also back it up offsite. I was caught in-between, with no access to my old backups and no proper backup plan in place. Crashplan, the offsite backup company I was using at the time, had been throttling my uploads to conserve their bandwidth, and the offsite backups hadn’t completed. If they hadn’t been actively engaged in throttling user uploads, as was their practice at the time, I could have recovered all my files.
  • Drobos are easy to use and allow one to consolidate their data on one volume that can keep growing and growing and growing with the addition of new drives — that part I always liked;
  • Even if one drive fails, the data’s still safe; I had that happen to me once in years past and yes, the data was fine — that part I also always liked;
  • I should have had the USB Drobo plugged into a UPS, just like the Firewire Drobo. Should have, would have, could have…

The cons:

  • Two catastrophic failures within one week… c’mon, this is ridiculous to say the least…
  • I had no idea you could lose all your data from the Drobo if the power went out, and that it would cost upwards of $5,000 to get it back. Ugh… if it costs so much to get the data back, Data Robotics should seriously consider doing what ioSafe has been doing from the start: offering Data Recovery Services standard with every one of their products.
  • Apparently the newer Drobo products, like the 5D, the Drobo Mini and the 5N, have a built-in battery that protects against power outages, so what happened with my USB Drobo won’t happen to them, theoretically. Well… if they knew about this, couldn’t they have put that battery/capacitor in the older Drobos as well? I sure wish they had taken every step possible to make sure any data placed on a Drobo stayed safe. After all, they’ve been saying for years that they “protect what matters”. Do they really?
  • Earlier this year, a friend of mine wrote to me telling me of his experience with his USB Drobo. It was the very same thing that happened to mine, minus the power outage. He would eject it from the desktop and unplug it from the outlet, thinking that was safe. It was, for a while, until he lost all his data. At the time, I thought he must have been doing something wrong. Until it happened to me. Is it just me, or do we not do the same with a regular external drive? Do we not eject it then turn it off? Does the electricity not go off sometimes, and even if the Drobo is plugged into a UPS, the electricity may be out for more than 10 minutes, meaning the UPS will also go down? And when the electricity comes back up, could we not also lose our data even though we did everything right?
  • I remember jokingly asking someone from Data Robotics in years past what would happen if a drive went bad right after you inserted a new one and the Drobo was still in Data Protection mode. Their response: not very likely that could happen. The Drobo would know that a drive was about to go bad and would give me warning. It’s quite apparent to me now that that’s not the way things work sometimes. Sometimes a drive does go bad during Data Protection and not only do you get no warning, you also get catastrophic data loss. Double whammy.
  • Someone from Drobo Sales happened to write to me to invite me to a webinar as I was in the throes of my data loss. I replied and told him of my troubles, and that was the last I heard from him. Was that the way to treat a long-time customer? An adequate response would have been “Hey, I’m really sorry about that. Are you getting the help you need from Tech Support? If not, do you want me to step in for you?”

This whole affair left me exhausted and completely disappointed. I wasted so much time trying to recover my data and my bearings. The experience has taught me a few things, one of them being the loss of my trust in Drobo. It is a beautiful little black box. It works great and you begin to rely on it more and more, till you can’t do without it; till you think it works so good your data’s going to be fine all the time. Then it fails and your data is gone.

If you’d like to know how I recovered some of my data, I wrote about it here. I also experienced another major data loss event with my Drobo 5D just last month, and that was after more data loss with my 5D in 2015 and 2017. Add to that ongoing file system corruption in one of my Drobo 5N units and the “Drobo data loss package” is complete. I don’t know, I must have unlocked some kind of secret data loss level-up in the data storage game…

Standard
Thoughts

Fun with technology

I’ve had multiple Drobo units since 2007. To this day, I still enjoy adding a hard drive to a Drobo. It’s one of those things that can be an ordeal on other tech, but on a Drobo, it’s been made fun through proper planning and design.

It lets you that it’s low on space, you order a drive, and when it comes, you look at the app, which tells you exactly what size-drive is in each bay. Pressing a small lever on the side of the bay releases the drive, which slides out. You put the new one in, the Drobo immediately checks it and formats it, then begins striping the data set across it. By the way, that’s a screen shot showing my Drobo 5D.

Screen Shot 2017-12-21 at 12.48.30.png

I love this process. It’s so simple and so fun! The Drobo doesn’t care what hard drive you buy, as long as it’s larger than what you already had. It allows you to grow the capacity of your Drobo in time, as the prices for newer, bigger hard drives decrease, without any sort of headaches. This is technology done right.

Standard
Reviews

A comparison of CrashPlan and Backblaze

I’ve been a paying CrashPlan customer since 2012 and my initial backup still hasn’t finished. I’ve been a paying Backblaze customer for less than a month and my initial backup is already complete. 

I’m not a typical customer for backup companies. Most people back up about 1 TB of data or less. The size of my minimum backup set is about 9 TB. If I count all the stuff I want to back up, it’s about 12 TB. And that’s a problem with most backup services.

First, let me say this: I didn’t write this post to trash CrashPlan. Their backup service works and it’s worked well for other members of my family. It just hasn’t worked for me. This is because they only offer a certain amount of bandwidth to each user. It’s called bandwidth throttling and it saves them money in two ways: (1) they end up paying less for their monthly bandwidth (which adds up to a lot for a company offering backup services) and (2) they filter out heavy users like me, who tend to fill up a lot of their drives with unprofitable data. My guess (from my experience with them) is that they throttle heavy users with large backup sets much more than they throttle regular users. The end result of this bandwidth throttling is that, even though I’ve been a customer since 2012 — at first, I was on the individual backup plan, then I switched to the family plan — my initial backup never completed and I was well on track to never completing it.

When I stopped using CrashPlan’s backup services, out of the almost 9 TB of data that I need to back up constantly, I had only managed to upload 0.9 TB in FOUR YEARS. Take a moment and think about that, and then you’ll realize how much bandwidth throttling CrashPlan does on heavy users like me.

Screen Shot 2016-10-20 at 23.37.07.png

After four years of continuous use, I backed up a grand total of 905.7 GB to CrashPlan

To be exact, counting the various versions of my data that had accummulated on the CrashPlan servers in these four years, I had a total of 2.8 TB stored on their servers, but even if you count that as the total, 2.8 TB in FOUR YEARS is still an awfully small amount.

Screen Shot 2016-10-27 at 00.42.14.png

Space used on CrashPlan’s servers: 2.8 TB

Tell me honestly, which one of you wants this kind of service from a backup company? You pay them for years in a row and your initial backup never finishes? If a data loss event occurs and your local backup is gone (say a fire, flood or burglary), you’re pretty much screwed and you’ll only be able to recover a small portion of your data from their servers, even though you’ve been a faithful, paying customer for years… That just isn’t right.

I talked with CrashPlan techs twice in these fours years about this very problematic data throttling. Given that they advertise their service as “unlimited backup”, this is also an ethical issue. The backup isn’t truly unlimited if it’s heavily throttled and you can never back up all of your data. The answer was the same both times, even the wording was the same, making me think it was scripted: they said that in an effort to keep costs affordable, they have to limit the upload speeds of every user. The first time I asked them, they suggested their Business plan has higher upload speeds, so in other words, they tried to upsell me. During both times, they advertised their “seed drive service”, which was a paid product (they stopped offering it this summer). The gist of their paid service was that they shipped asking customers a 1 TB drive so you could back up to it locally, then send it to them to jumpstart the backup. Again, given my needs of backing up at least 9 TB of data, this wasn’t a userful option.

Screen Shot 2016-10-31 at 15.57.25.png

This is false advertising

Screen Shot 2016-10-31 at 15.59.41.png

This is also false advertising

Some of you might perhaps suggest that I didn’t optimize my CrashPlan settings so that I could get the most out of it. I did. I tried everything they suggested in their online support notes. In addition to tricking out my Crashplan install, my computer has been on for virtually all of the last four years, in an effort to help the Crashplan app finish the initial backup, to no avail.

Another thing that bothered me about CrashPlan is that it would go into “maintenance mode” very often, and given the size of my backup set, this would take days, sometimes weeks, during which it wouldn’t back up. It would endlessly churn through its backup versions and compare them to my data, pruning out stuff, doing its own thing and eating up processor cycles with those activities instead of backing up my data.

Screen Shot 2016-10-22 at 19.40.33.png

Synchronizing block information…

Screen Shot 2016-10-23 at 14.39.36.png

Compacting data… for 22.8 days…

Screen Shot 2016-10-23 at 16.58.23.png

Maintaining backup files…

I understand why maintenance of the backups is important. But what I don’t understand is why it took so long. I can’t help thinking that maybe the cause is the Java-based backup engine that CrashPlan uses. It’s not a Mac-native app or a Windows-native app. It’s a Java app wrapped in Mac and Windows app versions. And most Java apps aren’t known for their speed. It’s true, Java apps could be fast, but the developers often get lazy and don’t optimize the code — or that’s the claim made by some experts in online forums.

Another way to look at this situation is that CrashPlan has a “freemium” business model. In other words, their app is free to use for local (DAS or NAS) backup or offsite backup (such as to a friend’s computer). And one thing I know is that you can’t complain about something that’s given freely to you. If it’s free, you either offer constructive criticism or you shut up about it. It’s free and the developers are under no obligation to heed your feedback or to make changes because you say so. As a matter of fact, I used CrashPlan as a free service for local backup for a couple of years before I started paying for their cloud backup service. But it was only after I started paying that I had certain expectations of performance. And in spite of those unmet expectations, I stuck with them for four years, patiently waiting for them to deliver on their promise of “no storage limits, bandwidth throttling or well-engineered excuses”… and they didn’t deliver.

Here I should also say that CrashPlan support is responsive. Even when I was using their free backup service, I could file support tickets and get answers. They always tried to resolve my issues. That’s a good thing. It’s important to point this out, because customer service is an important aspect of a business in the services industry — and online backups are a service.

About three weeks ago, I was talking with Mark Fuccio from Drobo about my issues with CrashPlan and he suggested I try Backblaze, because they truly have no throttling. So I downloaded the Backblaze app (which is a native Mac app, not a Java app), created an account and started to use their service. Lo and behold, the 15-day trial period wasn’t yet over and my backup to their servers was almost complete! I couldn’t believe it! Thank you Mark! 🙂

I optimized the Backblaze settings by allowing it to use as much of my ISP bandwidth as it needed (I have a 100 Mbps connection), and I also bumped the number of backup threads to 10, meaning the Backblaze app could initiate 10 separate instances of itself and upload all 10 instances simultaneously to their servers. I did have to put up with a slightly sluggish computer during the initial backup, but for the first time in many years, I was able to back up all of my critical data to the cloud. I find that truly amazing in and of itself.

Screen Shot 2016-10-14 at 21.36.27.png

This is what I did to optimize my Backblaze installation

As you can see from the image above, I got upload speeds over 100 Mbps when I optimized the backup settings. During most of the days of the initial upload, I actually got speeds in excess of 130 Mbps, which I think is pretty amazing given my situation: I live in Romania and the Backblaze servers are in California, so my data had to go through a lot of internet backbones and through the trans-Atlantic cables.

The short of it is that I signed up for a paid plan with Backblaze and my initial backup completed in about 20 days. Let me state that again: I backed up about 9 TB of data to Backblaze in about 20 days, and I managed to back up only about 1 TB of data to CrashPlan in about 4 years (1420 days). The difference is striking and speaks volumes about the ridiculous amount of throttling that CrashPlan puts in place for heavy users like me.

I also use CrashPlan for local network backup to my Drobo 5N, but I may switch to another app for this as well, for two reasons: it’s slow and it does a lot of maintenance on the backup set and because it doesn’t let me use Drobo shares mapped through the Drobo Dashboard app, which is a more stable way of mapping a Drobo’s network shares. CrashPlan refuses to see those shares and requires me to manually map network shares, which isn’t as stable a connection and leads to share disconnects and multiple mounts, which is something that screws up CrashPlan. I’m trying out Mac Backup Guru, which is a Mac-native app, is pretty fast and does allow me to back up to Drobo Dashboard-mapped shares. If this paragraph doesn’t make sense to you, it’s okay. You probably haven’t run into this issue. If you have, you know what I’m talking about.

Now, none of this stuff matters if you’re a typical user of cloud backup services. If you only have about 1 TB of data or less, any cloud backup service will likely work for you. You’ll be happy with CrashPlan and you’ll be happy with their customer service. But if you’re like me and you have a lot of data to back up, then a service like Backblaze that is truly throttle-free is exactly what you’ll need.

Standard
How To

How I recovered from catastrophic data loss

In late 2012 and right after New Year’s Eve this year, in 2015, I experienced two data loss events, both of which happened on my Drobo storage devices. I’ll write a separate post detailing my experiences in recent years with my Drobos but for now, I wanted to let you know how I recovered my files.

First, what do I mean by “catastrophic data loss”? Simple: the loss of terabytes of my very important data: photos, videos, documents. Among other things, I am a photographer and a filmmaker. Losing my photos and my videos is a catastrophic event, as my libraries and archives include both personal and professional photos and videos. If I were to lose these things, I’d lose both treasured memories and part of my livelihood.

Here I should also point out that all of us are at risk of data loss. Most of our stuff is digital these days (or going that way). What would you do if you’d lose all your photos and videos? Think about that question and put a plan of action in place. Follow through with it and make sure you’re covered.

Now let me get the bad part out of the way: in 2012, I lost somewhere between 25,000 – 30,000 photos and I still haven’t counted how many videos, but it was a lot, probably about 20% of my video library. This is stuff I’ll never get back. It’s gone. Period. Who’s to blame? The Drobo. More on that in this post.

Earlier this year, I could have lost an untold number of files once more but I didn’t. Why? Partially because the Drobo has improved in the way it’s handling errors but mostly because I had access to good software.

Here are the three methods of data recovery I’ll talk about below:

  • Data Rescue: it’s a piece of software that lets you mount bad drives and get your files backed up somewhere else. This let me copy all of the files it could read off the Drobo, although a lot of them ended up being corrupted, as detailed above.
  • Jeffrey Friedl’s Preview-Cache Image Extraction: this is a Lightroom plugin that allowed me to extract image previews for the lost images directly from my Lightroom catalogs. It’s a niche plugin but it’s super useful. You don’t realize just how good it is until you have to use it and then you thank the heavens that it exists.
  • Flickr and YouTube: I was able to download images and videos I’d published to Flickr and YouTube at their maximum upload resolution. They may not have been my digital negatives or my raw video files, which were lost forever, but at least I had something left. This is why I’ve started to upload to both Flickr and YouTube at the best resolution and quality possible, in case something like this happens again.

If you’re pressed for time, feel free to stop here. Make sure you use the methods outlined above and you’ll fare much better if you should lose your data, particularly if you’re working in visual media like I am. If you want the details, read on.

Data Rescue

Back in 2012, I was able to mount the corrupted Drobo volume using Data Rescue 3 and recover the bulk of my files. As mentioned above, Data Rescue was able to see all of the files, including the corrupted ones and it let me copy them off, but 25,000 – 30,000 of a total of about 130,000 photographs and I don’t know how many videos were corrupted and couldn’t be read by either Lightroom, Photoshop, Final Cut Pro or Quicktime, so they were of no use to me. They were gone. These were original RAW, DNG, TIF and JPG files from my cameras and SD and HD video files (MP4, MOV and AVI) from my video cameras. I also lost a great deal of family videos and films and cartoons I’d painstakingly digitized from VHS tapes and DVDs I’d purchased, as well as shows and films I’d recorded from TV using a DVR and then edited and stored on the Drobo. In most cases, the files just wouldn’t open up at all. In other cases, I could open them but half or more than half of the image was gone, as you see below.

This is one of my wedding photographs. Most of my wedding photographs look like this or worse…

At our weddingHere’s another. This used to be a photograph of a cliff.

Cheile Turului

I could give you many more examples but the point is, they were irreversibly damaged when the Drobo decided to go kaput.

I don’t know what I would have done without Data Rescue. Because I bought it and used it, I was able to save 70-80% of my data after my 1st catastrophic data loss event and 100% of my data during my recent data loss event.

You may say it’s not data loss and it’s not catastrophic if I was able to recover the data. To that I say that I’d have recovered 0% of my data in both cases without Data Rescue and 0% of over 8 TB of data is damned catastrophic in my book.

Jeffrey Friedl’s Preview-Cache Image Extraction

This super-useful and little-known plugin for Lightroom allows you to extract JPG files from the preview images stored in your Lightroom catalogs. That means that even if you lose the original raw files, you can still have the JPGs and that’s a huge thing.

There’s one caveat though: you need to have allowed Lightroom to keep the previews and you also need to have allowed Lightroom to store high-quality previews. I won’t get into the exact terminology here, there are plenty of tutorials on the internet that will teach you how to optimize those settings. Suffice it to say that I now have my catalogs set to create 1:1 previews and to never delete them, just in case I ever experience data loss again.

I didn’t do this in the past, which meant that I was only able to recover thumbnails or smaller JPGs for most of my corrupted photos, but this was still better than nothing. I have precious photos of my wife that are thumbnail-sized, but at least I have those, I was able to get something back from the gaping maw of data loss.

Flickr and YouTube

These two websites aren’t just for sharing photos and videos. They also let you download your originals. Well, Flickr lets you download your originals. YouTube only lets you download MP4 files of your videos but hey, it’s wonderful anyway.

By the way, the Flickr mobile app and the Google Plus mobile app (for iOS and Android) both let you automatically back up the photos taken with that phone to your respective accounts on both services. They’re set to private by default so only you see them. That’s really nice.

Flickr download options

YouTube download options

This is why I now upload all my published photos to Flickr at their highest resolution and quality and why I also upload all my published videos to YouTube at their highest resolution and quality. In case I ever experience data loss in the future, I’ll have part of my photo and video library on these sites and I’ll be able to download it. And this is also why I no longer put watermarks on my photos. It’s no good to be able to download your own original and have a watermark on it. You now either have to crop it or Photoshop it. I have no time for that sort of thing. I’d rather deal with more productive stuff.

Of course, JPGs aren’t DNGs or RAW files but if they’re the highest resolution, dpi and quality available, they’ll do just fine. And an edited 1080p MP4 file isn’t the same thing as the original Final Cut Pro event and project along with the original video and audio files that were used to create it, but if you don’t have those anymore, you’ll be very thankful to have the MP4.

Now, for some less-than-obvious stuff…

But Raoul, why don’t you back up your stuff? That would solve all your problems! 

I do back up my stuff. I’ve been using CrashPlan for years and I also use Time Machine to back up the files on my Mac (but not all my files are on my Mac, they don’t all fit on it). Unfortunately, during my first data loss event in 2012, I was re-structuring my backup sets and the Drobo couldn’t have picked a worst time to fail. If I had relied on my backups, I’d have recovered only about 25% of my data.

This year, I was doing a little better, although I was also re-structuring my backup sets. Somehow these things seem to know when to fail just to cause more headaches (my warranty had also just run out about 3-4 days before it failed). That brought to mind images of planned obsolescence…

This time I’d have recovered about 80% of my data from the backups. Not ideal but much better than before. I can go into my backup strategy at a later time, but it’s much more difficult for me to back up all my stuff than it is for you, simply because I have a ton of data and I always run into bandwidth issues. For example, one of my backup jobs has to keep up with 8.1 TB of data. The other, with 6 TB of data. And I don’t add small amounts of data to those backup sets, I add gigabytes, lots of gigabytes, whenever I have a studio shoot or take a trip, whether it be photos or videos.

But Raoul, why do you keep using the Drobo when it keeps failing? 

The basic premise of a Drobo, that of using SATA drives of different sizes, from different manufacturers in a single array that can show up as a single 16 TB volume on my Mac, and also allow for one (or two) of those drives to fail while keeping the data safe, still cannot be beaten by anything else on the market. If you know of anything else that meets those criteria, let me know. The Drobo has its drawbacks and data corruption is one of them. Drobos also brick themselves quite a lot, just search for that phrase and you’ll see what I mean. They’re not to be relied upon but they provide the basic benefit outlined above.

But Raoul, you could have used photo recovery software to get all those tens of thousands of photos back! Why didn’t you? 

Back in 2012, I knew of no such software. Now I believe there are several options available and some allow for batch processing of corrupted photos. I haven’t tried any of them yet so I can’t tell you anthing about them. I doubt that any software can do much when half of a photo’s pixels are missing. Besides, I didn’t need to use them after my latest data loss, I was able to get it all back with Data Rescue.

But Raoul, you could have sent your Drobo in to a professional data recovery service. Couldn’t they have done a much better job? 

Maybe. I did get a couple of quotes. They ran anywhere from $3,000 – over $10,000 and they couldn’t guarantee they’d get all my data back. What also made things more complicated and expensive was shipping my drives to the US, where these companies were located. I live abroad and the customs are such a headache I try to avoid dealing with them whenever I can.

Standard