Events

The time a Drobo lost over 30,000 of my photos and videos

The following is an account of what happened to me in late 2012, when two of my Drobo units malfunctioned and corrupted or erased over 30,000 of my photos and videos, rendering them either completely unusable or making them vanish into digital ether.

I have been a Drobo customer since late 2007, when I bought my first USB Drobo. I was glad to take advantage of the new storage possibilities it offered. Since then, I’ve bought one or two Firewire Drobos, two Drobo 5Ns and a Drobo 5D. After relying on my Drobos for several years, I experienced catastrophic data corruption on both of my Drobos at the time (a USB Drobo and a Firewire Drobo). That incident still haunts me to this day.

If you look through my past posts about the Drobo, you’ll see that I wrote about them and I defended them (speaking from my own experience at the time) when others said, among other things, that they “can’t be relied upon” and they often turn into “bricks” or lose people’s data. From my own experience, I had only good things to say about them. They hadn’t failed me for years, I hadn’t yet lost any data from them, the support was great and their design is beautiful. They are a pleasure to use when they work. Until recently, few things in hardware got me more excited than buying a new, bigger hard drive for my Drobo and sticking it in, then watching the capacity gauge show the new, larger volume and hearing it start to sync the data to the new drive…

Which brings me neatly right to the crux of the problem. I had bought a new 3TB hard drive for my Firewire Drobo and stuck it in, replacing an existing 2TB drive. The data sync had gone on for close to 30 hours and was getting close to completion (the Dashboard software was telling me there were only 5-6 hours left). All of a sudden, a red light next to the new 3TB hard drive went off. Before I had a chance to assess the situation, the Drobo crashed and rebooted. I thought, okay, it’s resetting itself and will begin to work once more. I waited for the reboot to complete, only to discover, to my dismay, that it rebooted again… and again… and again… During those brief moments when it was up, the red light switched places, moving from the 3TB hard drive to one of the existing 2TB hard drives. This wasn’t good. I decided to shut it off completely and unplug it. I filed a ticket with Drobo Support and waited.

This horrific experience came after I experienced catastrophic data loss on my USB Drobo just a couple of days earlier. That Drobo was and had been plugged into a surge protector (unlike my Firewire Drobo, which had been plugged into a UPS). The electricity had been cut off and when it came back up, the USB Drobo wouldn’t mount to the desktop. No big deal, I thought. The electricity had gone off before and when the Drobo wouldn’t mount, I would run Disk Utility on it and fix it right up. That’s what I did this time, only Disk Utility couldn’t repair it. Oh boy… I filed a ticket with Drobo Support. They told me they couldn’t help and that I should buy Disk Warrior or send it to a data recovery firm. I called the firm to get a quote, which they said would be anywhere from $1,200 to $7,900, depending on how much data they recovered. They told me that realistically, it would cost at least $4,000 if they recovered any data at all, and that it’s difficult to get the data off a corrupt Drobo volume, because of their complicated storage algorithms. They said it’s much easier with traditional RAID systems.

So I bought Disk Warrior and tried my luck. It was able to mount the Drobo temporarily, but couldn’t fix it. The error it gave me was that Disk Utility had messed up the catalog file when it tried to fix it and they couldn’t recreate it. I called their Tech Support and one of their technicians stayed on the phone with me for over an hour. We did a Screen Sharing session in iChat, where he kept running scans on the volume and checking the diagnostic files. Same result. Couldn’t be fixed. He suggested I get whatever data I could get off it directly through the Disk Warrior’s Preview feature. I started to copy off the data and got about 70GB off (onto the Firewire Drobo) which hadn’t crashed yet. Then the USB Drobo decided to call it quits and became completely unusable.

Out of over 3TB of data, I was able to salvage temporarily about 70GB. I lost the rest: over 15 years of personal and family information, photos, personal videos, letters, our movie collection (which I had painstakingly digitized and edited from VHS tapes and TV recordings over several years), our cartoon collection (also painstakingly digitized and edited from VHS tapes and TV over several years, and which I was looking forward to watching with our unborn baby — my wife was pregnant at the time; our daughter is now almost six years old) and other various files.

I decided not to send the Drobo to data recovery because the price to recover the data would have been higher than the price to buy all of the movies and cartoons on DVD once more, and because I don’t feel comfortable with other people rifling through my personal files. I didn’t know how much they’d be able to recover, if anything, and I consoled myself with the notion that at least my Firewire Drobo was still fine and I had my work files (my photo library of over 100,000 photos, the footage and FCP and iMovie events and projects from our shows (Romania Through Their Eyes and Ligia’s Kitchen), various other personal and published videos, etc. — basically, my creative work over the past 6-7 years.

Then, my Firewire Drobo went AWOL. As mentioned above, I filed a ticket with Drobo Support and waited to see what they would say. Their advice: buy Data Rescue 3 and run it on the 2TB drive that showed up with a red light next to it. Apparently the drive went bad during the data sync for the new 3TB drive. With Data Rescue 3, I could in theory clone it, then insert the cloned drive into the Drobo and recover my data. Well, I couldn’t clone the whole drive. I tried it a couple of times, but whenever the head came to a certain point on the platters, the drive would disconnect itself from the computer. So I did a reverse clone (where the cloning process begins at the end, not the start of the drive) to try and recover the rest of the data after the bad section on the drive. After that completed and with the blessing of Drobo Support, I called that a “best try” clone, stuck the new drive into the Drobo, put it in Read Only Mode and through Data Rescue 3, began to copy off my data.

Things looked good at the time. The software seemed to see all my files and although the copying process was slow, I had access to my data, which was the important thing. After 4-5 days of data retrieval, I was done and at Drobo Support’s advice, I reset the Drobo and formatted a new volume onto it, then began to copy my data back onto it. It was deemed healthy once again and I was sent a new power supply. I’m still not sure why the new power supply was needed, but alright.

After all my data was back onto the newly formatted Firewire Drobo, I began to access it. Only then did I discover data corruption at the file level. Photos, precious memories which I’d trusted my Drobo to keep were now forever lost. Photos of when I met Ligia (my wife), photos from the wedding, from our first years together… they’re gone now. Videos have turned into audio files (the video track can no longer be seen by video players or video editing programs. Just from the year 2007, I lost well over 17,000 photos (about 80% of them). The same percentage applied to the years 2000-2006, when I took less photos but lost the same proportion of files (about 80%). I still don’t know the exact number of photos I lost, but I estimate it to be well over 30,000. I still dread to open older video libraries in Final Cut Pro, because when I access my projects, I always find video clips that are either missing or completely unusable.

When I updated my ticket with Drobo Support to let them know about this, data recovery services were once more suggested. Now I’m not a data recovery expert, but after you’ve formatted a volume and written a LOT of data to it, doesn’t that make it harder, if not impossible, to get the old data back?

If you’d like to see what file-level corruption looks like, it’s not pretty. Here are a few screenshots. Those thumbnails are all I have left of photos I wanted to keep forever.

Let’s look at this thing from both sides.

The pros:

  • These two Drobos worked for five years, flawlessly;
  • My experience with Drobo Support was good from 2007 till 2012; even though the warranty had expired on both of these Drobos, they took up my cases and tried to help me;
  • I should have had my data backed in other places, I know, I know; That’s exactly what I as doing as the Drobos failed. I was reorganizing files and reshuffling my archives when it happened. If you were to ask me, as an IT professional, what I would recommend to others, it would have been the following: have the data in its primary location on a reliable and fast device, back it up to a server or NAS on the local network and also back it up offsite. I was caught in-between, with no access to my old backups and no proper backup plan in place. Crashplan, the offsite backup company I was using at the time, had been throttling my uploads to conserve their bandwidth, and the offsite backups hadn’t completed. If they hadn’t been actively engaged in throttling user uploads, as was their practice at the time, I could have recovered all my files.
  • Drobos are easy to use and allow one to consolidate their data on one volume that can keep growing and growing and growing with the addition of new drives — that part I always liked;
  • Even if one drive fails, the data’s still safe; I had that happen to me once in years past and yes, the data was fine — that part I also always liked;
  • I should have had the USB Drobo plugged into a UPS, just like the Firewire Drobo. Should have, would have, could have…

The cons:

  • Two catastrophic failures within one week… c’mon, this is ridiculous to say the least…
  • I had no idea you could lose all your data from the Drobo if the power went out, and that it would cost upwards of $5,000 to get it back. Ugh… if it costs so much to get the data back, Data Robotics should seriously consider doing what ioSafe has been doing from the start: offering Data Recovery Services standard with every one of their products.
  • Apparently the newer Drobo products, like the 5D, the Drobo Mini and the 5N, have a built-in battery that protects against power outages, so what happened with my USB Drobo won’t happen to them, theoretically. Well… if they knew about this, couldn’t they have put that battery/capacitor in the older Drobos as well? I sure wish they had taken every step possible to make sure any data placed on a Drobo stayed safe. After all, they’ve been saying for years that they “protect what matters”. Do they really?
  • Earlier this year, a friend of mine wrote to me telling me of his experience with his USB Drobo. It was the very same thing that happened to mine, minus the power outage. He would eject it from the desktop and unplug it from the outlet, thinking that was safe. It was, for a while, until he lost all his data. At the time, I thought he must have been doing something wrong. Until it happened to me. Is it just me, or do we not do the same with a regular external drive? Do we not eject it then turn it off? Does the electricity not go off sometimes, and even if the Drobo is plugged into a UPS, the electricity may be out for more than 10 minutes, meaning the UPS will also go down? And when the electricity comes back up, could we not also lose our data even though we did everything right?
  • I remember jokingly asking someone from Data Robotics in years past what would happen if a drive went bad right after you inserted a new one and the Drobo was still in Data Protection mode. Their response: not very likely that could happen. The Drobo would know that a drive was about to go bad and would give me warning. It’s quite apparent to me now that that’s not the way things work sometimes. Sometimes a drive does go bad during Data Protection and not only do you get no warning, you also get catastrophic data loss. Double whammy.
  • Someone from Drobo Sales happened to write to me to invite me to a webinar as I was in the throes of my data loss. I replied and told him of my troubles, and that was the last I heard from him. Was that the way to treat a long-time customer? An adequate response would have been “Hey, I’m really sorry about that. Are you getting the help you need from Tech Support? If not, do you want me to step in for you?”

This whole affair left me exhausted and completely disappointed. I wasted so much time trying to recover my data and my bearings. The experience has taught me a few things, one of them being the loss of my trust in Drobo. It is a beautiful little black box. It works great and you begin to rely on it more and more, till you can’t do without it; till you think it works so good your data’s going to be fine all the time. Then it fails and your data is gone.

If you’d like to know how I recovered some of my data, I wrote about it here. I also experienced another major data loss event with my Drobo 5D just last month, and that was after more data loss with my 5D in 2015 and 2017. Add to that ongoing file system corruption in one of my Drobo 5N units and the “Drobo data loss package” is complete. I don’t know, I must have unlocked some kind of secret data loss level-up in the data storage game…

Standard

Today is the International Day of the Romanian Blouse, and it is also the first day of an exhibition of a part of Ligia’s collection of traditional, authentic Romanian costumes (blouses, skirts and more). About twenty costumes are currently on display at the Museum of Natural Gas in Medias and can be seen there during normal business hours through the end of this week (the 29th of June). Admittance is free.

Some time ago, the two of us decided to call Ligia’s collection “Straie Alese“, because it includes authentic, rare, vintage, hand-woven and loom-woven articles of clothing from all parts of Romania. Most pieces are 50-70 years old and some are more than a century old. For those who have already asked, most are not for sale. A select few of the costumes are sometimes offered for sale.

The launch of the exhibition was held earlier today. You can see the photo gallery below. Now, an exhibition at a museum does not happen by itself. There were a tight-knit group of women who keep the tradition alive in our city and who were glad to find out about Ligia’s collection and they worked with Ligia to organize the showing, so we’d like to thank them here for their guidance and invaluable assistance. The exhibition would not have happened without them! 👏🙌🌻

We’re glad this first event happened right here in our city, which, situated as it is in the geographical heart of Romania and in the bosom of Transilvania, seems especially favored by fate to be a good starting place for a project that can reach far and wide. We are of course open to collaborations with other museums, here in Romania and abroad.

Enjoy the photographs!

Events

An exhibition of Ligia’s traditional Romanian costumes

Gallery
Events

On the evening of the summer solstice

It was the summer solstice a couple of days ago. On that longest day of the year, I thought it might be interesting to take photographs in our garden, just as the day drew to a close. I was especially interested in capturing the evening primroses, because we have so many of them this year, and because I was curious to see whether they’d open up sooner or just as dark fell, as is their custom.

Well, they didn’t open up sooner, I still had to wait until it was almost dark, but I found out something new; having never watched them open up before, I had a wonderful surprise in store for me. As it turns out, the blossoms open at a pace that can be observed live, without the aid of time lapse photography. It’s not as fast as a mimosa pudica’s reaction, but it’s fast enough for a person to stand (or sit) there and watch it open. Nature is wonderful!

I hope you enjoy this set of photographs. I took them with my PEN E-P3, whose IBIS mechanism was recently repaired by Olympus Czechia and is now working so well, it’s tempting to use it as my main camera. I plan to use it a lot more than before, that’s for sure. By the way, my setup included the M.Zuiko 25mm f1.8 lens and the lovely and inexpensive MCON-P02 Macro Adapter. It works with several more lenses from Olympus and it’s small and lightweight, so it even fits in your pocket.

Standard
Florin Rosoga Podcast
Events

An interview about entrepreneurship and more

I was recently interviewed by Florin Roșoga, an entrepreneur from Cluj-Napoca (one of the cities near and dear to my heart) on the topic of, well, me… The discussion centered around entrepreneurship, work ethic and money. I tried to be as frank and direct as possible. The interview is in Romanian, it’s in podcast format and was published here. I’ve embedded the YT and MP3 players below. I hope it’s helpful to you (provided you understand Romanian 😀)!

https://www.youtube.com/watch?v=m4GRsc2Q5wE

Standard
Events, Places, Video Log

Proud To Be Romanian at Cetatea Făgărașului

Ligia and I were in attendance at a wonderful event last night, organized by the good folks from Proud To Be Romanian at Cetatea Făgărașului, a medieval fortress whose construction began in 1310 and continued through various repairs and improvements well into the 1630s.

It goes without saying that I love medieval fortresses and castles. I feel right at home whenever I visit one. I loved the architecture here, the various tunnels and cellars that run under its walls, the beautiful, grandiose rooms and hallways but most of all and perhaps a little odd, the window encasements which were made of carved stone. They were so beautifully and delicately made and were perfect for the style of the castle.

We got there a little before the entertainment started. Being there ahead of time gave us the opportunity to explore the castle and take some photos, which you’ll get to see here. The evening’s festivities involved wine and champagne tastings, hors d’oeuvres, some networking, a ceremony celebrating those who are doing good things in Romania (Ligia was among those who were feted) and a concert in the castle’s inner courtyard.

I also shot some 360° video with my Giroptic camera (it’s embedded below and may not display properly in certain browsers like Safari on a Mac). I have to apologize for its quality. While the novelty of this kind of video kind of makes up for the camera’s technical inability to record proper HD video, it’s not enough to recommend it. And when you hear the bad sound recorded with its microphones, you’re even more put off. Again, sorry… If any of you know how to improve the quality of the video captured with this camera, please let me know.

But enough whining! The event was great, the champagne and the wine were great and the castle was amazing! We loved it! One of the people we got to meet was Mrs. Simona-Mirela Miculescu, Representative of the UN Secretary General and Head of the UN Office at United Nations (you’ll see her in one of the photos with Ligia) and you should have seen my face when she said she knew me and liked my show, Romania Through Their Eyes… 😳

Thank you Adriana and Rob for putting this wonderful shindig together!

Standard