How To

How I recovered from catastrophic data loss

In late 2012 and right after New Year’s Eve this year, in 2015, I experienced two data loss events, both of which happened on my Drobo storage devices. I’ll write a separate post detailing my experiences in recent years with my Drobos but for now, I wanted to let you know how I recovered my files.

First, what do I mean by “catastrophic data loss”? Simple: the loss of terabytes of my very important data: photos, videos, documents. Among other things, I am a photographer and a filmmaker. Losing my photos and my videos is a catastrophic event, as my libraries and archives include both personal and professional photos and videos. If I were to lose these things, I’d lose both treasured memories and part of my livelihood.

Here I should also point out that all of us are at risk of data loss. Most of our stuff is digital these days (or going that way). What would you do if you’d lose all your photos and videos? Think about that question and put a plan of action in place. Follow through with it and make sure you’re covered.

Now let me get the bad part out of the way: in 2012, I lost somewhere between 25,000 – 30,000 photos and I still haven’t counted how many videos, but it was a lot, probably about 20% of my video library. This is stuff I’ll never get back. It’s gone. Period. Who’s to blame? The Drobo. More on that in this post.

Earlier this year, I could have lost an untold number of files once more but I didn’t. Why? Partially because the Drobo has improved in the way it’s handling errors but mostly because I had access to good software.

Here are the three methods of data recovery I’ll talk about below:

  • Data Rescue: it’s a piece of software that lets you mount bad drives and get your files backed up somewhere else. This let me copy all of the files it could read off the Drobo, although a lot of them ended up being corrupted, as detailed above.
  • Jeffrey Friedl’s Preview-Cache Image Extraction: this is a Lightroom plugin that allowed me to extract image previews for the lost images directly from my Lightroom catalogs. It’s a niche plugin but it’s super useful. You don’t realize just how good it is until you have to use it and then you thank the heavens that it exists.
  • Flickr and YouTube: I was able to download images and videos I’d published to Flickr and YouTube at their maximum upload resolution. They may not have been my digital negatives or my raw video files, which were lost forever, but at least I had something left. This is why I’ve started to upload to both Flickr and YouTube at the best resolution and quality possible, in case something like this happens again.

If you’re pressed for time, feel free to stop here. Make sure you use the methods outlined above and you’ll fare much better if you should lose your data, particularly if you’re working in visual media like I am. If you want the details, read on.

Data Rescue

Back in 2012, I was able to mount the corrupted Drobo volume using Data Rescue 3 and recover the bulk of my files. As mentioned above, Data Rescue was able to see all of the files, including the corrupted ones and it let me copy them off, but 25,000 – 30,000 of a total of about 130,000 photographs and I don’t know how many videos were corrupted and couldn’t be read by either Lightroom, Photoshop, Final Cut Pro or Quicktime, so they were of no use to me. They were gone. These were original RAW, DNG, TIF and JPG files from my cameras and SD and HD video files (MP4, MOV and AVI) from my video cameras. I also lost a great deal of family videos and films and cartoons I’d painstakingly digitized from VHS tapes and DVDs I’d purchased, as well as shows and films I’d recorded from TV using a DVR and then edited and stored on the Drobo. In most cases, the files just wouldn’t open up at all. In other cases, I could open them but half or more than half of the image was gone, as you see below.

This is one of my wedding photographs. Most of my wedding photographs look like this or worse…

At our weddingHere’s another. This used to be a photograph of a cliff.

Cheile Turului

I could give you many more examples but the point is, they were irreversibly damaged when the Drobo decided to go kaput.

I don’t know what I would have done without Data Rescue. Because I bought it and used it, I was able to save 70-80% of my data after my 1st catastrophic data loss event and 100% of my data during my recent data loss event.

You may say it’s not data loss and it’s not catastrophic if I was able to recover the data. To that I say that I’d have recovered 0% of my data in both cases without Data Rescue and 0% of over 8 TB of data is damned catastrophic in my book.

Jeffrey Friedl’s Preview-Cache Image Extraction

This super-useful and little-known plugin for Lightroom allows you to extract JPG files from the preview images stored in your Lightroom catalogs. That means that even if you lose the original raw files, you can still have the JPGs and that’s a huge thing.

There’s one caveat though: you need to have allowed Lightroom to keep the previews and you also need to have allowed Lightroom to store high-quality previews. I won’t get into the exact terminology here, there are plenty of tutorials on the internet that will teach you how to optimize those settings. Suffice it to say that I now have my catalogs set to create 1:1 previews and to never delete them, just in case I ever experience data loss again.

I didn’t do this in the past, which meant that I was only able to recover thumbnails or smaller JPGs for most of my corrupted photos, but this was still better than nothing. I have precious photos of my wife that are thumbnail-sized, but at least I have those, I was able to get something back from the gaping maw of data loss.

Flickr and YouTube

These two websites aren’t just for sharing photos and videos. They also let you download your originals. Well, Flickr lets you download your originals. YouTube only lets you download MP4 files of your videos but hey, it’s wonderful anyway.

By the way, the Flickr mobile app and the Google Plus mobile app (for iOS and Android) both let you automatically back up the photos taken with that phone to your respective accounts on both services. They’re set to private by default so only you see them. That’s really nice.

Flickr download options

YouTube download options

This is why I now upload all my published photos to Flickr at their highest resolution and quality and why I also upload all my published videos to YouTube at their highest resolution and quality. In case I ever experience data loss in the future, I’ll have part of my photo and video library on these sites and I’ll be able to download it. And this is also why I no longer put watermarks on my photos. It’s no good to be able to download your own original and have a watermark on it. You now either have to crop it or Photoshop it. I have no time for that sort of thing. I’d rather deal with more productive stuff.

Of course, JPGs aren’t DNGs or RAW files but if they’re the highest resolution, dpi and quality available, they’ll do just fine. And an edited 1080p MP4 file isn’t the same thing as the original Final Cut Pro event and project along with the original video and audio files that were used to create it, but if you don’t have those anymore, you’ll be very thankful to have the MP4.

Now, for some less-than-obvious stuff…

But Raoul, why don’t you back up your stuff? That would solve all your problems! 

I do back up my stuff. I’ve been using CrashPlan for years and I also use Time Machine to back up the files on my Mac (but not all my files are on my Mac, they don’t all fit on it). Unfortunately, during my first data loss event in 2012, I was re-structuring my backup sets and the Drobo couldn’t have picked a worst time to fail. If I had relied on my backups, I’d have recovered only about 25% of my data.

This year, I was doing a little better, although I was also re-structuring my backup sets. Somehow these things seem to know when to fail just to cause more headaches (my warranty had also just run out about 3-4 days before it failed). That brought to mind images of planned obsolescence…

This time I’d have recovered about 80% of my data from the backups. Not ideal but much better than before. I can go into my backup strategy at a later time, but it’s much more difficult for me to back up all my stuff than it is for you, simply because I have a ton of data and I always run into bandwidth issues. For example, one of my backup jobs has to keep up with 8.1 TB of data. The other, with 6 TB of data. And I don’t add small amounts of data to those backup sets, I add gigabytes, lots of gigabytes, whenever I have a studio shoot or take a trip, whether it be photos or videos.

But Raoul, why do you keep using the Drobo when it keeps failing? 

The basic premise of a Drobo, that of using SATA drives of different sizes, from different manufacturers in a single array that can show up as a single 16 TB volume on my Mac, and also allow for one (or two) of those drives to fail while keeping the data safe, still cannot be beaten by anything else on the market. If you know of anything else that meets those criteria, let me know. The Drobo has its drawbacks and data corruption is one of them. Drobos also brick themselves quite a lot, just search for that phrase and you’ll see what I mean. They’re not to be relied upon but they provide the basic benefit outlined above.

But Raoul, you could have used photo recovery software to get all those tens of thousands of photos back! Why didn’t you? 

Back in 2012, I knew of no such software. Now I believe there are several options available and some allow for batch processing of corrupted photos. I haven’t tried any of them yet so I can’t tell you anthing about them. I doubt that any software can do much when half of a photo’s pixels are missing. Besides, I didn’t need to use them after my latest data loss, I was able to get it all back with Data Rescue.

But Raoul, you could have sent your Drobo in to a professional data recovery service. Couldn’t they have done a much better job? 

Maybe. I did get a couple of quotes. They ran anywhere from $3,000 – over $10,000 and they couldn’t guarantee they’d get all my data back. What also made things more complicated and expensive was shipping my drives to the US, where these companies were located. I live abroad and the customs are such a headache I try to avoid dealing with them whenever I can.

Standard
Thoughts

Where’s Google Photos in this drop-down menu?

If you use FeedBurner (which has been part of Google for a good number of years now), you probably know about the Photo Splicer feature, which allows you to merge your photo feed from services like Flickr, BuzzNet or Webshots into your site feed, providing extra content for your readers. It’s a great little option and I hope Google keeps it turned on for years to come.

My question for Googlers reading this is simple: where’s Google Photos (PicasaWeb) in that drop-down menu? Isn’t it about time for it to show up there?

Standard
Events

Father and Son: STS-1 and STS-135

Here’s Chris Bray and his father at the space shuttle launch, in 1981 and 30 years later, in 2011.

Excerpts from an interview with him done by the Flickr staff:

Early on the morning of April 12, 1981, after a couple of delays, the space shuttle Columbia was finally getting ready for lift-off to the first ever shuttle mission into space. One of the spectators, Flickr member arockalypse, was standing in the crowd, being terribly excited and anxious. He told us he grew up loving astronomy and following the space program, but until that day he had only seen Apollo launches on TV, so this was a huge thrill.
30 years later, when STS-135, the last space shuttle mission before retiring of the program, was planned, arockalypse and his family entered a lottery for tickets and managed to score passes to the Astronaut Hall of Fame viewing area.
When the launch finally took place, “it was bittersweet. We are big supporters and followers of the space program, and it is somewhat sad to see this great era come to an end. However, the Shuttle was a fantastic program and despite 2 horrible tragedies, it was an amazingly successful one. I’m looking forward to what comes next!”

via Father & Son « Flickr Blog. Photo by Chris Bray.

Standard
Thoughts

My wishlist for Flickr

I joined Flickr in December 2004, and stuck with it since then because I like it. In spite of the various censorship issues that have plagued it over time, it’s still one of the best (if not the best) places to share photos. (Videos are another matter…)

Here’s a short list of what could make it even better:

  • Ability to view ALL of the photos that my contacts have uploaded. This is already available in the Flickr app for iPod touch and iPhone, but it still hasn’t been implemented on the Flickr website. It should be a simple option that would allow me to choose between seeing All or Five or One new photo(s) from my contacts. If I choose All, then when I go to my Contacts page, I see everything they’ve uploaded. It would be really useful.
  • Ability to set a licensing price for our photos. Flickr should get a bit of that money, and since they’re in with Getty these days, I suppose Getty ought to get something too, but certainly not what they’re getting now with the “Request to License” option, which is obscene when you consider they’re doing no work whatsoever on behalf of the user to market those photos.
  • Ability to set a custom domain to our Flickr photostreams, if we so choose. WordPress and Blogger already let people do this, so it’s clearly something that’s feasible to do for millions of users — the path has already been paved.
  • Smart sets, that update automatically based on criteria we set. This would eliminate the need to use third-party tools and websites, which are nice, but let’s face it, things would be even nicer if we could do it directly on Flickr.
  • Grey and black backgrounds for our photostreams, in addition to the standard white color.

You can find me here on Flickr.

Standard
Thoughts

Where’s the SmugMug Publish Service for Lightroom?

I love the Flickr Publish Service in Lightroom 3, and would love to see SmugMug make their own.

The only thing missing for the Flickr service is for it to know which photos I’ve exported and uploaded to Flickr before the service became available, in previous versions of Lightroom. I for example have either tagged the photos uploaded to Flickr with, obviously enough, “Flickr”, or have added them to a Flickr collection in Lightroom, so I could easily find them.

Here’s where SmugMug has the chance to shine! I’d love to be able to publish my photos to SmugMug directly from Lightroom, using the Publish Services functionality, so I could always sync up any photos that I’ve re-developed or where I’ve updated the metadata. But for this service to really stand out, it needs to know which photos I’ve already uploaded.

You can see where this is going, right? I’ve already tagged all my SmugMug photos, and have already placed them in collection sets and collections that match my SmugMug categories breakdown exactly. With a little bit of computing power and some smart algorithms, the folks at SmugMug could put together a killer Publish Service for Lightroom that incorporates all the Flickr functionality and bests it by matching my already-uploaded photos.

What about the cost? The Flickr Publish Service is free to use for all Flickr users, but you cannot re-publish uploaded photos if you’ve changed them in Lightroom. (You can, but if you’re not a Pro, it’ll wipe out any comments and faves on the photo, so it’s not advisable.)

SmugMug could use a similar approach. Their Publish Service could be free for basic SmugMug users, with limited functionality, and it could offer full functionality to Power and Pro users. (I myself have the Pro membership.) I’d even be willing to pay a one-time fee to download and install the service, because I think the functionality would be amazing.

Standard
Reviews

Flickr launches People in Photos

Flickr launched a new feature they call People in Photos a few hours ago, on October 21, 2009. It lets you tag people in your photos or in your contacts’ photos. I guess it was only a matter of time before this happened. While Riya and iPhoto went the route of computer-aided facial recognition, which is a pretty cool feature indeed but processor-intensive, Facebook and now Flickr have gone the more low cost route of letting members manually tag people in their photos.

At any rate, the process is easy and real-time. You start typing in some identifier for a person you want to identify in a photo, such as a name or screen name or email address, the database of members is searched live, and you’re presented with a drop-down list of people that narrows down with each letter you type. Pretty cool. Flickr also went the extra mile and included the ability to let you determine who can add you to photos, and who can add people to your photos. Very nice touch there.

I added my wife and myself to a couple of photos where we appear, and took the following screenshots to show you what the new feature looks like. The only reason I noticed it is because I logged into my Recent Activity page a few minutes ago and saw a small change in the options, as you can see below.

flickr-people-in-photos-1

The option to add people to a photo is located in the sidebar, below the photostream and groups thumbnails and above the tags.

flickr-people-in-photos-2

As soon as I got done tagging my wife and I in the photos, I got an email from Flickr where they explained the new feature to me and allowed me to set the privacy options I mentioned above.

flickr-people-in-photos-3

Like I said, pretty cool implementation, user-friendly, too, and it was something that was bound to happen sooner rather than later. There’s also a post on Flickr’s official blog announcing the feature launch.

What I’d like to know now is if Flickr can read the iPhoto person tags and somehow match them up with Flickr members, so that photos uploaded to Flickr from iPhoto get people-tagged automatically. Or is that the next step down the road?

Standard