Cannot access drive

How I recovered from catastrophic data loss

In late 2012 and right after New Year’s Eve this year, in 2015, I experienced two data loss events, both of which happened on my Drobo storage devices. I’ll write a separate post detailing my experiences in recent years with my Drobos but for now, I wanted to let you know how I recovered my files.

First, what do I mean by “catastrophic data loss”? Simple: the loss of terabytes of my very important data: photos, videos, documents. Among other things, I am a photographer and a filmmaker. Losing my photos and my videos is a catastrophic event, as my libraries and archives include both personal and professional photos and videos. If I were to lose these things, I’d lose both treasured memories and part of my livelihood.

Here I should also point out that all of us are at risk of data loss. Most of our stuff is digital these days (or going that way). What would you do if you’d lose all your photos and videos? Think about that question and put a plan of action in place. Follow through with it and make sure you’re covered.

Now let me get the bad part out of the way: in 2012, I lost somewhere between 25,000 – 30,000 photos and I still haven’t counted how many videos, but it was a lot, probably about 20% of my video library. This is stuff I’ll never get back. It’s gone. Period. Who’s to blame? The Drobo. More on that in a later post.

Earlier this year, I could have lost an untold number of files once more but I didn’t. Why? Partially because the Drobo has improved in the way it’s handling errors but mostly because I had access to good software.

Here are the three methods of data recovery I’ll talk about below:

  • Data Rescue: it’s a piece of software that lets you mount bad drives and get your files backed up somewhere else. This let me copy all of the files it could read off the Drobo, although a lot of them ended up being corrupted, as detailed above.
  • Jeffrey Friedl’s Preview-Cache Image Extraction: this is a Lightroom plugin that allowed me to extract image previews for the lost images directly from my Lightroom catalogs. It’s a niche plugin but it’s super useful. You don’t realize just how good it is until you have to use it and then you thank the heavens that it exists.
  • Flickr and YouTube: I was able to download images and videos I’d published to Flickr and YouTube at their maximum upload resolution. They may not have been my digital negatives or my raw video files, which were lost forever, but at least I had something left. This is why I’ve started to upload to both Flickr and YouTube at the best resolution and quality possible, in case something like this happens again.

If you’re pressed for time, feel free to stop here. Make sure you use the methods outlined above and you’ll fare much better if you should lose your data, particularly if you’re working in visual media like I am. If you want the details, read on.

Data Rescue

Back in 2012, I was able to mount the corrupted Drobo volume using Data Rescue 3 and recover the bulk of my files. As mentioned above, Data Rescue was able to see all of the files, including the corrupted ones and it let me copy them off, but 25,000 – 30,000 of a total of about 130,000 photographs and I don’t know how many videos were corrupted and couldn’t be read by either Lightroom, Photoshop, Final Cut Pro or Quicktime, so they were of no use to me. They were gone. These were original RAW, DNG, TIF and JPG files from my cameras and SD and HD video files (MP4, MOV and AVI) from my video cameras. I also lost a great deal of family videos and films and cartoons I’d painstakingly digitized from VHS tapes and DVDs I’d purchased, as well as shows and films I’d recorded from TV using a DVR and then edited and stored on the Drobo. In most cases, the files just wouldn’t open up at all. In other cases, I could open them but half or more than half of the image was gone, as you see below.

This is one of my wedding photographs. Most of my wedding photographs look like this or worse…

At our weddingHere’s another. This used to be a photograph of a cliff.

Cheile Turului

I could give you many more examples but the point is, they were irreversibly damaged when the Drobo decided to go kaput.

I don’t know what I would have done without Data Rescue. Because I bought it and used it, I was able to save 70-80% of my data after my 1st catastrophic data loss event and 100% of my data during my recent data loss event.

You may say it’s not data loss and it’s not catastrophic if I was able to recover the data. To that I say that I’d have recovered 0% of my data in both cases without Data Rescue and 0% of over 8 TB of data is damned catastrophic in my book.

Jeffrey Friedl’s Preview-Cache Image Extraction

This super-useful and little-known plugin for Lightroom allows you to extract JPG files from the preview images stored in your Lightroom catalogs. That means that even if you lose the original raw files, you can still have the JPGs and that’s a huge thing.

There’s one caveat though: you need to have allowed Lightroom to keep the previews and you also need to have allowed Lightroom to store high-quality previews. I won’t get into the exact terminology here, there are plenty of tutorials on the internet that will teach you how to optimize those settings. Suffice it to say that I now have my catalogs set to create 1:1 previews and to never delete them, just in case I ever experience data loss again.

I didn’t do this in the past, which meant that I was only able to recover thumbnails or smaller JPGs for most of my corrupted photos, but this was still better than nothing. I have precious photos of my wife that are thumbnail-sized, but at least I have those, I was able to get something back from the gaping maw of data loss.

Flickr and YouTube

These two websites aren’t just for sharing photos and videos. They also let you download your originals. Well, Flickr lets you download your originals. YouTube only lets you download MP4 files of your videos but hey, it’s wonderful anyway.

By the way, the Flickr mobile app and the Google Plus mobile app (for iOS and Android) both let you automatically back up the photos taken with that phone to your respective accounts on both services. They’re set to private by default so only you see them. That’s really nice.

Flickr download options

YouTube download options

This is why I now upload all my published photos to Flickr at their highest resolution and quality and why I also upload all my published videos to YouTube at their highest resolution and quality. In case I ever experience data loss in the future, I’ll have part of my photo and video library on these sites and I’ll be able to download it. And this is also why I no longer put watermarks on my photos. It’s no good to be able to download your own original and have a watermark on it. You now either have to crop it or Photoshop it. I have no time for that sort of thing. I’d rather deal with more productive stuff.

Of course, JPGs aren’t DNGs or RAW files but if they’re the highest resolution, dpi and quality available, they’ll do just fine. And an edited 1080p MP4 file isn’t the same thing as the original Final Cut Pro event and project along with the original video and audio files that were used to create it, but if you don’t have those anymore, you’ll be very thankful to have the MP4.

Now, for some less-than-obvious stuff…

But Raoul, why don’t you back up your stuff? That would solve all your problems! 

I do back up my stuff. I’ve been using CrashPlan for years and I also use Time Machine to back up the files on my Mac (but not all my files are on my Mac, they don’t all fit on it). Unfortunately, during my first data loss event in 2012, I was re-structuring my backup sets and the Drobo couldn’t have picked a worst time to fail. If I had relied on my backups, I’d have recovered only about 25% of my data.

This year, I was doing a little better, although I was also re-structuring my backup sets. Somehow these things seem to know when to fail just to cause more headaches (my warranty had also just run out about 3-4 days before it failed). That brought to mind images of planned obsolescence…

This time I’d have recovered about 80% of my data from the backups. Not ideal but much better than before. I can go into my backup strategy at a later time, but it’s much more difficult for me to back up all my stuff than it is for you, simply because I have a ton of data and I always run into bandwidth issues. For example, one of my backup jobs has to keep up with 8.1 TB of data. The other, with 6 TB of data. And I don’t add small amounts of data to those backup sets, I add gigabytes, lots of gigabytes, whenever I have a studio shoot or take a trip, whether it be photos or videos.

But Raoul, why do you keep using the Drobo when it keeps failing? 

The basic premise of a Drobo, that of using SATA drives of different sizes, from different manufacturers in a single array that can show up as a single 16 TB volume on my Mac, and also allow for one (or two) of those drives to fail while keeping the data safe, still cannot be beaten by anything else on the market. If you know of anything else that meets those criteria, let me know. The Drobo has its drawbacks and data corruption is one of them. Drobos also brick themselves quite a lot, just search for that phrase and you’ll see what I mean. They’re not to be relied upon but they provide the basic benefit outlined above.

But Raoul, you could have used photo recovery software to get all those tens of thousands of photos back! Why didn’t you? 

Back in 2012, I knew of no such software. Now I believe there are several options available and some allow for batch processing of corrupted photos. I haven’t tried any of them yet so I can’t tell you anthing about them. I doubt that any software can do much when half of a photo’s pixels are missing. Besides, I didn’t need to use them after my latest data loss, I was able to get it all back with Data Rescue.

But Raoul, you could have sent your Drobo in to a professional data recovery service. Couldn’t they have done a much better job? 

Maybe. I did get a couple of quotes. They ran anywhere from $3,000 – over $10,000 and they couldn’t guarantee they’d get all my data back. What also made things more complicated and expensive was shipping my drives to the US, where these companies were located. I live abroad and the customs are such a headache I try to avoid dealing with them whenever I can.

Four wishes for Lightroom

It’s 2011, a new year, and it’s likely that Adobe will put out a new version of Lightroom this year. With that in mind, it would be wonderful if the Lightroom team could implement the following features in the next minor or major version of LR:

  • Find and Replace within metadata (details here)
  • Faster navigation and rendering when working with large catalogs (details here)
  • Filter catalog for metadata conflicts (details here)
  • More accurate time of capture for movie files captured with an iPhone 4 or a Nokia N95, or other video camera that doesn’t supply sidecar THM files (detailed explanation here and bug report here)

Thanks!

File corruption rears its ugly head

During the last few weeks, I’ve seen the following error in Adobe Lightroom 3.

There I am, editing photos, minding my own business, and when I try to view a photo for editing, I get the error message you see above: “The file appears to be unsupported or damaged.”

When I try to view the file in the finder, I get the same error message, this time directly from the OS: “The file […] could not be opened. It may be damaged…”

Here’s my workflow:

  1. Shoot RAW
  2. Import RAW files as DNG into Lightroom (LR converts them on the fly)
  3. Process in LR
  4. Export as JPG or as needed
  5. Back up the catalog and check its integrity weekly

It’s pretty straightforward, and that’s the way I like it. I currently have about 85,000 photos in my LR library, whose catalog is stored locally on my MBP, with the files (DNG, RAW, JPG and TIF) residing on a Firewire Drobo.

Fortunately, the file corruption is only temporary, meaning there’s an error somewhere along the way:

  • It could be Lightroom
  • It could be the DNG file format (because I haven’t gotten the error with RAW or JPG files)
  • It could be OS X: I wonder how much testing Apple did for 10.6.5 with volumes greater than 4TB
  • It could be the Drobo: it’s a big volume (4.4TB) with lots of data that’s constantly being updated, lots of I/O traffic

I don’t know, and I hope someone reading this has an answer.

What fixes the error every time is quitting LR, ejecting the Drobo, cycling its power, mounting it, and starting LR. I’ve also tried just restarting LR, or just ejecting the Drobo, but those methods didn’t work.

Time of capture metadata bug in iPhone 4 movie clips

Updated 9/12/10: I’m not sure any more if this is an iPhone 4 glitch or an Adobe Lightroom 3.2 bug. A thread has been opened in the Adobe Lightroom Forum, if you’d like to chime in there.

After upgrading the iPhone with iOS 4.1, I recorded a new video clip, imported it and some new photos into Lightroom, and the same wrong date and time appear for it.

According to a comment on my thread in the Apple Support Forums, the correct time of capture is displayed for iPhone 4 video clips elsewhere but Lightroom. And I also noticed that Lightroom displays the very same incorrect date and time of capture for video clips taken with the Nokia N95.

Updated 9/27/10: I’ve been in touch with Adobe, and it turns out this is a “designed” behavior. That is, because movie clips do not have EXIF data (there is no standard for EXIF data when it comes to them), they are assigned a random date and time as they’re imported into Lightroom. HDSLR video files are accompanied by a .THM file which stores the necessary EXIF data, and that’s why they show up properly.

Quoting from Davide M.’s (Adobe) response:

So I then had a look at our bug database and it turns our this is a known issue with mobile phones although somewhat out-with our control. Movie files do not technically have EXIF data or at least the standard has not yet been established. Since the import process can assign a timestamp to a movie file, we ignore this time stamp since it can be inaccurate, as shown by the example of your video file being changed by the simple process of email. Other applications while appearing to work fine, in fact are simply showing you the files creation date. If you were to duplicate the file, you will see that the timestamp in these other applications will change to the time the file duplication took place.

The reason why most DSLRs work is because they create a sidecar file containing that information. Files with no timestamp, such as the ones from the iPhone and the Nokia N95 do not create this and hence default to 1/1/04 when looking at the Loupe information overlay.

In the example you used, the Canon 7D creates a .THM sidecar file with the same name as the video file it generates. This contains all the data associated with the video file.

Still, this is problematic behavior, as it introduces erroneous times of capture in these movie files. So I asked Davide if it would be possible for Lightroom to be updated so that it writes a more accurate time of capture for these movie files. Thankfully, he agreed to log it as a feature request. Time will tell if this will make it into a future LR update. Quoting him below:

That’s certainly something I can log in our feature request list. Because this has been deemed ‘as designed’ by our engineering team (due to the lack of EXIF data in movie files) it is not technically a bug. None the less, I can see that this would be a useful addition to our application. Thanks for bringing this to our attention.

Thank you, Davide!

After downloading a few movie clips taken with an iPhone 4 (running iOS 4.0.1) onto my computer, I saw right away that their time of capture was incorrect, even though the iPhone’s time had been set up correctly. I took a few screenshots of the movie clips in Lightroom, which you can see below. Click on each to view them large.

This time metadata error happens when using either the main (back-facing) HD video camera, as shown above, or the front-facing VGA camera, as you can see from the screenshot below.

It looks like iPhone 4 records the same time for all video clips recorded with it, set at 1/1/04 1:44:24 AM.

It goes without saying that any digital video camera worth its salt will record the time of capture properly. The question, naturally, is when Apple will fix this glaring bug?

For comparison purposes, here is a screenshot of a Canon 7D movie clip, also shown in Lightroom. The time of capture was recorded properly, as was to be expected.

SmugMug, are you listening?

I’m disappointed with SmugMug over their continued lack of support for proper export and maintenance of photographs directly from Lightroom. Back in July, I wrote about the Flickr Publish Service in Lightroom, and wondered when SmugMug would introduce their own.

What I was really looking for (and I said this in the post) was a way for the publish service to identify what I’ve already uploaded and allow me to re-publish those photos where I’ve made changes to the metadata or to the processing. The official Flickr Publish Service didn’t offer that option.

A few of my readers (Gary, Chris, Russell, thanks!) pointed me to Jeffrey Friedl’s excellent plugins for Lightroom, and I’ve been using them ever since. As a matter of fact, I’ve switched over to them completely. I use them for all four web services where I currently publish photos (SmugMug, Flickr, Facebook and PicasaWeb). I don’t know what I’d do without them. Wait, I do know — I know for sure I’d be doing a LOT more work and spending a LOT more time uploading and maintaining my online collections.

With Jeffrey’s LR plugins, I was able to identify about 90% of the photos already uploaded to SmugMug, and about 75% of the photos already uploaded to Flickr. In the case of Flickr, I then did manual updates and re-identifies so I could get it to know 95% of the photos already uploaded. This means Lightroom now allows me to quickly identify, update and replace almost any photos I’ve got at SmugMug, Flickr, Facebook and PicasaWeb. This is huge.

There is a catch, though, and it’s a BIG one. I keep running into the same “Wrong Format ()” error with SmugMug, which means I still haven’t been able to straighten out the photos I’ve uploaded to them. Here are a couple of screenshots of the error messages I get. It starts with a “TimedOut” error, then I get the “Wrong Format ()” error, then the upload process aborts.

I get these errors almost every time I try to re-publish an updated photo, but I don’t get them as often when I try to upload new photos. To give you an idea of how bad things are, I’ve currently got 109 photos to update in one of my galleries at SmugMug, and last night, I had about 167 photos. I’ve had to restart the re-publish process about 30-40 times since last night. You do the math, but I think it works out to 1-2 photos per error. This sucks. I should be able to just click the Publish button and walk away, knowing all of my changes will propagate correctly.

I’ve contacted Jeffrey, and I’ve contacted SmugMug. I’ve had extensive email conversations with each. SmugMug alternates in their replies. They’ve said the following to me:

  • It’s a fault with the plugin
  • It’s something on their end but they’re working on it
  • There’s nothing they can do about it
  • I should use something else to upload photos
  • They blamed my setup, which we ruled out after some internet connectivity tests

Jeffrey says there’s nothing he can do about it, and I believe him more than I believe SmugMug. Want to know why? Because his other plugins work just fine. I’m able to re-publish updated photos to Flickr and Facebook and PicasaWeb without any problems. Only SmugMug somehow can’t handle my uploads.

I’ve tried reloading the plugin, installing it anew, removing and re-adding the publish service, upgrading the plugin, but nothing. I still get the same errors.

My question for the smug folks at SmugMug is this: how is it possible that Facebook and Flickr and PicasaWeb have worked out the re-publish issues, but you haven’t? What’s taking you so long? Why can’t you work out the same problem on your end?

I was hoping that with the release of Lightroom 3.2, and the release of the official SmugMug Publish Service for LR (hat tip to David Parry for the advance notice), that SmugMug would work out the kinks in their API, but it looks like they still haven’t done it. I tried their plugin, but of course they took the easy route, like Flickr, and haven’t introduced any functionality that would identify photos already uploaded to their service. Only Jeffrey Friedl’s plugins offer this feature.

This leaves me terribly disappointed. As a SmugMug Pro, I don’t want to bother with error messages. I don’t want to bother with posts like this. I’d rather post photographs and update my SmugMug galleries in peace, but I can’t.

If you’re having the same problems with SmugMug, please, write to them, and ask them when they’re going to get their act together. This problem’s existed for several months. How much more time will it take until they deal with it?

Where’s the SmugMug Publish Service for Lightroom?

I love the Flickr Publish Service in Lightroom 3, and would love to see SmugMug make their own.

The only thing missing for the Flickr service is for it to know which photos I’ve exported and uploaded to Flickr before the service became available, in previous versions of Lightroom. I for example have either tagged the photos uploaded to Flickr with, obviously enough, “Flickr”, or have added them to a Flickr collection in Lightroom, so I could easily find them.

Here’s where SmugMug has the chance to shine! I’d love to be able to publish my photos to SmugMug directly from Lightroom, using the Publish Services functionality, so I could always sync up any photos that I’ve re-developed or where I’ve updated the metadata. But for this service to really stand out, it needs to know which photos I’ve already uploaded.

You can see where this is going, right? I’ve already tagged all my SmugMug photos, and have already placed them in collection sets and collections that match my SmugMug categories breakdown exactly. With a little bit of computing power and some smart algorithms, the folks at SmugMug could put together a killer Publish Service for Lightroom that incorporates all the Flickr functionality and bests it by matching my already-uploaded photos.

What about the cost? The Flickr Publish Service is free to use for all Flickr users, but you cannot re-publish uploaded photos if you’ve changed them in Lightroom. (You can, but if you’re not a Pro, it’ll wipe out any comments and faves on the photo, so it’s not advisable.)

SmugMug could use a similar approach. Their Publish Service could be free for basic SmugMug users, with limited functionality, and it could offer full functionality to Power and Pro users. (I myself have the Pro membership.) I’d even be willing to pay a one-time fee to download and install the service, because I think the functionality would be amazing.

Stop the headache – generate 1:1 previews before editing

One of my gripes with Lightroom ever since I started using it was the image blurring that took place as it generated image previews or re-rendered images while in Develop mode. (I started using LR in February 2007.)

It looks like Adobe listened, and the image preview rendering that takes place as I develop photos isn’t noticeable anymore — that, or my faster laptop has something to do with it, too. However, the very noticeable lag in generating either standard or 1:1 previews still occurs as I browse through images in my catalog, and that can’t be helped even by my zippy MacBook Pro. As you move through images, Lightroom will blur them until it generates a standard preview, then blur them some more you zoom in, until it generates the 1:1 preview.

Fortunately, there’s a solution for it. I’m not sure if this existed from the start, or if it was introduced in later versions of the software, but you can choose to generate 1:1 previews for a set of photos before you begin working with them. (I found this out thanks to this article from Steve Paxton at O’Reilly.) Just select the images you plan to work with on a given day, and go to Library >> Previews >> Render 1:1 Previews.

lightroom-generate-previews

If you choose that option, Lightroom will also render Standard-Sized Previews in addition to the full ones, allowing you to work with the photos right away, in standard or loupe view, with no lag or blurring (well, your computer’s specs might also have something to do with it). Still, if you’re sorting through a large set of images (hundreds or thousands), this pays off handsomely, in ways that you cannot even appreciate until you start popping aspirins to deal with the tension headache caused by all that screen blurring you could have avoided if you planned ahead.

If you’ve been a long-time subscriber to my site, then you may know about an article of mine written in January ’08, entitled “The next stage for Lightroom“. In it, I described the need for Lightroom to:

  1. Allow the storage of photos from its catalog on multiple volumes
  2. Allow people to work with photos from the catalog even when external volumes were disconnected
  3. Allow for the storage of the previews database (which can be very sizable) on an external volume, or for its splitting into two parts

Guess what? Most readers just couldn’t get what I was saying. But Adobe listened. Points one and two have already been implemented in later versions. Now I can store my photos on multiple volumes, and I can work with their meta-data even when I can’t access the image files because I’ve disconnected the external drives. As for the the storage of the previews, that’s now easily solved, too.

Because I’m now storing my Lightroom catalog on my laptop, where hard drive space is an issue, this means I have to limit the size of the previews database. I do this by giving it very little play. I tell LR to generate medium-quality 1024 px standard previews, and to delete 1:1 previews after a day. The previews database is no joking matter. Just a short while back, LR was set to discard 1:1 previews after a month, and my previews database had ballooned to over 60 GB!

lightroom-previews-settings

So, even though I allow myself the luxury of generating 1:1 previews for hundreds of photos, as you’ve seen above, the size of my previews folder stays manageable, and the free space on my hard drive stays where it needs to stay, because Lightroom cleans up after itself. Instead of worrying about free space, I allow my MacBook Pro to flex its processing muscle for 15-30 minutes before I dig into a large set of photos from a particular location, and then I can work undisturbed and headache-free for that day.