How I recovered from catastrophic data loss

In late 2012 and right after New Year’s Eve this year, in 2015, I experienced two data loss events, both of which happened on my Drobo storage devices. I’ll write a separate post detailing my experiences in recent years with my Drobos but for now, I wanted to let you know how I recovered my files.

First, what do I mean by “catastrophic data loss”? Simple: the loss of terabytes of my very important data: photos, videos, documents. Among other things, I am a photographer and a filmmaker. Losing my photos and my videos is a catastrophic event, as my libraries and archives include both personal and professional photos and videos. If I were to lose these things, I’d lose both treasured memories and part of my livelihood.

Here I should also point out that all of us are at risk of data loss. Most of our stuff is digital these days (or going that way). What would you do if you’d lose all your photos and videos? Think about that question and put a plan of action in place. Follow through with it and make sure you’re covered.

Now let me get the bad part out of the way: in 2012, I lost somewhere between 25,000 – 30,000 photos and I still haven’t counted how many videos, but it was a lot, probably about 20% of my video library. This is stuff I’ll never get back. It’s gone. Period. Who’s to blame? The Drobo. More on that in a later post.

Earlier this year, I could have lost an untold number of files once more but I didn’t. Why? Partially because the Drobo has improved in the way it’s handling errors but mostly because I had access to good software.

Here are the three methods of data recovery I’ll talk about below:

  • Data Rescue: it’s a piece of software that lets you mount bad drives and get your files backed up somewhere else. This let me copy all of the files it could read off the Drobo, although a lot of them ended up being corrupted, as detailed above.
  • Jeffrey Friedl’s Preview-Cache Image Extraction: this is a Lightroom plugin that allowed me to extract image previews for the lost images directly from my Lightroom catalogs. It’s a niche plugin but it’s super useful. You don’t realize just how good it is until you have to use it and then you thank the heavens that it exists.
  • Flickr and YouTube: I was able to download images and videos I’d published to Flickr and YouTube at their maximum upload resolution. They may not have been my digital negatives or my raw video files, which were lost forever, but at least I had something left. This is why I’ve started to upload to both Flickr and YouTube at the best resolution and quality possible, in case something like this happens again.

If you’re pressed for time, feel free to stop here. Make sure you use the methods outlined above and you’ll fare much better if you should lose your data, particularly if you’re working in visual media like I am. If you want the details, read on.

Data Rescue

Back in 2012, I was able to mount the corrupted Drobo volume using Data Rescue 3 and recover the bulk of my files. As mentioned above, Data Rescue was able to see all of the files, including the corrupted ones and it let me copy them off, but 25,000 – 30,000 of a total of about 130,000 photographs and I don’t know how many videos were corrupted and couldn’t be read by either Lightroom, Photoshop, Final Cut Pro or Quicktime, so they were of no use to me. They were gone. These were original RAW, DNG, TIF and JPG files from my cameras and SD and HD video files (MP4, MOV and AVI) from my video cameras. I also lost a great deal of family videos and films and cartoons I’d painstakingly digitized from VHS tapes and DVDs I’d purchased, as well as shows and films I’d recorded from TV using a DVR and then edited and stored on the Drobo. In most cases, the files just wouldn’t open up at all. In other cases, I could open them but half or more than half of the image was gone, as you see below.

This is one of my wedding photographs. Most of my wedding photographs look like this or worse…

At our weddingHere’s another. This used to be a photograph of a cliff.

Cheile Turului

I could give you many more examples but the point is, they were irreversibly damaged when the Drobo decided to go kaput.

I don’t know what I would have done without Data Rescue. Because I bought it and used it, I was able to save 70-80% of my data after my 1st catastrophic data loss event and 100% of my data during my recent data loss event.

You may say it’s not data loss and it’s not catastrophic if I was able to recover the data. To that I say that I’d have recovered 0% of my data in both cases without Data Rescue and 0% of over 8 TB of data is damned catastrophic in my book.

Jeffrey Friedl’s Preview-Cache Image Extraction

This super-useful and little-known plugin for Lightroom allows you to extract JPG files from the preview images stored in your Lightroom catalogs. That means that even if you lose the original raw files, you can still have the JPGs and that’s a huge thing.

There’s one caveat though: you need to have allowed Lightroom to keep the previews and you also need to have allowed Lightroom to store high-quality previews. I won’t get into the exact terminology here, there are plenty of tutorials on the internet that will teach you how to optimize those settings. Suffice it to say that I now have my catalogs set to create 1:1 previews and to never delete them, just in case I ever experience data loss again.

I didn’t do this in the past, which meant that I was only able to recover thumbnails or smaller JPGs for most of my corrupted photos, but this was still better than nothing. I have precious photos of my wife that are thumbnail-sized, but at least I have those, I was able to get something back from the gaping maw of data loss.

Flickr and YouTube

These two websites aren’t just for sharing photos and videos. They also let you download your originals. Well, Flickr lets you download your originals. YouTube only lets you download MP4 files of your videos but hey, it’s wonderful anyway.

By the way, the Flickr mobile app and the Google Plus mobile app (for iOS and Android) both let you automatically back up the photos taken with that phone to your respective accounts on both services. They’re set to private by default so only you see them. That’s really nice.

Flickr download options

YouTube download options

This is why I now upload all my published photos to Flickr at their highest resolution and quality and why I also upload all my published videos to YouTube at their highest resolution and quality. In case I ever experience data loss in the future, I’ll have part of my photo and video library on these sites and I’ll be able to download it. And this is also why I no longer put watermarks on my photos. It’s no good to be able to download your own original and have a watermark on it. You now either have to crop it or Photoshop it. I have no time for that sort of thing. I’d rather deal with more productive stuff.

Of course, JPGs aren’t DNGs or RAW files but if they’re the highest resolution, dpi and quality available, they’ll do just fine. And an edited 1080p MP4 file isn’t the same thing as the original Final Cut Pro event and project along with the original video and audio files that were used to create it, but if you don’t have those anymore, you’ll be very thankful to have the MP4.

Now, for some less-than-obvious stuff…

But Raoul, why don’t you back up your stuff? That would solve all your problems! 

I do back up my stuff. I’ve been using CrashPlan for years and I also use Time Machine to back up the files on my Mac (but not all my files are on my Mac, they don’t all fit on it). Unfortunately, during my first data loss event in 2012, I was re-structuring my backup sets and the Drobo couldn’t have picked a worst time to fail. If I had relied on my backups, I’d have recovered only about 25% of my data.

This year, I was doing a little better, although I was also re-structuring my backup sets. Somehow these things seem to know when to fail just to cause more headaches (my warranty had also just run out about 3-4 days before it failed). That brought to mind images of planned obsolescence…

This time I’d have recovered about 80% of my data from the backups. Not ideal but much better than before. I can go into my backup strategy at a later time, but it’s much more difficult for me to back up all my stuff than it is for you, simply because I have a ton of data and I always run into bandwidth issues. For example, one of my backup jobs has to keep up with 8.1 TB of data. The other, with 6 TB of data. And I don’t add small amounts of data to those backup sets, I add gigabytes, lots of gigabytes, whenever I have a studio shoot or take a trip, whether it be photos or videos.

But Raoul, why do you keep using the Drobo when it keeps failing? 

The basic premise of a Drobo, that of using SATA drives of different sizes, from different manufacturers in a single array that can show up as a single 16 TB volume on my Mac, and also allow for one (or two) of those drives to fail while keeping the data safe, still cannot be beaten by anything else on the market. If you know of anything else that meets those criteria, let me know. The Drobo has its drawbacks and data corruption is one of them. Drobos also brick themselves quite a lot, just search for that phrase and you’ll see what I mean. They’re not to be relied upon but they provide the basic benefit outlined above.

But Raoul, you could have used photo recovery software to get all those tens of thousands of photos back! Why didn’t you? 

Back in 2012, I knew of no such software. Now I believe there are several options available and some allow for batch processing of corrupted photos. I haven’t tried any of them yet so I can’t tell you anthing about them. I doubt that any software can do much when half of a photo’s pixels are missing. Besides, I didn’t need to use them after my latest data loss, I was able to get it all back with Data Rescue.

But Raoul, you could have sent your Drobo in to a professional data recovery service. Couldn’t they have done a much better job? 

Maybe. I did get a couple of quotes. They ran anywhere from $3,000 – over $10,000 and they couldn’t guarantee they’d get all my data back. What also made things more complicated and expensive was shipping my drives to the US, where these companies were located. I live abroad and the customs are such a headache I try to avoid dealing with them whenever I can.

Four wishes for Lightroom

It’s 2011, a new year, and it’s likely that Adobe will put out a new version of Lightroom this year. With that in mind, it would be wonderful if the Lightroom team could implement the following features in the next minor or major version of LR:

  • Find and Replace within metadata (details here)
  • Faster navigation and rendering when working with large catalogs (details here)
  • Filter catalog for metadata conflicts (details here)
  • More accurate time of capture for movie files captured with an iPhone 4 or a Nokia N95, or other video camera that doesn’t supply sidecar THM files (detailed explanation here and bug report here)

Thanks!

File corruption rears its ugly head

During the last few weeks, I’ve seen the following error in Adobe Lightroom 3.

There I am, editing photos, minding my own business, and when I try to view a photo for editing, I get the error message you see above: “The file appears to be unsupported or damaged.”

When I try to view the file in the finder, I get the same error message, this time directly from the OS: “The file […] could not be opened. It may be damaged…”

Here’s my workflow:

  1. Shoot RAW
  2. Import RAW files as DNG into Lightroom (LR converts them on the fly)
  3. Process in LR
  4. Export as JPG or as needed
  5. Back up the catalog and check its integrity weekly

It’s pretty straightforward, and that’s the way I like it. I currently have about 85,000 photos in my LR library, whose catalog is stored locally on my MBP, with the files (DNG, RAW, JPG and TIF) residing on a Firewire Drobo.

Fortunately, the file corruption is only temporary, meaning there’s an error somewhere along the way:

  • It could be Lightroom
  • It could be the DNG file format (because I haven’t gotten the error with RAW or JPG files)
  • It could be OS X: I wonder how much testing Apple did for 10.6.5 with volumes greater than 4TB
  • It could be the Drobo: it’s a big volume (4.4TB) with lots of data that’s constantly being updated, lots of I/O traffic

I don’t know, and I hope someone reading this has an answer.

What fixes the error every time is quitting LR, ejecting the Drobo, cycling its power, mounting it, and starting LR. I’ve also tried just restarting LR, or just ejecting the Drobo, but those methods didn’t work.

Time of capture metadata bug in iPhone 4 movie clips

Updated 9/12/10: I’m not sure any more if this is an iPhone 4 glitch or an Adobe Lightroom 3.2 bug. A thread has been opened in the Adobe Lightroom Forum, if you’d like to chime in there.

After upgrading the iPhone with iOS 4.1, I recorded a new video clip, imported it and some new photos into Lightroom, and the same wrong date and time appear for it.

According to a comment on my thread in the Apple Support Forums, the correct time of capture is displayed for iPhone 4 video clips elsewhere but Lightroom. And I also noticed that Lightroom displays the very same incorrect date and time of capture for video clips taken with the Nokia N95.

Updated 9/27/10: I’ve been in touch with Adobe, and it turns out this is a “designed” behavior. That is, because movie clips do not have EXIF data (there is no standard for EXIF data when it comes to them), they are assigned a random date and time as they’re imported into Lightroom. HDSLR video files are accompanied by a .THM file which stores the necessary EXIF data, and that’s why they show up properly.

Quoting from Davide M.’s (Adobe) response:

So I then had a look at our bug database and it turns our this is a known issue with mobile phones although somewhat out-with our control. Movie files do not technically have EXIF data or at least the standard has not yet been established. Since the import process can assign a timestamp to a movie file, we ignore this time stamp since it can be inaccurate, as shown by the example of your video file being changed by the simple process of email. Other applications while appearing to work fine, in fact are simply showing you the files creation date. If you were to duplicate the file, you will see that the timestamp in these other applications will change to the time the file duplication took place.

The reason why most DSLRs work is because they create a sidecar file containing that information. Files with no timestamp, such as the ones from the iPhone and the Nokia N95 do not create this and hence default to 1/1/04 when looking at the Loupe information overlay.

In the example you used, the Canon 7D creates a .THM sidecar file with the same name as the video file it generates. This contains all the data associated with the video file.

Still, this is problematic behavior, as it introduces erroneous times of capture in these movie files. So I asked Davide if it would be possible for Lightroom to be updated so that it writes a more accurate time of capture for these movie files. Thankfully, he agreed to log it as a feature request. Time will tell if this will make it into a future LR update. Quoting him below:

That’s certainly something I can log in our feature request list. Because this has been deemed ‘as designed’ by our engineering team (due to the lack of EXIF data in movie files) it is not technically a bug. None the less, I can see that this would be a useful addition to our application. Thanks for bringing this to our attention.

Thank you, Davide!

After downloading a few movie clips taken with an iPhone 4 (running iOS 4.0.1) onto my computer, I saw right away that their time of capture was incorrect, even though the iPhone’s time had been set up correctly. I took a few screenshots of the movie clips in Lightroom, which you can see below. Click on each to view them large.

This time metadata error happens when using either the main (back-facing) HD video camera, as shown above, or the front-facing VGA camera, as you can see from the screenshot below.

It looks like iPhone 4 records the same time for all video clips recorded with it, set at 1/1/04 1:44:24 AM.

It goes without saying that any digital video camera worth its salt will record the time of capture properly. The question, naturally, is when Apple will fix this glaring bug?

For comparison purposes, here is a screenshot of a Canon 7D movie clip, also shown in Lightroom. The time of capture was recorded properly, as was to be expected.

SmugMug, are you listening?

I’m disappointed with SmugMug over their continued lack of support for proper export and maintenance of photographs directly from Lightroom. Back in July, I wrote about the Flickr Publish Service in Lightroom, and wondered when SmugMug would introduce their own.

What I was really looking for (and I said this in the post) was a way for the publish service to identify what I’ve already uploaded and allow me to re-publish those photos where I’ve made changes to the metadata or to the processing. The official Flickr Publish Service didn’t offer that option.

A few of my readers (Gary, Chris, Russell, thanks!) pointed me to Jeffrey Friedl’s excellent plugins for Lightroom, and I’ve been using them ever since. As a matter of fact, I’ve switched over to them completely. I use them for all four web services where I currently publish photos (SmugMug, Flickr, Facebook and PicasaWeb). I don’t know what I’d do without them. Wait, I do know — I know for sure I’d be doing a LOT more work and spending a LOT more time uploading and maintaining my online collections.

With Jeffrey’s LR plugins, I was able to identify about 90% of the photos already uploaded to SmugMug, and about 75% of the photos already uploaded to Flickr. In the case of Flickr, I then did manual updates and re-identifies so I could get it to know 95% of the photos already uploaded. This means Lightroom now allows me to quickly identify, update and replace almost any photos I’ve got at SmugMug, Flickr, Facebook and PicasaWeb. This is huge.

There is a catch, though, and it’s a BIG one. I keep running into the same “Wrong Format ()” error with SmugMug, which means I still haven’t been able to straighten out the photos I’ve uploaded to them. Here are a couple of screenshots of the error messages I get. It starts with a “TimedOut” error, then I get the “Wrong Format ()” error, then the upload process aborts.

I get these errors almost every time I try to re-publish an updated photo, but I don’t get them as often when I try to upload new photos. To give you an idea of how bad things are, I’ve currently got 109 photos to update in one of my galleries at SmugMug, and last night, I had about 167 photos. I’ve had to restart the re-publish process about 30-40 times since last night. You do the math, but I think it works out to 1-2 photos per error. This sucks. I should be able to just click the Publish button and walk away, knowing all of my changes will propagate correctly.

I’ve contacted Jeffrey, and I’ve contacted SmugMug. I’ve had extensive email conversations with each. SmugMug alternates in their replies. They’ve said the following to me:

  • It’s a fault with the plugin
  • It’s something on their end but they’re working on it
  • There’s nothing they can do about it
  • I should use something else to upload photos
  • They blamed my setup, which we ruled out after some internet connectivity tests

Jeffrey says there’s nothing he can do about it, and I believe him more than I believe SmugMug. Want to know why? Because his other plugins work just fine. I’m able to re-publish updated photos to Flickr and Facebook and PicasaWeb without any problems. Only SmugMug somehow can’t handle my uploads.

I’ve tried reloading the plugin, installing it anew, removing and re-adding the publish service, upgrading the plugin, but nothing. I still get the same errors.

My question for the smug folks at SmugMug is this: how is it possible that Facebook and Flickr and PicasaWeb have worked out the re-publish issues, but you haven’t? What’s taking you so long? Why can’t you work out the same problem on your end?

I was hoping that with the release of Lightroom 3.2, and the release of the official SmugMug Publish Service for LR (hat tip to David Parry for the advance notice), that SmugMug would work out the kinks in their API, but it looks like they still haven’t done it. I tried their plugin, but of course they took the easy route, like Flickr, and haven’t introduced any functionality that would identify photos already uploaded to their service. Only Jeffrey Friedl’s plugins offer this feature.

This leaves me terribly disappointed. As a SmugMug Pro, I don’t want to bother with error messages. I don’t want to bother with posts like this. I’d rather post photographs and update my SmugMug galleries in peace, but I can’t.

If you’re having the same problems with SmugMug, please, write to them, and ask them when they’re going to get their act together. This problem’s existed for several months. How much more time will it take until they deal with it?

Where’s the SmugMug Publish Service for Lightroom?

I love the Flickr Publish Service in Lightroom 3, and would love to see SmugMug make their own.

The only thing missing for the Flickr service is for it to know which photos I’ve exported and uploaded to Flickr before the service became available, in previous versions of Lightroom. I for example have either tagged the photos uploaded to Flickr with, obviously enough, “Flickr”, or have added them to a Flickr collection in Lightroom, so I could easily find them.

Here’s where SmugMug has the chance to shine! I’d love to be able to publish my photos to SmugMug directly from Lightroom, using the Publish Services functionality, so I could always sync up any photos that I’ve re-developed or where I’ve updated the metadata. But for this service to really stand out, it needs to know which photos I’ve already uploaded.

You can see where this is going, right? I’ve already tagged all my SmugMug photos, and have already placed them in collection sets and collections that match my SmugMug categories breakdown exactly. With a little bit of computing power and some smart algorithms, the folks at SmugMug could put together a killer Publish Service for Lightroom that incorporates all the Flickr functionality and bests it by matching my already-uploaded photos.

What about the cost? The Flickr Publish Service is free to use for all Flickr users, but you cannot re-publish uploaded photos if you’ve changed them in Lightroom. (You can, but if you’re not a Pro, it’ll wipe out any comments and faves on the photo, so it’s not advisable.)

SmugMug could use a similar approach. Their Publish Service could be free for basic SmugMug users, with limited functionality, and it could offer full functionality to Power and Pro users. (I myself have the Pro membership.) I’d even be willing to pay a one-time fee to download and install the service, because I think the functionality would be amazing.

Stop the headache – generate 1:1 previews before editing

One of my gripes with Lightroom ever since I started using it was the image blurring that took place as it generated image previews or re-rendered images while in Develop mode. (I started using LR in February 2007.)

It looks like Adobe listened, and the image preview rendering that takes place as I develop photos isn’t noticeable anymore — that, or my faster laptop has something to do with it, too. However, the very noticeable lag in generating either standard or 1:1 previews still occurs as I browse through images in my catalog, and that can’t be helped even by my zippy MacBook Pro. As you move through images, Lightroom will blur them until it generates a standard preview, then blur them some more you zoom in, until it generates the 1:1 preview.

Fortunately, there’s a solution for it. I’m not sure if this existed from the start, or if it was introduced in later versions of the software, but you can choose to generate 1:1 previews for a set of photos before you begin working with them. (I found this out thanks to this article from Steve Paxton at O’Reilly.) Just select the images you plan to work with on a given day, and go to Library >> Previews >> Render 1:1 Previews.

lightroom-generate-previews

If you choose that option, Lightroom will also render Standard-Sized Previews in addition to the full ones, allowing you to work with the photos right away, in standard or loupe view, with no lag or blurring (well, your computer’s specs might also have something to do with it). Still, if you’re sorting through a large set of images (hundreds or thousands), this pays off handsomely, in ways that you cannot even appreciate until you start popping aspirins to deal with the tension headache caused by all that screen blurring you could have avoided if you planned ahead.

If you’ve been a long-time subscriber to my site, then you may know about an article of mine written in January ’08, entitled “The next stage for Lightroom“. In it, I described the need for Lightroom to:

  1. Allow the storage of photos from its catalog on multiple volumes
  2. Allow people to work with photos from the catalog even when external volumes were disconnected
  3. Allow for the storage of the previews database (which can be very sizable) on an external volume, or for its splitting into two parts

Guess what? Most readers just couldn’t get what I was saying. But Adobe listened. Points one and two have already been implemented in later versions. Now I can store my photos on multiple volumes, and I can work with their meta-data even when I can’t access the image files because I’ve disconnected the external drives. As for the the storage of the previews, that’s now easily solved, too.

Because I’m now storing my Lightroom catalog on my laptop, where hard drive space is an issue, this means I have to limit the size of the previews database. I do this by giving it very little play. I tell LR to generate medium-quality 1024 px standard previews, and to delete 1:1 previews after a day. The previews database is no joking matter. Just a short while back, LR was set to discard 1:1 previews after a month, and my previews database had ballooned to over 60 GB!

lightroom-previews-settings

So, even though I allow myself the luxury of generating 1:1 previews for hundreds of photos, as you’ve seen above, the size of my previews folder stays manageable, and the free space on my hard drive stays where it needs to stay, because Lightroom cleans up after itself. Instead of worrying about free space, I allow my MacBook Pro to flex its processing muscle for 15-30 minutes before I dig into a large set of photos from a particular location, and then I can work undisturbed and headache-free for that day.

Get the tiltshift look right from Adobe Lightroom

If you use Adobe Lightroom and want to apply a tiltshift effect to your photos, you can spend hundreds of dollars on expensive Photoshop plugins, or you can do it for free, with an Adobe AIR app called TiltShift.

If you’ve used TiltShift before, you know you can open any photo in it and apply tiltshift effects to it, but did you know you can do this right from Lightroom? Here’s how.

In Lightroom, open up the Export window and add a new Export Preset. See the screenshot below. I called mine TiltShift, so I can easily remember it. Adjust any of the settings, like color space, sizing, sharpening, etc. They don’t really matter, although it’s better to keep the image smaller so TiltShift can work faster with it.

The really important option is in the post-processing section — the very last one in the Export window. There, you’ve got to make sure you tell Lightroom to “Open [your photo] in Other Application…”, then click on the Choose button and browse to find the TiltShift app. This is pretty much it.

lightroom-tiltshift-setup

Lightroom will automatically pass your image to TiltShift, which will open it and allow you add tiltshift effects to it, to your liking. For example, I initially processed this image of a medieval water pump found on the streets of Medias, Romania, in Lightroom.

The old water fountain

Then I exported it into TiltShift using the export preset set up as described above, and adjusted the settings there to get the effect I wanted. This is how the controls and the image looked inside TiltShift.

tiltshift-screenshot

Once I did that, I saved the photo and uploaded it here. This is how the final image looks.

water-fountain-tiltshift

It couldn’t be easier, and again, let me remind you TiltShift is a free app.

[TiltShift home page] [Download TiltShift]

What's on my desk?

The idea for this post has been sitting among my drafts for a long time. I always thought it’d be a good idea to share how I work and what equipment I use, for the benefit of those of you that want to put together a similar setup. Let me make it clear that I’m not doing this to brag. I realize my equipment is fairly expensive and out of reach for some. Believe me, I’m keenly aware of its cost, and have made certain sacrifices in order to acquire it.

The hardware I have now is the best hardware I’ve had so far, and it allows me to easily develop for my clients, write for my sites, process my photographs and edit my videos and movies. See the photo enclosed below. Each important piece of equipment is marked with a number, and each number is explained in detail.

Here’s what you see above:

  1. 15″ Apple MacBook Pro: I love its dedicated 512 MB video card, and its design and speed. It allows me to focus on my work, not twiddle my thumbs while the computer plays catchup to my commands.
  2. Drobo: I have two of them. I store backups of my photos on one, and my videos on the other. Both of them are shared over my home network, so that my other Mac, a 20″ iMac G5, can access the videos and play them in Front Row.
  3. 2TB WD My Book Studio Edition II drive, running in RAID 0 (striped) mode: WD was kind enough to give me this drive as a gift of sorts, considering the significant problems I had with my 1TB WD MyBook Pro Edition II drive. I think they got tired of my complaining, and sent it to me to shut me up. Well, it worked. This drive has done great so far, and I have no complaints. I plan to write a review for it soon. I’ve been putting it through its paces so far, and it’s held up nicely. I connect it to my MBP through the Firewire 800 port, and I love the transfer speeds.
  4. 250GB WD Passport drive: I bought this last year, and it has been very useful. It’s small, stylish, and it’s powered solely by the USB port. I use it a LOT to transfer big files between computers.
  5. Turbo.264: a nifty little gadget built by the folks at Elgato, it speeds up H.264 conversions significantly. I’ve been putting this through its paces as well, and plan to write a review soon.
  6. mStand laptop stand: I love this stand, and reviewed it recently. It’s the best-designed stand for the MBP.
  7. newerTech miniStack v1: This is the first version of the popular miniStack drive enclosure. It uses PATA drives, and it has built-in FireWire and USB hubs (3 ports each). I use it as a USB hub, and it sits underneath my laptop stand — that’s why it’s not clearly visible in the photo. Both my Drobos are connected into it, and I have a spare USB port that I can use for my CF card reader, etc. If you want to have a better look at this enclosure, you can see it better in my Drobo review, where it appears in both the photos and the video.

As you can see, I aimed for simplicity in my setup. I wanted the most amount of processing power and hard disk space, in the smallest amount of physical space. You’re looking at about 5TB of storage in the photo.

Most of the heavy lifting (in terms of data crunching) happens on the WD Studio drive. I use it as a staging area for video processing, and transfer the finished, edited ones to one of the Drobos. I also use it to store and work with my Lightroom photo libraries. Working with photos in Lightroom from a Drobo is too slow for me, and the WD Studio Drive offers a visible advantage as I process each photo. I then back up my laptop and the WD Studio Drive to the Drobo through Time Machine.

You may recall I initially moved my photo libraries to one of the Drobos because I kept experiencing Lightroom library corruption and thought the WD Studio drive was at fault. As it turns out, Lightroom itself is to blame. Once your photo library reaches tens of thousands of photos (I have over 40,000 photos in one of my libraries), Lightroom will experience library corruption on a fairly regular basis. It’s just an unstable piece of software, and I hope that the next version will be better.

Just a quick note for those of you that noticed it. There’s a wire that runs behind the laptop, along the wall. The photo isn’t crooked, the wire is. I use it to hang small wires or notes to myself on it, and being loaded down with a few toward the center (which is only partially shown in the photo), it appears slanted. Sorry about that. It bothered me, but I wasn’t going to spend an hour or so in Photoshop cloning it out.

Hope this helped give you some ideas! If you have any questions, let me know.

How to transfer photos between Lightroom catalogs

This screencast will show you how to transfer photos between different Lightroom catalogs, and it will go beyond that by also demonstrating how the whole process could be made a LOT easier if Adobe wanted to.

In the screencast, I refer to an article I wrote a while back, entitled “The next stage for Lightroom“, where I put forth a proposal for improving the way Lightroom stores photos, with an eye on catalog portability (laptops, for example). If you have the time, please read through that article after you see the video.

The video is about 10 1/2 minutes long, file size is 78MB, and it’s 720p HD, MOV. You can view it online by clicking on the screenshot below, or download it.

Problems with meta-data entry in Lightroom

I’ve been having problems with meta-data entry in Lightroom, particularly the location meta-data. It’s not my computer that’s to blame. The problems exist on both the PC and Mac versions of Lightroom, although they’re more apparent on the Mac version.

When I first mentioned this on Twitter and Friendfeed (see here, here and here) I promised I’d put together a video if I found a good screencasting application. Ken Verburg pointed me to iShowU, which I used to make a short screencast demonstrating the problems. Thanks Ken!

I uploaded the video to my Vimeo account last night, but don’t try to view it there. The compression is terrible and you won’t be able to see what I’m doing. Instead, go there and scroll down till you see the download link. Click on that and view it locally on your computer (you’ll need Quicktime or iTunes). Another option is to download the video directly from my site. The original video has no loss in quality and you’ll be able to see everything properly.

The video will do a good job of conveying my frustration with the user interface in Lightroom. I really do hope the final version of Lightroom 2 will address these issues, because entering location metadata in Lightroom right now is terribly frustrating.

Shenandoah Valley panoramas

You are about to see several panoramic photos that have taken me well over 35 hours to create — and I’m not counting travel time, setup time, time it took to take the photos, and the time it took to write this post.

This slideshow requires JavaScript.

A couple of weekends ago, Ligia and I got into our MINI and drove up to Shenandoah National Park, for a single purpose: to take a few panoramic photos of the valley from the tops of the Appalachian Mountains. Fortunately, that simply meant driving on beautiful, scenic Skyline Drive and stopping at various points to set up the tripod and take series of shots that would later be stitched together. Unfortunately, the weather wasn’t exactly collaborating, in spite of the cheery weather report. The day was neither sunny nor cloudy. The light was diffuse and had that washed, in-between quality that doesn’t really make it good for anything. But, I was there, and if that’s what I had to work with, so be it.

As it turned out, driving out there and taking the photos was the easiest part of the whole thing. Like I mentioned in the opening paragraph, putting together the panoramas was by far the longest, most excruciatingly slow stretch of processing work I have ever done. I do not recommend it to anyone, for multiple reasons, which I’ll mention below. If you just want to see the photos, skip ahead.

A few thoughts on the whole thing

I will not do panoramas very often in the future, unless I’m commissioned to do specific ones. If and when I do another panorama for myself (not for a client), it will likely only be a 5 to 10 photo image, simply because it takes an enormous amount of time to stitch and process them on the computer if they’re made up of more images than that.

For one thing, you would need a super fast, quad-core or better computer loaded to the gills with RAM to get any sort of decent speed while processing panoramas. A Mac Pro worth about $7,000 or better should do the trick. Seriously, every single simple operation, like cropping or rotating, took at least 10 minutes or more to execute. Sometimes just assembling a single panorama in Photoshop (through the Photomerge feature) took about 45 minutes. I have the latest MacBook Pro laptop (2.5 GHz Intel Core 2 Duo processor, 512 MB video card, 4 GB RAM), and it still took what seemed like forever to get through each panorama.

The resolution of the photos also matters. My individual photos are 12-megapixels each, at 240 dpi, made by a Canon 5D. Just imagine how much processing power is needed to put together 20 or 30 of these photos into a single image!

People don’t appreciate panoramas. I bet you most people will skim this post, unimpressed, and move on. You can’t really appreciate panoramic photos unless they’re printed out in their full size and spread out on a wall, right in front of you. You can’t appreciate their size on your computer monitor, no matter how large it might be. The largest single monitors nowadays are 30″ and have a maximum resolution of 2560×1600. That’s equivalent to about 6 megapixels at 72 dpi. You can’t possibly appreciate a 12 megapixel photo at 1:1 size on a current-day monitor, much less a panorama made up of 20 of those photos.

As an aside, don’t confuse monitor size with resolution. There are LCD HDTVs on the market that are 42″, 46″ or more in size, but they can only display up to 1920×1080 pixels, which is much less resolution than a 30″ monitor.

I can’t show you the full panoramas on my site, because of photo theft. Not that I think my panoramas (these ones in particular) are spectacular and would fetch amazing prices, but I know for a fact that if I post my panoramas at full resolution, there will be people who will steal them and try to profit from them.

How does the new Lightroom 2 Beta handle panoramas?

After I processed the photos in Photoshop CS3, which worked without crashing for the whole bunch, although it ate an amazing amount of space on my hard drive for its scratch disk, I imported them into the new Lightroom 2 Beta, to see how it would handle them.

Most of the photos were over 1 GB in size, uncompressed. Because I saved most as TIFs, using ZIP compression, their file size on disk was significantly lower. Lightroom did amazingly well to start with. It created small previews very quickly, and also created the 1:1 previews much quicker than Photoshop would have been able to render them. I was able to use the spot heal brush to remove sensor dust spots, and also used the new selective retouching brushes, without any problems. Lightroom 2 was able to do these things without significant delays, and would show the effects instantaneously.

LR2 only started hiccuping when I started to add some meta-data to the photos. As I went through and added meta-data, then opened them at 1:1 size once more, it would hang, literally forever. I had to keep force quitting it, and had to do that regularly, for each and every photo that I wanted to look at. Interestingly enough, when I wanted to export the panoramas to use them here on my site, it did it without any problems, and without crashing. It’s certainly odd behavior, but it is in Beta after all.

On to the photos

While I cannot post the panoramas at full resolution here, I did post them at higher sizes than I would normally post, in order to give you a better idea of what they look like. I also created 1:1 previews of regions of each photo, to help you realize how big they really are.

If you click on each panorama (not its 1:1 detail), it will take you to its photo page, where it will tell you how large it is (in megapixels), and how many photos went into making it. If you click on it again (on that page), it will take you to its larger size. Sorry for the double-clicking, but that’s how things work in WordPress these days.

First, a panoramic of Skyline Drive itself. This road is amazing, and I’m so glad the US government decided to build it back in the 1930s. It literally hugs the tops of the Appalachian mountains and lets average John and Jane drive on top of the world (as high as possible in this area of the world, anyway).

We stopped along Skyline Drive, parked our car, and took a hike through the forest on one of the paths marked out there. In the middle of nowhere (literally), we found this cabin, called Range View.

It was a darling little place built out of stone and off the grid (in spite of the fact that wires ran right above it). The fireplace was outside the cabin, by the front door. While the place was locked up and the windows equipped with thick wire and netting, Ligia and I could spot beds and various pieces of old furniture inside. Don’t know what it’s used for nowadays, but it is used, because there was an open bottle of wine standing in plain sight near one of the windows, and it was of recent vintage.

The rest of the photos, including the 1:1 previews, are found in the gallery below. Click on each to get to the photo page, then click again to see it in a larger size. Enjoy!