Events

The time a Drobo lost over 30,000 of my photos and videos

The following is an account of what happened to me in late 2012, when two of my Drobo units malfunctioned and corrupted or erased over 30,000 of my photos and videos, rendering them either completely unusable or making them vanish into digital ether.

I have been a Drobo customer since late 2007, when I bought my first USB Drobo. I was glad to take advantage of the new storage possibilities it offered. Since then, I’ve bought one or two Firewire Drobos, two Drobo 5Ns and a Drobo 5D. After relying on my Drobos for several years, I experienced catastrophic data corruption on both of my Drobos at the time (a USB Drobo and a Firewire Drobo). That incident still haunts me to this day.

If you look through my past posts about the Drobo, you’ll see that I wrote about them and I defended them (speaking from my own experience at the time) when others said, among other things, that they “can’t be relied upon” and they often turn into “bricks” or lose people’s data. From my own experience, I had only good things to say about them. They hadn’t failed me for years, I hadn’t yet lost any data from them, the support was great and their design is beautiful. They are a pleasure to use when they work. Until recently, few things in hardware got me more excited than buying a new, bigger hard drive for my Drobo and sticking it in, then watching the capacity gauge show the new, larger volume and hearing it start to sync the data to the new drive…

Which brings me neatly right to the crux of the problem. I had bought a new 3TB hard drive for my Firewire Drobo and stuck it in, replacing an existing 2TB drive. The data sync had gone on for close to 30 hours and was getting close to completion (the Dashboard software was telling me there were only 5-6 hours left). All of a sudden, a red light next to the new 3TB hard drive went off. Before I had a chance to assess the situation, the Drobo crashed and rebooted. I thought, okay, it’s resetting itself and will begin to work once more. I waited for the reboot to complete, only to discover, to my dismay, that it rebooted again… and again… and again… During those brief moments when it was up, the red light switched places, moving from the 3TB hard drive to one of the existing 2TB hard drives. This wasn’t good. I decided to shut it off completely and unplug it. I filed a ticket with Drobo Support and waited.

This horrific experience came after I experienced catastrophic data loss on my USB Drobo just a couple of days earlier. That Drobo was and had been plugged into a surge protector (unlike my Firewire Drobo, which had been plugged into a UPS). The electricity had been cut off and when it came back up, the USB Drobo wouldn’t mount to the desktop. No big deal, I thought. The electricity had gone off before and when the Drobo wouldn’t mount, I would run Disk Utility on it and fix it right up. That’s what I did this time, only Disk Utility couldn’t repair it. Oh boy… I filed a ticket with Drobo Support. They told me they couldn’t help and that I should buy Disk Warrior or send it to a data recovery firm. I called the firm to get a quote, which they said would be anywhere from $1,200 to $7,900, depending on how much data they recovered. They told me that realistically, it would cost at least $4,000 if they recovered any data at all, and that it’s difficult to get the data off a corrupt Drobo volume, because of their complicated storage algorithms. They said it’s much easier with traditional RAID systems.

So I bought Disk Warrior and tried my luck. It was able to mount the Drobo temporarily, but couldn’t fix it. The error it gave me was that Disk Utility had messed up the catalog file when it tried to fix it and they couldn’t recreate it. I called their Tech Support and one of their technicians stayed on the phone with me for over an hour. We did a Screen Sharing session in iChat, where he kept running scans on the volume and checking the diagnostic files. Same result. Couldn’t be fixed. He suggested I get whatever data I could get off it directly through the Disk Warrior’s Preview feature. I started to copy off the data and got about 70GB off (onto the Firewire Drobo) which hadn’t crashed yet. Then the USB Drobo decided to call it quits and became completely unusable.

Out of over 3TB of data, I was able to salvage temporarily about 70GB. I lost the rest: over 15 years of personal and family information, photos, personal videos, letters, our movie collection (which I had painstakingly digitized and edited from VHS tapes and TV recordings over several years), our cartoon collection (also painstakingly digitized and edited from VHS tapes and TV over several years, and which I was looking forward to watching with our unborn baby — my wife was pregnant at the time; our daughter is now almost six years old) and other various files.

I decided not to send the Drobo to data recovery because the price to recover the data would have been higher than the price to buy all of the movies and cartoons on DVD once more, and because I don’t feel comfortable with other people rifling through my personal files. I didn’t know how much they’d be able to recover, if anything, and I consoled myself with the notion that at least my Firewire Drobo was still fine and I had my work files (my photo library of over 100,000 photos, the footage and FCP and iMovie events and projects from our shows (Romania Through Their Eyes and Ligia’s Kitchen), various other personal and published videos, etc. — basically, my creative work over the past 6-7 years.

Then, my Firewire Drobo went AWOL. As mentioned above, I filed a ticket with Drobo Support and waited to see what they would say. Their advice: buy Data Rescue 3 and run it on the 2TB drive that showed up with a red light next to it. Apparently the drive went bad during the data sync for the new 3TB drive. With Data Rescue 3, I could in theory clone it, then insert the cloned drive into the Drobo and recover my data. Well, I couldn’t clone the whole drive. I tried it a couple of times, but whenever the head came to a certain point on the platters, the drive would disconnect itself from the computer. So I did a reverse clone (where the cloning process begins at the end, not the start of the drive) to try and recover the rest of the data after the bad section on the drive. After that completed and with the blessing of Drobo Support, I called that a “best try” clone, stuck the new drive into the Drobo, put it in Read Only Mode and through Data Rescue 3, began to copy off my data.

Things looked good at the time. The software seemed to see all my files and although the copying process was slow, I had access to my data, which was the important thing. After 4-5 days of data retrieval, I was done and at Drobo Support’s advice, I reset the Drobo and formatted a new volume onto it, then began to copy my data back onto it. It was deemed healthy once again and I was sent a new power supply. I’m still not sure why the new power supply was needed, but alright.

After all my data was back onto the newly formatted Firewire Drobo, I began to access it. Only then did I discover data corruption at the file level. Photos, precious memories which I’d trusted my Drobo to keep were now forever lost. Photos of when I met Ligia (my wife), photos from the wedding, from our first years together… they’re gone now. Videos have turned into audio files (the video track can no longer be seen by video players or video editing programs. Just from the year 2007, I lost well over 17,000 photos (about 80% of them). The same percentage applied to the years 2000-2006, when I took less photos but lost the same proportion of files (about 80%). I still don’t know the exact number of photos I lost, but I estimate it to be well over 30,000. I still dread to open older video libraries in Final Cut Pro, because when I access my projects, I always find video clips that are either missing or completely unusable.

When I updated my ticket with Drobo Support to let them know about this, data recovery services were once more suggested. Now I’m not a data recovery expert, but after you’ve formatted a volume and written a LOT of data to it, doesn’t that make it harder, if not impossible, to get the old data back?

If you’d like to see what file-level corruption looks like, it’s not pretty. Here are a few screenshots. Those thumbnails are all I have left of photos I wanted to keep forever.

Let’s look at this thing from both sides.

The pros:

  • These two Drobos worked for five years, flawlessly;
  • My experience with Drobo Support was good from 2007 till 2012; even though the warranty had expired on both of these Drobos, they took up my cases and tried to help me;
  • I should have had my data backed in other places, I know, I know; That’s exactly what I as doing as the Drobos failed. I was reorganizing files and reshuffling my archives when it happened. If you were to ask me, as an IT professional, what I would recommend to others, it would have been the following: have the data in its primary location on a reliable and fast device, back it up to a server or NAS on the local network and also back it up offsite. I was caught in-between, with no access to my old backups and no proper backup plan in place. Crashplan, the offsite backup company I was using at the time, had been throttling my uploads to conserve their bandwidth, and the offsite backups hadn’t completed. If they hadn’t been actively engaged in throttling user uploads, as was their practice at the time, I could have recovered all my files.
  • Drobos are easy to use and allow one to consolidate their data on one volume that can keep growing and growing and growing with the addition of new drives — that part I always liked;
  • Even if one drive fails, the data’s still safe; I had that happen to me once in years past and yes, the data was fine — that part I also always liked;
  • I should have had the USB Drobo plugged into a UPS, just like the Firewire Drobo. Should have, would have, could have…

The cons:

  • Two catastrophic failures within one week… c’mon, this is ridiculous to say the least…
  • I had no idea you could lose all your data from the Drobo if the power went out, and that it would cost upwards of $5,000 to get it back. Ugh… if it costs so much to get the data back, Data Robotics should seriously consider doing what ioSafe has been doing from the start: offering Data Recovery Services standard with every one of their products.
  • Apparently the newer Drobo products, like the 5D, the Drobo Mini and the 5N, have a built-in battery that protects against power outages, so what happened with my USB Drobo won’t happen to them, theoretically. Well… if they knew about this, couldn’t they have put that battery/capacitor in the older Drobos as well? I sure wish they had taken every step possible to make sure any data placed on a Drobo stayed safe. After all, they’ve been saying for years that they “protect what matters”. Do they really?
  • Earlier this year, a friend of mine wrote to me telling me of his experience with his USB Drobo. It was the very same thing that happened to mine, minus the power outage. He would eject it from the desktop and unplug it from the outlet, thinking that was safe. It was, for a while, until he lost all his data. At the time, I thought he must have been doing something wrong. Until it happened to me. Is it just me, or do we not do the same with a regular external drive? Do we not eject it then turn it off? Does the electricity not go off sometimes, and even if the Drobo is plugged into a UPS, the electricity may be out for more than 10 minutes, meaning the UPS will also go down? And when the electricity comes back up, could we not also lose our data even though we did everything right?
  • I remember jokingly asking someone from Data Robotics in years past what would happen if a drive went bad right after you inserted a new one and the Drobo was still in Data Protection mode. Their response: not very likely that could happen. The Drobo would know that a drive was about to go bad and would give me warning. It’s quite apparent to me now that that’s not the way things work sometimes. Sometimes a drive does go bad during Data Protection and not only do you get no warning, you also get catastrophic data loss. Double whammy.
  • Someone from Drobo Sales happened to write to me to invite me to a webinar as I was in the throes of my data loss. I replied and told him of my troubles, and that was the last I heard from him. Was that the way to treat a long-time customer? An adequate response would have been “Hey, I’m really sorry about that. Are you getting the help you need from Tech Support? If not, do you want me to step in for you?”

This whole affair left me exhausted and completely disappointed. I wasted so much time trying to recover my data and my bearings. The experience has taught me a few things, one of them being the loss of my trust in Drobo. It is a beautiful little black box. It works great and you begin to rely on it more and more, till you can’t do without it; till you think it works so good your data’s going to be fine all the time. Then it fails and your data is gone.

If you’d like to know how I recovered some of my data, I wrote about it here. I also experienced another major data loss event with my Drobo 5D just last month, and that was after more data loss with my 5D in 2015 and 2017. Add to that ongoing file system corruption in one of my Drobo 5N units and the “Drobo data loss package” is complete. I don’t know, I must have unlocked some kind of secret data loss level-up in the data storage game…

Standard
Thoughts

The flip side of digital photography

Should you be old enough, you’ll remember how different photography was before the arrival of digital cameras. Not only was it difficult to get great photos, the kind that were good enough for publication, but it was difficult to develop them and reproduce them. There were real barriers to entry and to success in the field. They weren’t insurmountable, but they were there.

Nowadays, digital cameras make it so easy for us. Even a novice can occasionally get a great photo simply by clicking the shutter button, because modern cameras can pretty much handle all situations. They don’t do everything, you still need to know what you’re doing in some scenarios, but they’ll get you pretty close to your desired result by themselves, most of the time. So not only is it easy to take photos, but it’s also easy to “develop” them using your computer, and you can reproduce them endlessly. The barriers to entry and success in the field are now almost gone.

However, one thing we all learn as we age is that everything comes with pluses and minuses. Just like film photography had certain minuses, digital photography comes with plenty of unpleasantries on its flip side.

Publications that used to hire photographers and pay them good wages are dwindling. How many do you know of that still have on-staff photographers, or hire photographers for their stories? And how do their salaries compare with those of photographers in the past if they’re adjusted for inflation?

Stock agencies are decreasing the payouts to photographers. There is a lot of competition in that market, paired with a real glut of photographs. And when the supply always outnumbers the demand, prices will fall. There are but a few stock agencies left. There are a ton of microstock agencies which sell photos for piddly sums and pay cents on the dollar to photographers, and they’re also getting bought out and merging with each other in order to survive. If it wasn’t clear a few years ago, it’s becoming painfully clear now that a photographer cannot make a living selling microstock. There are a few who manage to do it, but it’s clear that on average, microstock yields a non-livable income.

There are so many photographs being made that people don’t truly appreciate them anymore. Do you remember how we used to admire photographs in the past? We’d stare at them for 5-10 minutes at a time, taking in each detail. We’d cut them out of magazines and paste them in scrapbooks. We’d look at them and look at them and look at them… Now we’re lucky if a photo gets 5 seconds of someone’s time. There are so many of them that people just gloss right over a photo that took days or hundreds of tries to make. Perhaps you’ll understand this better if I compare it to a periodical cicada emergence. In just a few days, animals that would eagerly consume them as they came out, would become so glutted that they’d simply lay on the ground and watch them crawl around and over them, unable to eat a single morsel. That’s what’s going on with photographs now. Each of us has a rhythm, a rate of “ingesting” digital content and we’ve all reached our max, but the photographs just keep coming. They keep coming and their rate of production is actually increasing. We cannot keep up.

Digital photography gear is made to become obsolete, causing you to spend more money every few years. Remember how you could use the same film camera for 10-20 years, even a lifetime, if you took care of it? That’s not the case with digital cameras, which typically last about 4-5 years before something goes bad. Even if you’re willing to pay a repair shop to have it fixed, camera manufacturers stop stocking parts for older cameras after a certain number of years, because they want to force you to buy a new model. I wanted to send my Canon 5D in for repairs last year, but I couldn’t. The repair shop said I shouldn’t bother, because Canon actually doesn’t allow them to work on the 1st gen 5D anymore and they’ve stopped stocking parts. Not that Canon repair experiences were so great to begin with, but at least they got the job done. I also sent in my Olympus PEN E-P2 in for repairs last year, but it didn’t get repaired. It came back just as I sent it, with a message that offered apologies for the inconvenience and explained that they’d stopped stocking parts for that model just a few months back; support had been discontinued by Olympus. I don’t understand it: there’s money to be made with service and repairs, so why stop supporting a model? Why not keep servicing it for as long as the customer is willing to use it? That business model has been proven to work a long time ago by the car industry.

Cameras, lenses and flashes are getting more expensive each year. Manufacturers can call them inflation adjustments all they want, but price hikes still feel very much like price hikes. And when they’re coupled with no real way to make money from your photos anymore, what are you left with? Doing weddings? Yuck. I don’t know how photographers are coping with all of this. I have a nagging feeling that wedding photographers are pretty much the only ones making money from photography these days. They’re certainly the bulk of the paying customers for camera manufacturers. It’s them and the online “experts” that have sprouted like mushrooms after rain, offering “advice” about which camera model to buy on YouTube and other video sites. It’s a new model/brand each week of course, unless they’re getting paid by a manufacturer to promote a certain brand.

There are real costs associated with processing, storing and archiving digital photographs. We’re told that digital photographs are pretty much free and there’s never been a better time to take many, many photos in order to learn the craft, but there are significant costs that come into play when you add the price of a good computer and good software and the storage and backup solutions that you will absolutely need unless you want your photos and your hard work to go up in a puff of virtual smoke. I’d like to challenge you to add up the costs of your camera gear (camera, lenses, flashes, adapters, tripods, etc.) and computer equipment (laptop/desktop, external hard drives, backup equipment/services) and once you have a total, divide it by the number of photographs you’ve taken with your camera so far. That’ll give you a pretty good idea of the cost per image, and you’ll see that digital photographs are not free. Granted, that cost per image will go down the longer you keep your current equipment and the more photos you take with it, although the cost of storage and backup will still be there for your larger collection of photographs. Do you realize you’ll likely need to pay for a backup subscription for the rest of your life? It’s no wonder that more and more people choose to take photos with their smartphones and edit them directly on those devices, forgoing the cost of computer equipment. And when smartphone manufacturers also offer direct and almost instantaneous cloud backup of the images and videos taken with the phones (at somewhat reasonable prices) it becomes a very attractive offer.

It’s so easy to reproduce digital photographs that it’s actually a problem, because anyone can steal and plagiarize them. Theft of online photographs is rampant. It’s one thing for a fan to repost your photos on another site — I’d go so far as to say that’s fine… but it’s quite another thing for someone to download your photos, enlarge them in Photoshop and repost them on a stock site or use them in ad campaigns, and this is happening quite a lot.

There is no consistent way to attribute photographs online, which means a photographer’s name is likely to get lost in the shuffle. Sure, you can use a caption that lists the photographer’s name, but that only works if you’re the primary publication and you’ve worked with the photographer. Most software used to export and compress images for online publication generally strips EXIF and IPTC copyright information. And most online platforms also have no consistent way of keeping that information inside the photographs, instead offering excuses about file size and compression algorithms which sound very empty given how far we’ve come with computer technology. Have you ever tried to find a photographer’s name for a photo reposted on social media? Good luck… Unless they’ve got a tasteful watermark somewhere on the photo, the metadata’s been wiped clean by these sites. Even Flickr still does not keep a photographer’s name in the metadata of a photo. Should you be able to download a photo from a Flickr contact, you’ll get a link to the page where it was found and maybe a caption, but you will not get something as basic as the photographer’s name, much less the rest of the copyright information.

I’m not saying we should go back to film and analog equipment. I love digital cameras and their ease of use. And I love the various advances being made in digital camera gear. Some of the minuses listed above can even be fixed. I’m just not enthusiastic about their flip side. When photographs were harder to make, we appreciated them more and good photographers stood a good chance of making good money with them. Now that photographs are easy to make, we don’t appreciate them and income from photographs has gone down to pennies on the dollar, if at all. Thank goodness I take photographs for the sake of it, as a creative endeavour that relaxes me after working on my various projects, but I wonder how others are coping with these changes. And it’s also not to say that I wouldn’t mind making money from my photographs on my own terms.

Standard
Reviews

A review of the Stellar Phoenix Photo Recovery software

Having lost photos and videos in the past, I am fairly cautious about my media these days. I keep local and remote backups and I use hardware that writes my data redundantly onto sets of drives, so that I don’t lose anything if one of the drives goes down. I have also purchased data recovery software, just in case something goes bad: I own both Disk Warrior and Data Rescue.

When someone from Stellar Phoenix contacted me to see if I’d be interested in looking at their Photo Recovery software, I agreed. I wanted to see how it compared with what I have. In the interest of full disclosure, you should know they gave me a license key for their paid version of the software.

I put it to a test right away, on what I deemed the hardest task for data recovery software: seeing if it could get anything at all from one of the drives I pulled out of one of my Drobo units.

As you may (or may not) know, Data Robotics, the company that makes the Drobo, uses their own, proprietary version of RAID called BeyondRAID. While this is fine for the Drobo and simple to use for Drobo owners, it also means that data recovery software typically can’t get anything off a drive from a Drobo drive set. Indeed, after several hours of checking, Stellar Phoenix’s software couldn’t find any recoverable files on the drive. I expected as much, because I know specialized, professional-grade software is needed for this, but I gave it a shot because who knows, someday we may be able to buy affordable software that can do this.

Screen Shot 2018-02-24 at 16.30.13

The Seagate 8TB drive is the one I pulled out of the Drobo

Screen Shot 2018-02-24 at 18.29.31.png

What the software found is data gibberish; there were no MP3 or GIF files on that drive

Now onto the bread and butter of this software: recovering photos and videos from SD cards. I made things harder for it again, because I wanted to see what I’d get. I put a single SD card through several write/format cycles by using it in one of my cameras. I took photos until I filled a portion of the card, downloaded them to my computer, put the card back in the camera, formatted it and repeated the cycle. After I did this, I put the software to work on the card.

Before I tell you what happened, I need to be clear about something: because no camera that I know of and no SD card that I know of has any hard and fast rules about where (more precisely what sector) to write new data after you’ve formatted the card, the camera may very well write the bits for new photos/videos right over the bits of the photos/videos you’ve just taken before formatting the card. This makes the recovery of those specific photos that have been written over virtually impossible. What I’m trying to tell you is that what I did results in a file recovery crapshoot: you don’t know what you’re going to get until you run the software on the card.

When I did run it, it took about 40 minutes to check the card and it found 578 RAW files, 579 JPG files and 10 MOV files. Since I write RAW+JPG to the card (I have my camera set to record each photo in both RAW and JPG format simultaneously), I knew those files should be the same images, and they were.

Screen Shot 2018-03-12 at 18.05.31

The software found photos and videos from several sessions and dates

Screen Shot 2018-03-12 at 18.05.54.png

As you can see from the dates, they ranged from March 11 to February 13

I then told the software to save the media onto an external drive, so I could check what it found.

Screen Shot 2018-03-12 at 18.37.08.png

It took about 30-40 minutes to recover the data

When I checked the files, I saw that it recovered two sets of JPG files: each one contained 579 files, but one of the sets began its file names with “T1-…”; they were the thumbnails of the images. All of the JPG files were readable on my Mac. It was a different story with the RAW files. It recovered three sets of RAW files, each containing 578 files. The first set was readable by my Mac. The second set, marked with “T1-…” wasn’t readable at all and the file sizes were tiny, around 10KB in size; they were the thumbnails of the RAW files. The third set, marked with “T2-…” was readable, but the file sizes were around 1MB a piece; they were the mRAW files written automatically by the camera, at a resolution of 3200×2400 pixels. A typical RAW file from the camera I used for my testing ranges in size from 12-14MB and its resolution is 4032×3024 pixels. It’s kind of neat that the mRAW (or sRAW) files were recovered as well.

Now I took 3,328 photos with that camera from February 13th – March 11th. It recovered 578 photos, so that’s a 17% recovery rate. Granted, I made it very hard for it by writing to the card in several cycles and reformatting after each cycle. When I only look at the last set of photos recorded to the card, before the last reformat, I see that I took 523 photos on March 10th and 3 photos on March 11th. The software recovered 525 photos on March 10th (so there’s some doubling up of images somewhere) and 2 photos on March 11th. However, don’t forget about the JPG files, which contained the missing image. So that’s a 100% recovery rate.

In all fairness, there is free software out there that can do basic recovery of images from SD cards and other media, so the quality of a piece of software of this nature is determined by how much media it recovers when the free stuff doesn’t work. I believe I made things hard enough for it,and it still recovered quite a bit of data. That’s a good thing.

Let’s not forget about the video files. Those were written to the card with another camera and they ranged in dates from November 3-6, 2017. I’m surprised it recovered any at all. It gave me 10 video files, out of which 5 were readable, so that’s a 50% recovery rate.

Just for kicks, I decided to run Data Rescue on the SD card as well. It also found 579 JPG files and 578 RAW files. All were readable by my Mac. It also found 10 video files, but none were readable. However, I have Data Rescue 3, which is quite a bit old. Data Rescue 5 is now out, but I haven’t upgraded yet. It’s possible this new version might have found some more files.

Price-wise, Stellar Phoenix Photo Recovery comes in three flavors: $49 for the standard version (this is the one I got), $59 for the professional version (it repairs corrupt JPG files) and $99 for the premium version (it repairs corrupt video files in addition to the rest).

The one thing I didn’t like is that the Buy button didn’t go away from the software even after I entered the license key they gave me. As for the rest, it’s fine. I think it crashed once during testing and it didn’t happen while actually recovering data. The design is intuitive and at $49, this is software you should definitely have around in case something bad happens to your photos or videos. It may not recover all of what you lost, but whatever you get back, it’s much better than nothing, which is what you will definitely get if you don’t have it. It’s also a good idea to have multiple brands of this kind of software if you can afford them, because you never know which one will help you more until you try them all. And believe me, when you’re desperate to get your data back, you’ll try almost anything…

Remember, back up your data and have at least one brand of data recovery software in your virtual toolbelt. Stay safe!

Standard
Reviews

A review of Google’s Backup and Sync

google-drive-to-backup-sync

Google launched this new service in the second half of 2017. I remember being prompted by the Google Drive app to install an upgrade, and after it completed, I noticed a new app called “Backup and Sync” had been installed, and the Google Drive app had become an alias.

Screen Shot 2017-12-26 at 14.08.28.png

The new app sat there unused for some time, until I discovered its new capability, namely to back up and sync other folders on my computer, not just the Google Drive folder. This was and is good, new functionality for Google, because it ties in very nicely with its Photos service, which has already been offering the ability to back up all of the photos and videos taken with mobile devices to the cloud through the Google Photos mobile app. I’ve been using Google Photos for several years, going back to when it was called Picasa Web.

I set it to back up all of my photos and videos, allowing Google to compress them so I could back up the whole lot. (It’s the “High quality (free unlimited storage)” option selected in the screenshot posted below.)

Screen Shot 2017-12-26 at 15.01.14.png

I already back up all of my data with Backblaze, which I love and recommend, but it doesn’t hurt to have a second online backup of my media, even if it gets compressed. Having lost some 30,000 images and videos a few years back, I know full well the sting of losing precious memories and when it comes down to it, I’d rather have a compressed backup of my stuff than none at all.

Screen Shot 2017-12-26 at 14.15.12.png

The thing is, there are shortcomings and errors with this new service from Google, which I will detail below. The backup itself was fast. Even though I have several terabytes of personal media, they were uploaded within a week. So that’s not the issue. After all, Google has a ton of experience with uploads, given how much video is uploaded to YouTube every single day.

Screen Shot 2017-12-26 at 14.08.51.png

As you can see from the screenshot posted above, it was unable to upload quite a few files. The app offers the option of uploading RAW files in addition to the typical JPG, PNG and videos, but it couldn’t upload RAW files from Olympus (ORF), Adobe (DNG) and Canon (CR2). They were listed among the over 2700 files that couldn’t be backed up.

Screen Shot 2017-12-26 at 14.09.56.png

I ended up having to add the extensions of RAW, PSD, TIFF and other files to an “ignore” list located within the app preferences. This is the full list I’ve added there so far: DNG, TIFF, RAF, CRW, MOV, PSD, DB, GRAPHDB, PLIST, and LIJ. It seems there’s a file size limit on images and on videos, because most of my large images (stitched panoramas) and videos of several GB or more didn’t get uploaded. That’s a problem for an app that promises to back up all your media.

There were also quite a bit of crashes. The app crashed daily during the upload process and even now, it crashes every once in a while. I set up my computer to send crash reports to Apple and to the app developers, so I assume that Google got them and will at some point issue an upgrade that fixes those bugs.

I also kept running out of space on my Google account. Given that I’d set the app to compress my images so I’d get “free unlimited storage”, and I’d also set it to back up only my images and videos, this didn’t and doesn’t make sense. Add to this the fact that it’s trying to back up unsuccessfully all sorts of other non-image files (see the paragraph above where I had to add all sorts of extensions to the ignore list) and once again, this app seems like it’s not fully baked. I ended up having to upgrade my storage plan with Google to 1 TB, so it’s costing me $9.99/month to back up most (not all) of my images and videos, compressed, to a service that offers “free, unlimited storage”. The app says I’ve now used up 408 GB of my 1 TB plan. Before I started backing up my media, I was using about 64 GB or so, adding together Gmail and Google Drive. So about 340 GB are getting mysteriously used by some invisible files that I can’t see in Google Photos or Google Drive, but they’re obviously stored somewhere by the Backup and Sync app.

Remember, this is Google. They have a ton of experience with apps, with images and with videos, so why did they push this out when it still has all these issues?

Standard
Thoughts

Fun with technology

I’ve had multiple Drobo units since 2007. To this day, I still enjoy adding a hard drive to a Drobo. It’s one of those things that can be an ordeal on other tech, but on a Drobo, it’s been made fun through proper planning and design.

It lets you that it’s low on space, you order a drive, and when it comes, you look at the app, which tells you exactly what size-drive is in each bay. Pressing a small lever on the side of the bay releases the drive, which slides out. You put the new one in, the Drobo immediately checks it and formats it, then begins striping the data set across it. By the way, that’s a screen shot showing my Drobo 5D.

Screen Shot 2017-12-21 at 12.48.30.png

I love this process. It’s so simple and so fun! The Drobo doesn’t care what hard drive you buy, as long as it’s larger than what you already had. It allows you to grow the capacity of your Drobo in time, as the prices for newer, bigger hard drives decrease, without any sort of headaches. This is technology done right.

Standard
Reviews

A comparison of CrashPlan and Backblaze

I’ve been a paying CrashPlan customer since 2012 and my initial backup still hasn’t finished. I’ve been a paying Backblaze customer for less than a month and my initial backup is already complete. 

I’m not a typical customer for backup companies. Most people back up about 1 TB of data or less. The size of my minimum backup set is about 9 TB. If I count all the stuff I want to back up, it’s about 12 TB. And that’s a problem with most backup services.

First, let me say this: I didn’t write this post to trash CrashPlan. Their backup service works and it’s worked well for other members of my family. It just hasn’t worked for me. This is because they only offer a certain amount of bandwidth to each user. It’s called bandwidth throttling and it saves them money in two ways: (1) they end up paying less for their monthly bandwidth (which adds up to a lot for a company offering backup services) and (2) they filter out heavy users like me, who tend to fill up a lot of their drives with unprofitable data. My guess (from my experience with them) is that they throttle heavy users with large backup sets much more than they throttle regular users. The end result of this bandwidth throttling is that, even though I’ve been a customer since 2012 — at first, I was on the individual backup plan, then I switched to the family plan — my initial backup never completed and I was well on track to never completing it.

When I stopped using CrashPlan’s backup services, out of the almost 9 TB of data that I need to back up constantly, I had only managed to upload 0.9 TB in FOUR YEARS. Take a moment and think about that, and then you’ll realize how much bandwidth throttling CrashPlan does on heavy users like me.

Screen Shot 2016-10-20 at 23.37.07.png

After four years of continuous use, I backed up a grand total of 905.7 GB to CrashPlan

To be exact, counting the various versions of my data that had accummulated on the CrashPlan servers in these four years, I had a total of 2.8 TB stored on their servers, but even if you count that as the total, 2.8 TB in FOUR YEARS is still an awfully small amount.

Screen Shot 2016-10-27 at 00.42.14.png

Space used on CrashPlan’s servers: 2.8 TB

Tell me honestly, which one of you wants this kind of service from a backup company? You pay them for years in a row and your initial backup never finishes? If a data loss event occurs and your local backup is gone (say a fire, flood or burglary), you’re pretty much screwed and you’ll only be able to recover a small portion of your data from their servers, even though you’ve been a faithful, paying customer for years… That just isn’t right.

I talked with CrashPlan techs twice in these fours years about this very problematic data throttling. Given that they advertise their service as “unlimited backup”, this is also an ethical issue. The backup isn’t truly unlimited if it’s heavily throttled and you can never back up all of your data. The answer was the same both times, even the wording was the same, making me think it was scripted: they said that in an effort to keep costs affordable, they have to limit the upload speeds of every user. The first time I asked them, they suggested their Business plan has higher upload speeds, so in other words, they tried to upsell me. During both times, they advertised their “seed drive service”, which was a paid product (they stopped offering it this summer). The gist of their paid service was that they shipped asking customers a 1 TB drive so you could back up to it locally, then send it to them to jumpstart the backup. Again, given my needs of backing up at least 9 TB of data, this wasn’t a userful option.

Screen Shot 2016-10-31 at 15.57.25.png

This is false advertising

Screen Shot 2016-10-31 at 15.59.41.png

This is also false advertising

Some of you might perhaps suggest that I didn’t optimize my CrashPlan settings so that I could get the most out of it. I did. I tried everything they suggested in their online support notes. In addition to tricking out my Crashplan install, my computer has been on for virtually all of the last four years, in an effort to help the Crashplan app finish the initial backup, to no avail.

Another thing that bothered me about CrashPlan is that it would go into “maintenance mode” very often, and given the size of my backup set, this would take days, sometimes weeks, during which it wouldn’t back up. It would endlessly churn through its backup versions and compare them to my data, pruning out stuff, doing its own thing and eating up processor cycles with those activities instead of backing up my data.

Screen Shot 2016-10-22 at 19.40.33.png

Synchronizing block information…

Screen Shot 2016-10-23 at 14.39.36.png

Compacting data… for 22.8 days…

Screen Shot 2016-10-23 at 16.58.23.png

Maintaining backup files…

I understand why maintenance of the backups is important. But what I don’t understand is why it took so long. I can’t help thinking that maybe the cause is the Java-based backup engine that CrashPlan uses. It’s not a Mac-native app or a Windows-native app. It’s a Java app wrapped in Mac and Windows app versions. And most Java apps aren’t known for their speed. It’s true, Java apps could be fast, but the developers often get lazy and don’t optimize the code — or that’s the claim made by some experts in online forums.

Another way to look at this situation is that CrashPlan has a “freemium” business model. In other words, their app is free to use for local (DAS or NAS) backup or offsite backup (such as to a friend’s computer). And one thing I know is that you can’t complain about something that’s given freely to you. If it’s free, you either offer constructive criticism or you shut up about it. It’s free and the developers are under no obligation to heed your feedback or to make changes because you say so. As a matter of fact, I used CrashPlan as a free service for local backup for a couple of years before I started paying for their cloud backup service. But it was only after I started paying that I had certain expectations of performance. And in spite of those unmet expectations, I stuck with them for four years, patiently waiting for them to deliver on their promise of “no storage limits, bandwidth throttling or well-engineered excuses”… and they didn’t deliver.

Here I should also say that CrashPlan support is responsive. Even when I was using their free backup service, I could file support tickets and get answers. They always tried to resolve my issues. That’s a good thing. It’s important to point this out, because customer service is an important aspect of a business in the services industry — and online backups are a service.

About three weeks ago, I was talking with Mark Fuccio from Drobo about my issues with CrashPlan and he suggested I try Backblaze, because they truly have no throttling. So I downloaded the Backblaze app (which is a native Mac app, not a Java app), created an account and started to use their service. Lo and behold, the 15-day trial period wasn’t yet over and my backup to their servers was almost complete! I couldn’t believe it! Thank you Mark! 🙂

I optimized the Backblaze settings by allowing it to use as much of my ISP bandwidth as it needed (I have a 100 Mbps connection), and I also bumped the number of backup threads to 10, meaning the Backblaze app could initiate 10 separate instances of itself and upload all 10 instances simultaneously to their servers. I did have to put up with a slightly sluggish computer during the initial backup, but for the first time in many years, I was able to back up all of my critical data to the cloud. I find that truly amazing in and of itself.

Screen Shot 2016-10-14 at 21.36.27.png

This is what I did to optimize my Backblaze installation

As you can see from the image above, I got upload speeds over 100 Mbps when I optimized the backup settings. During most of the days of the initial upload, I actually got speeds in excess of 130 Mbps, which I think is pretty amazing given my situation: I live in Romania and the Backblaze servers are in California, so my data had to go through a lot of internet backbones and through the trans-Atlantic cables.

The short of it is that I signed up for a paid plan with Backblaze and my initial backup completed in about 20 days. Let me state that again: I backed up about 9 TB of data to Backblaze in about 20 days, and I managed to back up only about 1 TB of data to CrashPlan in about 4 years (1420 days). The difference is striking and speaks volumes about the ridiculous amount of throttling that CrashPlan puts in place for heavy users like me.

I also use CrashPlan for local network backup to my Drobo 5N, but I may switch to another app for this as well, for two reasons: it’s slow and it does a lot of maintenance on the backup set and because it doesn’t let me use Drobo shares mapped through the Drobo Dashboard app, which is a more stable way of mapping a Drobo’s network shares. CrashPlan refuses to see those shares and requires me to manually map network shares, which isn’t as stable a connection and leads to share disconnects and multiple mounts, which is something that screws up CrashPlan. I’m trying out Mac Backup Guru, which is a Mac-native app, is pretty fast and does allow me to back up to Drobo Dashboard-mapped shares. If this paragraph doesn’t make sense to you, it’s okay. You probably haven’t run into this issue. If you have, you know what I’m talking about.

Now, none of this stuff matters if you’re a typical user of cloud backup services. If you only have about 1 TB of data or less, any cloud backup service will likely work for you. You’ll be happy with CrashPlan and you’ll be happy with their customer service. But if you’re like me and you have a lot of data to back up, then a service like Backblaze that is truly throttle-free is exactly what you’ll need.

Standard
Thoughts

The value of a good backup

While working on the fifth episode of RTTE, I learned first hand the value of a good backup. The hard drive on my editing computer (my MacBook Pro) died suddenly and without warning. Thankfully, my data was backed up in two geographically different locations.

The day my hard drive died, I’d just gotten done with some file cleanups, and was getting ready to leave for a trip abroad. I shut down my computer, then realized I needed to check on a couple things, and booted it up again, only this time, it wouldn’t start. I kept getting a grey screen, meaning video was working, but it refused to boot into the OS. And I kept hearing the “click of death” as the hard drive churned. I tried booting off the Snow Leopard DVD, but that didn’t work either. I’d tested the hard drive’s SMART status just a couple of weeks before, and the utility had told me the drive had no problems whatsoever.

I had reason to worry for a couple of reasons:

  1. The laptop refused to boot up from the OS X DVD, potentially indicating other problems than a dead hard drive. I do push my laptop quite a bit as I edit photos and video, and I’d already replaced its motherboard once. I was worried I might have to spend more than I wanted to on repairs.
  2. All of the footage for the fifth episode of RTTE was on my laptop. Thankfully, it was also backed up in a couple of other places, but still, I hadn’t had reason to test those backups until now. What if I couldn’t recover it?

I had no time for further troubleshooting. I had to leave, and my laptop was useless to me. I left it home, and drove away, worried about what would happen when I returned.

A week later, I got home and tried to boot off the DVD again. No luck. I had to send it in, to make sure nothing else was wrong. In Romania, there’s only one Apple-authorized repair shop. They’re in Bucharest, and they’re called Noumax. I sent it to them for a diagnosis, and a couple of days later, I heard back from them: only the hard drive was defective, from what they could tell.

I was pressed for time. I had to edit and release the fifth episode of RTTE, and I also had to shoot some more footage for it. I didn’t have time to wait for the store to fix the laptop, so I asked them to get it back to me, while I ordered a replacement hard drive from an online store with fast, next-day delivery (eMag).

The hard drive and the laptop arrived the next day. I replaced the hard drive, using this guide, and also cleaned the motherboard and CPU fans of dust, then restored the whole system from the latest Time Machine backup. This meant that I got back everything that was on my laptop a few hours before it died.

I’d have preferred to do a clean OS install, then install the apps I needed one by one, then restore my files, especially since I hadn’t reformatted my laptop since I bought it a few years ago, but that would have been a 2-3 day job, and I just didn’t have the time. Thankfully, OS X is so stable that even a 3-year old install, during which I installed and removed many apps, still works fairly fast and doesn’t crash.

Some might say, what’s the big deal? The laptop was backed up, and you restored it… whoopee… Not so fast, grasshopper! The gravity of the situation doesn’t sink in until you realize it’s your work — YEARS of hard work — that you might have just lost because of a hardware failure. That’s when your hands begin to tremble and your throat gets dry, and a few white hairs appear instantly on your head. Even if the data’s backed up (or so you think) until your data’s restored and it’s all there, you just don’t know if you can get it back.

I’ve worked in IT for about 15 years. I’ve restored plenty of machines, desktops and servers alike. I’ve done plenty of backups. But my own computer has never gone down. I’ve never had a catastrophic hardware failure like this one until now. So even though I’ve been exposed to this kind of thing before, I just didn’t realize how painful it is until now. And I didn’t appreciate the value of a good backup until now.

So, here’s my advice to you, as if you didn’t hear it plenty of times in the past… BACK UP YOUR COMPUTER!

If you have a Mac, definitely use Time Machine. It just works. It’s beautifully simple. I’ve been backing up my laptop with Time Machine to the same reliable drive for years. It’s this little LaCie hard drive.

But the LaCie drive might fail at some point, which is why I also back up my data with CrashPlan. For this second backup, I also send my data to a geographically-different location. Since we live in Romania these days, I back up to my parents’ house in the US, where the backup gets stored on a Drobo. And the backup is also encrypted automatically by CrashPlan, which means it can’t be intercepted along the way.

It’s because of my obsessive-compulsive backup strategy that I was able to recover so quickly from the hardware failure. Thankfully, these days backups are made so easy by software like Time Machine and CrashPlan that anyone can keep their work safe. So please, back up your data, and do it often!

One more thing. You know the old saying, every cloud has a silver lining? It was true in my case. When I ordered the new drive for my laptop, I was able to upgrade from its existing 250GB SATA hard drive with an 8MB buffer and 5400 rpm to a spacious 750GB SATA hard drive with a 32MB buffer and 7200 rpm, which means my laptop now churns along a little faster, and has a lot more room for the 1080p footage of my shows. 🙂

Standard
Reviews

An update on CrashPlan

Updated 11/01/16: I’ve revised my opinion of CrashPlan. See here for the details.

Back in April, I wrote about CrashPlan, a wonderful, multi-platform piece of software that lets you back up to friends’ computers for free. I said that I used it to do trans-Atlantic backups, from my computer in Romania to my parents’ computer back in the USA.

It’s been a while since then, so I thought I’d give you a quick update. I’m still using it, and I still like it. It works.

When I wrote the other post, I mentioned I kept hitting some bandwidth ceiling somewhere along the line between Romania and the US, around 2 Mbps. Somehow, that ceiling has since disappeared. I’m now getting speeds up to 25 Mbps, though it’s usually around 5-7 Mbps.

So if you’re in need of a way to back up to a remote location, inexpensively, then CrashPlan is the way to do it. If you don’t have a friend who’s willing to help, that’s okay, you can back up to the CrashPlan servers for $4.50/month, or you can get the family plan, which covers all of your household computers, for $8.33/month. That’s very reasonable, and it’s definitely worth it.

Standard
Thoughts

Metadata: DNG vs RAW

Generally speaking, I prefer the Adobe DNG format over the proprietary RAW format given to me by a camera, because I like the fact that it’s more or less future-proof. With a DNG file, the meta-data resides inside the file — like with a JPG — but the format is lossless, same as a RAW file — and unlike a JPG.

In spite of the fact that it’s a “publicly available archive format”, I would like to see more camera manufacturers adopt it, so I can feel more comfortable using it. I realize companies like Hasselblad and Leica have already adopted it, and you can take photos directly in DNG format on some of their cameras, but until the big camera manufacturers like Canon and Nikon adopt it, it won’t have the mass acceptance it needs to ensure its long-term survivability.

Still, I have begun to convert the RAW files in my photo library to DNG. By my count, I have converted about 30% of my 77,000 photographs to DNG format, and I am converting more of them every day. Let’s hope Adobe sticks to its word in the future and I’m not left holding the bag, having locked my photos into a format that might become obsolete.

Long-term benefits and potential caveats aside, I should point out a more current disadvantage between DNG and RAW. It has to do with metadata.

Yes, it’s true that with a RAW file, you’re stuck working with your metadata in sidecar XMP file, and that file may get corrupted, or you may lose it, thus losing your metadata and the processing directives for Camera Raw or Lightroom or whatever you’re using to process your photos. With a DNG, everything resides inside the file. There’s no XMP file, which is a good thing, most of the time.

But when you’re backing up your library, and let’s say for the sake of the argument that you’ve got to back up 20,000 photos, which is what I’m doing right now, and you’ve made minute changes to the metadata of all those files — only changed one EXIF or IPTC field — the backup software won’t care. You’ll have to back up 20,000 DNG files, each (in my case) between 12-24 MB. That’s going to take a LOT longer than backing up 20,000 XMP sidecar files, each of which is only 15-25 KB, because those are the only files that will have been changed if I update the EXIF or IPTC data for a whole bunch of RAW files.

That’s one area where RAW trumps DNG. I’m willing to overlook it if DNG will indeed prove to be a future-proof format, but that remains to be seen.

Standard
Reviews

CrashPlan works for transatlantic backups

Updated 11/01/16: I’ve revised my opinion of CrashPlan. See here for the details.

Last week, I wrote an article called “What’s On Your Drobo“, and in it, I mentioned that I was going to try to use an app called CrashPlan to do backups from my photo library in Romania to my backup location in the US. I’m happy to say that it works as expected, and no, this isn’t an April Fool’s joke. Here’s a screenshot of an active backup. At the time, I was getting 2.7 Mbps throughput.

There is a bandwidth bottleneck somewhere, though I’m not sure where it is. My broadband connection in Romania sits at 30 Mbps up and down, as I mentioned here, and my parents’ broadband connection clocks in around 16 Mbps down and 4 Mbps up. Theoretically, since I’m uploading and they’re downloading, I should be getting at least 15 Mbps, but I’m not. So it looks like there’s either a bottleneck as my data exits Romania, or as it goes through the transatlantic fiber optic cables. If someone can chime in on this, I’d love to find out more. I do know that I hit that same 2.5 Mbps ceiling as I upload to SmugMug, YouTube and blip.tv.

Bottlenecks aside, I’m just happy I can do off-site backups, and at least given my current setup, it’s free! CrashPlan works as advertised! I have to admit I was a skeptic when I downloaded it and installed it. I figured it would work on the local network, which is where I did the initial backups, but it would surely run into some firewall issues when I tried it from another location. Nightmares of re-configuring my parents’ firewall remotely flashed before my eyes… Amazingly enough, I didn’t have to do any of that! It just works!

So, if you’re interested in doing this sort of thing, download CrashPlan (it’s multi-platform), install it on both computers where you want to use it, configure it (use the help section), test it, then let it do its thing!

One thing I need to mention is that if one of the computers falls asleep, the backup will be paused until it wakes up. Even though I set my parents’ iMac to wake up for network traffic, CrashPlan doesn’t seem to be able to wake it up when I try to start the backup from my end. Keep that in mind and plan your backups accordingly.

Standard