The value of a good backup

While working on the fifth episode of RTTE, I learned first hand the value of a good backup. The hard drive on my editing computer (my MacBook Pro) died suddenly and without warning. Thankfully, my data was backed up in two geographically different locations.

The day my hard drive died, I’d just gotten done with some file cleanups, and was getting ready to leave for a trip abroad. I shut down my computer, then realized I needed to check on a couple things, and booted it up again, only this time, it wouldn’t start. I kept getting a grey screen, meaning video was working, but it refused to boot into the OS. And I kept hearing the “click of death” as the hard drive churned. I tried booting off the Snow Leopard DVD, but that didn’t work either. I’d tested the hard drive’s SMART status just a couple of weeks before, and the utility had told me the drive had no problems whatsoever.

I had reason to worry for a couple of reasons:

  1. The laptop refused to boot up from the OS X DVD, potentially indicating other problems than a dead hard drive. I do push my laptop quite a bit as I edit photos and video, and I’d already replaced its motherboard once. I was worried I might have to spend more than I wanted to on repairs.
  2. All of the footage for the fifth episode of RTTE was on my laptop. Thankfully, it was also backed up in a couple of other places, but still, I hadn’t had reason to test those backups until now. What if I couldn’t recover it?

I had no time for further troubleshooting. I had to leave, and my laptop was useless to me. I left it home, and drove away, worried about what would happen when I returned.

A week later, I got home and tried to boot off the DVD again. No luck. I had to send it in, to make sure nothing else was wrong. In Romania, there’s only one Apple-authorized repair shop. They’re in Bucharest, and they’re called Noumax. I sent it to them for a diagnosis, and a couple of days later, I heard back from them: only the hard drive was defective, from what they could tell.

I was pressed for time. I had to edit and release the fifth episode of RTTE, and I also had to shoot some more footage for it. I didn’t have time to wait for the store to fix the laptop, so I asked them to get it back to me, while I ordered a replacement hard drive from an online store with fast, next-day delivery (eMag).

The hard drive and the laptop arrived the next day. I replaced the hard drive, using this guide, and also cleaned the motherboard and CPU fans of dust, then restored the whole system from the latest Time Machine backup. This meant that I got back everything that was on my laptop a few hours before it died.

I’d have preferred to do a clean OS install, then install the apps I needed one by one, then restore my files, especially since I hadn’t reformatted my laptop since I bought it a few years ago, but that would have been a 2-3 day job, and I just didn’t have the time. Thankfully, OS X is so stable that even a 3-year old install, during which I installed and removed many apps, still works fairly fast and doesn’t crash.

Some might say, what’s the big deal? The laptop was backed up, and you restored it… whoopee… Not so fast, grasshopper! The gravity of the situation doesn’t sink in until you realize it’s your work — YEARS of hard work — that you might have just lost because of a hardware failure. That’s when your hands begin to tremble and your throat gets dry, and a few white hairs appear instantly on your head. Even if the data’s backed up (or so you think) until your data’s restored and it’s all there, you just don’t know if you can get it back.

I’ve worked in IT for about 15 years. I’ve restored plenty of machines, desktops and servers alike. I’ve done plenty of backups. But my own computer has never gone down. I’ve never had a catastrophic hardware failure like this one until now. So even though I’ve been exposed to this kind of thing before, I just didn’t realize how painful it is until now. And I didn’t appreciate the value of a good backup until now.

So, here’s my advice to you, as if you didn’t hear it plenty of times in the past… BACK UP YOUR COMPUTER!

If you have a Mac, definitely use Time Machine. It just works. It’s beautifully simple. I’ve been backing up my laptop with Time Machine to the same reliable drive for years. It’s this little LaCie hard drive.

But the LaCie drive might fail at some point, which is why I also back up my data with CrashPlan. For this second backup, I also send my data to a geographically-different location. Since we live in Romania these days, I back up to my parents’ house in the US, where the backup gets stored on a Drobo. And the backup is also encrypted automatically by CrashPlan, which means it can’t be intercepted along the way.

It’s because of my obsessive-compulsive backup strategy that I was able to recover so quickly from the hardware failure. Thankfully, these days backups are made so easy by software like Time Machine and CrashPlan that anyone can keep their work safe. So please, back up your data, and do it often!

One more thing. You know the old saying, every cloud has a silver lining? It was true in my case. When I ordered the new drive for my laptop, I was able to upgrade from its existing 250GB SATA hard drive with an 8MB buffer and 5400 rpm to a spacious 750GB SATA hard drive with a 32MB buffer and 7200 rpm, which means my laptop now churns along a little faster, and has a lot more room for the 1080p footage of my shows. :-)

Hardware preview: Apple iPad 2

Apple iPad 2

The new iPad 2 will become available on March 11th (see a neat video from Apple introducing it). In my post about the original iPad, I said the following:

“In spite of the failures of their predecessors, I think Apple will pull this off. I think the iPad will be very successful.”

I’m glad to see that I was right. Not that I had anything to do with the success of the device. The credit goes entirely to Apple, and to the people who bought it and used it so well.

I got to watch the March 2 keynote today (a few days later). Much to my surprise, Steve Jobs was on stage to present it. I was very glad to see him able to stand up and hold a meeting, given all the tabloid rumors about him — though I have to say he was skinny as a board. Thank goodness he’s still around. I hope he gets better, and continues to be around for a few more decades.

Here’s a quick summary of the salient features of the iPad 2:

  • 33% thinner than the original iPad (0.34″), and lighter (1.33 lbs)
  • Comes in both black and white finishes
  • Dual-core A5 chip, up to 2x faster
  • Graphics are up to 9x faster
  • Same great 10-hour battery life
  • Same 1024×768 display
  • 2 cameras (front and back) for video or photos, in HD
  • Magnetic smart cover designed specifically for it
  • Instant on
  • AirPlay to your TV via Apple TV
  • Video mirroring (up to 1080p) via $39 cable
  • AirPrint

My only disappointment with the iPad 2 is that it doesn’t have a Retina Display. Word on the grapevine is that they’re still difficult to make in that size. Who knows… It would have been nice if this iPad had it. Still, I believe iPad 3 will have a Retina Display.

iPad 2 Measurements

iPad 2 A5 Chip

I am however very glad to see that the iPad 2 does have a video camera — and not just one, but two. In my review of the original iPad, I said this:

“It’s very likely the next gen iPad will have a video camera, and it will have iChat as well.”

Glad to see I was right on that count as well. It was, after all, a logical step.

Facetime on iPad 2

iPad 2 Smart Cover Line-up

Smart Cover for iPad 2

There are some new accessories for the iPad 2, which will be offered in addition to the ones designed for the original iPad.

  • The Smart Cover, naturally, which comes in 10 colors, 5 of them polyurethane and 5 leather, as seen above (see the Smart Cover in action in this video)
  • The Digital AV Adapter provides an HDMI-out port with video and audio routed to it, in addition to a 30-pin connector which lets you charge the iPad while playing content to an HDTV
  • The iPad 2 Dock is designed for the thinner iPad 2, and also works with Digital AV Adapter

iPad 2 Dock

Of course, given that the iPad has Bluetooth, you can stick it in a dock and use the Apple Wireless Keyboard to type on it.

Apple Wireless Keyboard

The Smart Cover is so nicely designed.

iPad 2 Smart Cover (1)

iPad 2 Smart Cover (2)

iPad 2 Smart Cover (3)

The Digital AV Adapter will make it so easy to display content from the iPad on an HDTV.

iPad 2 Digital AV Adapter

iPad 2 Video Mirroring

iPad 2 Airplay

The iPad 2 will come in 16GB, 32GB and 64GB models, as well as WiFi-only or WiFi+3G (GSM or CDMA) models. My guess is that iPad 3 will have a combined 4G GSM/CDMA chip, eliminating the need to offer separate 3G models. The pricing grid for the various models (there are 18 possible models, given that there are two color finishes and two 3G providers), goes as follows:

  • WiFi-only: $499/16GB, $599/32GB, $699/64GB (black or white finish)
  • WiFi + 3G: $629/16GB, $729/32GB, $829/64GB (black or white finish, AT&T/Verizon)

Images used courtesy of Apple Computer, Inc.

What’s next in data storage?

My recent musings on high definition and the state of the technology behind it have spurred me to think about data storage (not that it’s a new subject for me). But so far, I’ve commented only on what’s already been developed, and didn’t take the time to think about what’s next.

What’s the motivation behind this post? It’s simple. For Ligia’s Kitchen, it costs me about 10.5 GB for 5 minutes of final, edited footage of show, with a one-camera setup. What goes into the 10.5GB? There’s the raw footage (and sound files, if I use a standalone mic), the edits, and the final, published footage. When I use two cameras, the space needed can easily go up by 1.5-2.5x, depending on the shots I need to get. I shoot and edit in 1080p, and output to 720p.

My storage needs are okay for now. I’ve got plenty of space, and if I keep going at this rate, I should be fine. But… and there’s always a but, isn’t there… I have more show ideas in mind. And there’s the hypothetical possibility of shooting with a RED camera at some point in the future, if certain factors come together to allow it. So I’m thinking ahead.

Current hard drive technology (bits of data on disks) has certainly come a long way. Those of us who’ve been in the business long enough know what prices used to be like for capacities that are laughable by today’s standards. Back in 1999, I paid $275 for a 27GB hard drive. My laptop’s drive in college could store a grand total of 120MB. And when I began to learn programming, I’d load the code into memory from tape…

I remember being really excited about Hitachi’s new Perpendicular Magnetic Recording Technology, which came out in early 2006. They even had an animation on their website, which they’ve taken down since. That technology is behind all of the new hard drives that are on the market today, by the way. Hitachi came up with a way to get the bits of data to stand up (hence the term perpendicular) instead of lying down on hard drive platters, thus doubling the amount of data that could be stored onto them.

There are two roads ahead when it comes to data storage, of which one is more likely to succeed:

  • Optical storage (this is probably the future of storage)
  • Biological storage

Let’s first look at biological storage. One particular article made the rounds lately: researchers at the Chinese University in Hong Kong have managed to store 90GB of data in 1g of bacteria. While it sounds exciting, the idea of storing my data in petri dishes on my desk doesn’t readily appeal to me, and certain complications come up:

  • 1g of bacteria is about 10 million cells (that’s a LOT); one must start thinking about the potential for bio hazards when you work with bacteria.
  • The data is stored in a bacteria’s DNA, which means it’s encrypted (a good thing), but it’s also subject to significant mutation (a bad thing) and it takes a long time to retrieve it because you need a gene sequencer, which is tedious and expensive (a bad thing).

I’m not against this. Hey, if they can make it safe and fast, okay. But I believe this is going to be relegated to special applications. The article suggests the technique is currently used to store copyright information for newly created organisms (I wonder how many new bacteria researchers as a whole have created, and is it any wonder antibiotics have such a hard time working against them when we keep playing God). I also see this sort of data storage as a way for spies to operate, or for governments to keep certain secrets.

Okay, onto more cheery stuff, like optical storage. I’ve always thought there was massive potential here, and am glad to see significant work has already been done to make this a reality. There are two technologies which are feasible, according to research that’s already been done:

  • HDSS (Holographic Data Storage Systems), which so far can store up to 1TB of data in a crystal the size of a sugar cube, but doesn’t yet allow rewrites
  • 3D optical data storage, which so far can store up to 1TB of data onto a 1.2mm thick optical disc

These developments are very encouraging. Optical storage is safe, and its potential capacities are huge, possibly endless. And when you think about computer hardware, and how manufacturers are looking at using optical technology in the bridges and buses and wires inside the hardware, because it’s incredibly fast, you start to see how optical makes sense. Let’s also not forget fiber optic cabling, and its incredible capacity to carry data. It certainly looks like optical is the future!

So what’s going to happen to the standard 3.5″ form factor of today’s hard drives? Well, it’s likely that it will stay the same, even though it the storage technology inside it might change. We’ll have crystals and lasers instead of platters and heads, but they’ll likely be able to fit them in there somehow. I don’t think we’ll need to start keeping crystal libraries on our desks, like in Superman’s Crystal Cave, and sticking various-sized crystals into our computers any time soon, although it did look pretty cool when Christopher Reeve did it in the movie.

It really all depends on how soon this new technology will come to market. Right now, there’s clearly enough vested interest in the 3.5″ and 2.5″ form factors to motivate drive manufacturers to shoehorn the new technologies into those shapes, but if optical hard drives won’t be here for the next 5-10 years, then it’s possible that the form factor will change as well. We are after all moving to smaller, sleeker shapes for most computers, notebooks and desktops alike.

What comes after High Definition?

Producing (set design, lighting, filming, directing, editing) my wife’s cooking show has gotten me thinking about what comes after HD, because there obviously is a large discrepancy in resolution between full 1080p HD and properly exposed 35mm film (up to 3500p) — as I already mentioned in my post on preserving classic movies.

Yes, high definition is a huge improvement over standard definition, which in turn was a large improvement over early television signals. But televisions and VCRs, in spite of their popularity, are a dismal failure in picture quality compared to what they replaced: film reels and projectors.

Nowadays, we’ve gained some foothold back when it comes to consumer/prosumer video quality. We have ready access to video cameras that will record in HD (at various qualities, given the model and the price), and we have newer computers and televisions that will allow us to play back those videos at their native (720p or 1080p) resolutions. Even websites have begun in recent years to allow us to play back HD videos, and the quality of broadband internet connections has increased to the point where one doesn’t have to wait a half hour or more in order to download/buffer an HD video and play it properly on their computer. We can even play back HD videos from the internet directly on our televisions, thanks to standalone or built-in media players.

But if we’re to get back to the quality of 35mm film and best it, we must keep moving forward. Thankfully, some visionaries have already taken the first steps and have come up with a camera that can record at a similar-to-film resolution: the RED One, which can give us 2300p of extremely high definition digital video. It’s not quite 3000p or 3500p (which would be the equivalent of properly exposed film), but it gets us pretty close, and it’s certainly much better than 1080p.

The RED camera captures each frame of video as a 12-bit RAW image, which means we, as videographers, have much greater freedom than before when editing the video, just like photographers do when they switch from JPG to RAW files. All of a sudden, white balance, exposure, recovery, blacks, vibrance, saturation, and tone adjustments can be made with much more accuracy.

One area where I’d love to see more improvement — although I’m sure it’ll come with time — is RED’s ability to capture more color depth, say 14-bit or 16-bit. Bit depth is still an area where improvement can be made across the board when it comes to digital cameras.

But let’s leave tech specs alone, and think about how we can edit and enjoy the videos we could make with a RED camera. That’s where difficulties come in, because you see, we still can’t properly do that, certainly not with consumer, and not even with prosumer equipment. No, we’d be looking at professional equipment and serious prices. The market just hasn’t caught up.

There are no computers that can display that kind of resolution at full screen, and there are no televisions that can do it, either. TVs and computers are still caught up in the world of 720p and 1080p. And to make things even more complicated, now we’ve got to worry about 3D video, which is nice for some applications, but from my point of view, it’s a distraction, because it adds yet another barrier, another detour, on the road to achieving proper video resolution across the board. Manufacturers, TV stations and filmmakers are jumping on the 3D bandwagon, when they should be worried about resolution.

So, what costs would a filmmaker be looking at if he or she wanted to shoot at the highest possible digital resolution available today (a RED setup)? I crunched some numbers, and mind you, these are just approximations. The costs are likely to be 1.5-2x that much when you account for everything you might need. On a side note, the folks at RED and at Final Cut Pro have worked together quite a bit to ensure that we can edit RED video natively, directly in Final Cut Pro, on a Mac. See this video for an overview.

  • RED One camera: $25,000
  • 35mm RED lens: $4,250
  • 18-85mm RED lens: $9,975
  • RED LCD: $2,500
  • RED CF media and cards: $1,500
  • RED rig: about $2,500
  • add extra $$$ for power, accessories, tripods, other media, etc.
  • RED video card, for encoding and editing video: $4,750
  • Mac Pro editing station: about $7,000-$12,000, depending on your needs, and you may need more than one of these, depending on how big your production is
  • 30″ display: about $1,000-$3,000, depending on your needs, and you may need more than one of these as well, depending on the number of workstations and your display setup
  • Final Cut Studio software: $1,000
  • HDD-based storage for editing and archival: $2,000-$20,000, depending on your needs
  • LTO tape or additional HDD-based storage for backup: costs will vary quite a bit here
  • Specialized cinema hardware and display for showing movies at full resolution: I have no idea what this costs, but it’s likely to go into the hundreds of thousands of dollars, and not every cinema has it

So at a minimum, we’d be talking about an investment of more than $60,000 in order to work with a RED setup today.

But let’s not get tied up in talking solely about RED cameras. Clearly the entire industry needs to take steps in order to ensure that videos at resolutions greater than 1080p HD can be played across all the usual devices. Unfortunately, they’re still tied up in SD and HD video. Most TV channels still transmit in SD or lower-than-SD video quality (lower than 480p). It’s true, most have always transmitted at broadcast quality (500p or better) but we’ve always had to contend with a lot of signal loss. And nowadays, we still have to pay extra for HD channels, even though they should be the norm, and we should be looking forward.

To that effect, computer displays need to get bigger and better, computer hardware needs to get faster, computer storage needs to expand, media players need to increase their processing power, televisions need to get better and bigger, and broadband internet needs to get faster, ideally around the gigabit range (see this talk from Vinton Cerf on that subject), so that full resolution, 4000K video can move across the internet easily.

For now, if I were to start working on RED, I’d still have to output to 720p or 1080p and keep my full resolution originals archived for another day, somewhere in the future, when consumer-grade electronics have evolved to the point where they can play my videos and films natively.

I for one look forward to the day when YouTube starts to stream 3500p videos, and when we can all play them conveniently and at full resolution on our computers and televisions!

Hardware preview: the ioSafe Rugged Portable

I’m a fan of ioSafe‘s rugged hard drives, which you can trust with your data through very rough conditions. I was glad to write about the ioSafe Solo (their first product), and the ioSafe SSD, because no other company on the market offered such unparalleled protection from destruction.

Now they’re launching a new model, the ioSafe Rugged Portable — their first drive made for travel.

As you can see, they’re using a new design, with a machined Aluminum or Titanium enclosure that can withstand up to 5,000 lbs (Ti) or 2,500 lbs of pressure (Al). The drive is suspended on all six axes of motion, and can withstand a drop from 20′ (SSD version) or 10′ (HDD version).

Of course, the drive can still withstand immersion in water — up to 10′ (Al) or 30′ (Ti), both up to 3 days in duration. And it’s got a whole other bunch of protections built in as well:

  • ChemSafeTM Technology – Full immersion in diesel fuel, oils, hydraulic fluids, aircraft fuels, 12” depth for 1 hour with no data loss per MIL-STD-810G Method 504
  • EnviroSafeTM Technology – Continuous exposure to UV, blowing sand, blowing dust, rain, salt fog, icing or freezing rain, 24 hours with no data loss per MIL-STD-810G Methods 505.4, 506.4, 509.4 and 510
  • AltiSafeTM Technology – High altitude operation. 15K ft. (Alum.) and 30K ft. (SSD and Ti.) rated altitudes per MIL-STD-810G Method 500.4
  • Theft Resistant Kensington® Lock compatible slot solid metal construction – theft protection

As you’ve no doubt gathered so far, there are multiple flavors of the drive, with HDDs or SSDs inside, and Aluminum or Titanium enclosures. And it ships with USB 3.0, FW800 and USB 2.0 connections — your choice.

Possibly the best feature — given the drive’s title — is its weight, which comes to 1 lbs for the Al enclosure and 1.5 lbs for the Ti enclosure.

Drobo S gets USB 3.0 upgrade

Drobo S, the 5-bay, 3-interface storage array from Data Robotics got a neat upgrade to USB 3.0 today. Instead of USB 2.0, it will now have eSATA, FW800 and USB 3.0 ports on its back. This is a hardware refresh, which means the USB 2.0 model has been phased out. You should expect it to be “up to 50% faster than FW800” when using the USB 3.0 port, according to Data Robotics.

During the press briefing, I asked Mark Fuccio (Data Robotics) whether the 4-bay Firewire Drobo will get a USB 3.0 upgrade as well. He said they have no plans to upgrade it. It will stay as is. Apparently its bandwidth maxes out at FW800 speeds, and adding USB 3.0 to it would be superfluous. Data Robotics recommends that folks get the Drobo S if they want heavy-duty performance.

In other Drobo news, Drobo Sync, the nifty disaster recovery software that was announced for the DroboPro FS, is now available. If you were wondering how Data Robotics is doing these days, they’re growing. And HP has just begun selling Drobos, which is great for them and useful for us. You’ll be able to go online and get a performance HP desktop to help you zoom through your photo or video processing, and also get a Drobo S to help you safely store all those terabytes of data.

The upgraded Drobo S will be available from all the usual retailers like Amazon and B&H Photo, and of course, HP Small Business Direct.

Hardware preview: DroboPro FS

Today, the DroboPro FS will be announced by Data Robotics. It’s the bigger brother of the Drobo FS, and it’s aimed at creative professionals, SOHOs and SMBs.

There was a gap in Drobo’s file sharing lineup. They had a NAS device, sure (the Drobo FS), but it didn’t have the capacity and the bandwidth of the DroboPro FS, which has two Gigabit Ethernet ports and eight drive bays.

Its ethernet ports can be configured two ways:

  1. On two different subnets, for offsite file replication, and
  2. Failover mode, where one port takes over if the other one dies

The really nice part here is the DroboPro FS works seamlessly with a new piece of software (to be introduced shortly) by Data Robotics, called Drobo Sync. The way it works makes the DroboPro FS the perfect candidate for offsite backups, for disaster recovery purposes.

Say you have two DroboPro FS units. You keep the first one in your company’s server room, do your internal backups to it, over the local network, and then run a quick DroboPro FS to DroboPro FS initial sync job using Drobo Sync. When that’s done, you can take the 2nd DroboPro FS unit to your disaster recovery site, plug it in over there, and Drobo Sync will start doing incremental backups automatically, saving you bandwidth and validating each file on a block-by-block level to make sure the data is identical on both Drobos. This is a solution that would be perfect for overworked sysadmins, because the setup and administration are easy, and the total cost is less than that of comparative solutions.

It uses the same form factor as the DroboPro and the DroboElite, which means it shares the same measures, and the same rack-mount kit. And you can also customize it with Drobo Apps, just like the Drobo FS.

A base DroboPro FS costs around $1,999, but you can buy it bundled with hard drive sets (8TB, 16TB and 32TB). It’s available directly from Data Robotics, or from B&H Photo, Amazon and other retailers.