Permanent data storage

We need to focus our efforts on finding more permanent ways to store data. What we have now is inadequate. Hard drives are susceptible to failure, data corruption and data erasure (see effects of EM pulses for example). CDs and DVDs become unreadable after several years and archival-quality optical media also stops working after 10-15 years, not to mention that the hardware itself that reads and writes to media changes so fast that media written in the past may become unreadable in the future simply because there’s nothing to read it anymore. I don’t think digital bits and codecs are a future-proof solution, but I do think imagery (stills or sequences of stills) and text are the way to go. It’s the way past cultures and civilizations have passed on their knowledge. However, we need to move past pictographs on cave walls and cuneiform writing on stone tablets. Our data storage needs are quite large and we need systems that can accommodate these requirements.

We need to be able to read/write data to permanent media that stores it for hundreds, thousands and even tens of thousands of years, so that we don’t lose our collective knowledge, so that future generations can benefit from all our discoveries, study us, find out what worked and what didn’t.

We need to find ways to store our knowledge permanently in ways that can be easily accessed and read in the future. We need to start thinking long-term when it comes to inventing and marketing data storage devices. I hope this post spurs you on to do some thinking of your own about this topic. Who knows what you might invent?

Advertisements

How to create a Fusion Drive on a mid-2011 iMac

Yes, you can enable Fusion Drive on older Macs. I’m not sure how this method will work with Macs older than 2011, but I know for sure that it works on mid-2011 iMacs, and quite possibly on other Macs made since then. I have just completed this process for my iMac and I thought it would help you if I detailed it here.

I like Fusion Drive because it’s simple and automated, like Time Machine. Some geekier Mac users will likely prefer to install an SSD and manually separate the system and app files from the user files which take up the most space, which is something that gives them more control over what works faster and what doesn’t, but that’s a more involved process. Fusion Drive works automatically once you set it up, moving the files that are used more often onto the SSD and keeping the ones that are accessed less often on the hard drive. This results in a big performance increase without having to fiddle with bash commands too much.

The hardware

My machine is a 27″ mid-2011 iMac with a 3.4 GHz processor and 16GB of RAM. I bought it with a 1TB hard drive, which I recently considered upgrading to a 3TB hard drive but decided against, given the fan control issues with the temperature sensor and the special connector used on the factory drive.

imac-basic-specs

I purchased a 128GB Vertex4 SSD from OCZ. It’s a SATA III (6 Gbps) drive and when I look in System Info, my iMac sees it as such and is able to communicate with it at 6 Gbps, which is really nice.

ocz-vertex4-ssd-128gb

ssd-specs

The hardware installation is somewhat involved, as you will need to not only open the iMac but also remove most of the connections and also unseat the motherboard so you can get at the SATA III connector on its back. You will also need a special SATA wire, which is sold as a kit from both OWC and iFixit. The kit includes the suction cups used to remove the screen (held into place with magnets) and a screwdriver set.

2nd-drive-ssd-kit

You can choose to do the installation yourself if you are so inclined, but realize that you may void the warranty on the original hard drive if something goes wrong, and this is according to Apple Tech Support, with whom I checked prior to ordering the kit. Here are a couple of videos that show you how to do this:

In my case, it just so happened that my iMac needed to go in for service (the video card, SuperDrive and display went bad) and while I had it in there, I asked the technicians to install the SSD behind the optical drive for me. This way, my warranty stayed intact. When I got my iMac back home, all I had to do was to format both the original hard drive and the SSD and proceed with enabling the Fusion Drive (make sure to back up thoroughly first). You can opt to do the same, or you can send your computer into OWC for their Turnkey Program, where you can elect to soup it up even more.

The software

Once I had backed up everything thoroughly through Time Machine, I used the instructions in this Macworld article to proceed. There are other articles that describe the same method, and the first man to realize this was doable and blog about it was Patrick Stein, so he definitely deserves a hat tip. I’ll reproduce the steps I used here; feel free to also consult the original articles.

1. Create a Mountain Lion (10.8.2) bootup disk. Use an 8GB or 16GB stick for this, it will allow you to reformat everything on the computer, just to clean things up. Otherwise you may end up with two recovery partitions when you’re done. I used the instructions in this Cult of Mac post to do so. The process involves re-downloading 10.8.2 from the Apple Store (if you haven’t bought it yet, now is the time to do so) and an app called Lion Diskmaker.

2. Format both the original HD and the SSD, just to make sure they’re clean and ready to go. Use Disk Utility to do this, or if you’re more comfortable with the command line, you can also do that (just be aware you can blow away active partitions with it if you’re not careful).

2. List the drives so you can get their correct names. In my case, they were /dev/disk1 and /dev/disk2.

diskutil list

3. Create the Fusion Drive logical volume group. When this completes, you’ll get something called a Core Storage LGV UUID. Copy that number, you’ll need it for the following step.

diskutil coreStorage create myFusionDrive /dev/disk1 /dev/disk2

4. Create the Fusion Drive logical volume. I used the following command:

diskutil coreStorage createVolume paste-lgv-uuid-here jhfs+ "Macintosh HD" 100%

5. Quit Terminal and begin a fresh install of Mountain Lion onto the new disk called “Macintosh HD”.

6. Restore your apps, files and system settings from the Time Machine backup using the Migration Assistant once you’ve booted up. Here’s an article that shows you how to do that. When that completes, you’re done!

The result

Was it worth it? Yes. The boot-up time went from 45-60 seconds to 15 seconds, right away. And over time, the apps and files I use most often will be moved onto the SSD, thus decreasing the amount of time it’ll take to open and save them.

At some point, I expect Apple to issue a utility, like Boot Camp, that will allow us to do this more easily and automatically. Until then, that’s how I set up Fusion Drive on my iMac, and I hope it’s been helpful to you!

Hardware preview: ioSafe N2 NAS

ioSafe, the company famous for its line of rugged external drives that can withstand disasters such as floods, fires and even crushing weight, has come up with a new product: the N2 NAS (Network Attached Storage) device.

The N2 device comes at the right time. The market for NAS devices is maturing and demand is growing. Western Digital has even come out with a line of hard drives, the WD Red, specifically targeted to NAS enclosures. To my knowledge there is no such other NAS device out there, so ioSafe’s got the lead on this.

The N2 appliance is powered by Synology® DiskStation Manager (DSM) and is aimed at the SOHO, SMB and Remote Office Branch Office (ROBO) markets.

The high performance 2-bay N2 provides up to 8TB of storage capacity and is equipped with a 2GHz Marvel CPU and 512MB of memory. The N2 uses redundant hard drives as well as ioSafe’s patented DataCast, HydroSafe and FloSafe technologies to protect data from loss in fire up to 1550°F and submersion in fresh or salt water up to a 10 foot depth for 3 days.

Features:

  • Local and Remote File Sharing: Between virtually any device from any location online
  • Cloud Station: File syncing between multiple computers and N2 (like Dropbox)
  • iTunes Server
  • Surveillance Station: Video surveillance application
  • Media Server: Stream videos and music
  • Photo Sharing: Photo sharing with friends and family
  • Mail Server: Email server
  • VPN Server: Manage Virtual Private Network
  • Download Station: Post files for others to download
  • Audio Station: Stream audio to smartphone (iOS/Android)
  • FTP Server: Remote file transfers
  • Multi-platform compatibility with Mac/PC/MS Server/Linux

Hardware:

  • Dual Redundant Disk, RAID 0/1, Up to 8TB (4TB x 2)
  • 2GHz Marvel CPU and 512MB memory
  • Gigabit Ethernet Port
  • Additional ports for USB 3, SD Memory Card
  • User replaceable drives
  • Protects Data From Fire: DataCast Technology. 1550°F, 1/2 hr per ASTM E119 with no data loss.
  • Protects Data From Flood: HydroSafe Technology. Full immersion, 10 ft. 3 days with no data loss.
  • FloSafe Vent Technology: Active air cooling during normal operation. FloSafe Vents automatically block destructive heat during fire by water vaporization – no moving parts
  • Physical theft protection (optional floor mount, padlock door security – coming Q1 2013)
  • Kensington® Lock Compatible

Support and Data Recovery Service (DRS):

  • 1 Year No-Hassle Warranty (for N2 Diskless)
  • 1 Year No-Hassle Warranty + Data Recovery Service (DRS) Standard (for loaded N2)
  • DRS included $2500/TB for forensic recovery costs for any reason if required
  • DRS and Warranty are upgradeable to 5 years ($.99/TB per month)
  • DRS Pro available includes $5000/TB + coverage of attached server ($2.99/TB per month)

Operating Environment:

  • Operating: 0-35° C (95°F)
  • Non-operating: 0-1550°F, 1/2 hr per ASTM E119
  • Operating Humidity: 20% – 80% (non-condensing)
  • Non-operating Humidity: 100%, Full immersion, 10 feet, 3 days, fresh or salt water

Physical:

  • Size: 5.9″W x 9.0″H x 11.5″L
  • Weight: 28 lbs

The N2 appliance is being brought to market with funding obtained through IndieGogo. I know it’s hard to believe it when you look at their products, but ioSafe only has about 20 employees. Sometimes they have to be creative in the ways they fund their R&D.

The ioSafe N2 will begin shipping in January 2013 and will be available in capacities up to 8TB. Introductory pricing for the ioSafe N2 diskless version is available for $499 on Indiegogo ($100 off the retail price of $599.99) if you want to get your own hard drives.

I’ve also written about ioSafe Solo, the ioSafe Rugged Portable and the ioSafe SSD devices.

Mac Pro line overdue for a hardware refresh

Because of the recent hardware failure on my MacBook Pro, I started to think about getting a new computer.

Given my intensive computing needs, I naturally looked toward the Mac Pro, but I was disappointed to find that its specs are lagging behind the times. The differences between it and what other Apple computers offer are enough for me to hold off on making the purchase.

The areas where it needs to improve can (almost) be summarized by this screenshot:

Thunderbolt

The iMac already has Thunderbolt ports, but the Mac Pro (the top-of-the-line model) doesn’t. Why isn’t it there already?

If Thunderbolt is “the fastest, most versatile I/O in a desktop”, why is it MIA on the Mac Pro? Shouldn’t it be on Apple’s most powerful desktop? If it’s got 10 Gbps data channels, and it’s tens of times faster than FW800 or USB 2.0, why isn’t it on the Mac Pro?

It even made it onto the MBP… Will Thunderbolt make it onto the MacBook before it makes it onto the Mac Pro?

SATA 6GB/S

The Mac Pro’s internal architecture can support SATA 6Gb/s speeds. There are already SATA 6Gb/s hard drives on the market, at very affordable prices. Why doesn’t the Mac Pro, where storage bandwidth really matters, include SATA 6Gb/s technology? Why is it still stuck at SATA 3Gb/s?

USB 3.0

Where is USB 3.0 on the Mac Pro? It still only has USB 2.0 ports. I already have USB 3.0 peripherals, which I wouldn’t be able to use at their native speed if I got a Mac Pro.

Furthermore, where is USB 3.0 on any Mac? USB 3.0 is here to stay, it’s fast, and it’s on a lot of peripherals. It goes without saying that Apple needs to include it on the Mac Pro and on the rest of its computers.

What’s worrying is that Mac OS X Lion may not ship with support for USB 3.0, as this article suggests. No, no, no, Apple, please don’t do that…

30-inch Display

This is more of an annoyance, but it’s worth mentioning. Ever since it came out, I always dreamed of owning a 30-inch Cinema Display. Given some of the stuff I’m doing these days, I could really use it, too. But Apple no longer makes it. The only large display they make these days is the 27-inch Cinema Display, which is very nice, and it uses LED instead of LCD technology, but it’s not a 30-inch, is it?

What’s next in data storage?

My recent musings on high definition and the state of the technology behind it have spurred me to think about data storage (not that it’s a new subject for me). But so far, I’ve commented only on what’s already been developed, and didn’t take the time to think about what’s next.

What’s the motivation behind this post? It’s simple. For Ligia’s Kitchen, it costs me about 10.5 GB for 5 minutes of final, edited footage of show, with a one-camera setup. What goes into the 10.5GB? There’s the raw footage (and sound files, if I use a standalone mic), the edits, and the final, published footage. When I use two cameras, the space needed can easily go up by 1.5-2.5x, depending on the shots I need to get. I shoot and edit in 1080p, and output to 720p.

My storage needs are okay for now. I’ve got plenty of space, and if I keep going at this rate, I should be fine. But… and there’s always a but, isn’t there… I have more show ideas in mind. And there’s the hypothetical possibility of shooting with a RED camera at some point in the future, if certain factors come together to allow it. So I’m thinking ahead.

Current hard drive technology (bits of data on disks) has certainly come a long way. Those of us who’ve been in the business long enough know what prices used to be like for capacities that are laughable by today’s standards. Back in 1999, I paid $275 for a 27GB hard drive. My laptop’s drive in college could store a grand total of 120MB. And when I began to learn programming, I’d load the code into memory from tape…

I remember being really excited about Hitachi’s new Perpendicular Magnetic Recording Technology, which came out in early 2006. They even had an animation on their website, which they’ve taken down since. That technology is behind all of the new hard drives that are on the market today, by the way. Hitachi came up with a way to get the bits of data to stand up (hence the term perpendicular) instead of lying down on hard drive platters, thus doubling the amount of data that could be stored onto them.

There are two roads ahead when it comes to data storage, of which one is more likely to succeed:

  • Optical storage (this is probably the future of storage)
  • Biological storage

Let’s first look at biological storage. One particular article made the rounds lately: researchers at the Chinese University in Hong Kong have managed to store 90GB of data in 1g of bacteria. While it sounds exciting, the idea of storing my data in petri dishes on my desk doesn’t readily appeal to me, and certain complications come up:

  • 1g of bacteria is about 10 million cells (that’s a LOT); one must start thinking about the potential for bio hazards when you work with bacteria.
  • The data is stored in a bacteria’s DNA, which means it’s encrypted (a good thing), but it’s also subject to significant mutation (a bad thing) and it takes a long time to retrieve it because you need a gene sequencer, which is tedious and expensive (a bad thing).

I’m not against this. Hey, if they can make it safe and fast, okay. But I believe this is going to be relegated to special applications. The article suggests the technique is currently used to store copyright information for newly created organisms (I wonder how many new bacteria researchers as a whole have created, and is it any wonder antibiotics have such a hard time working against them when we keep playing God). I also see this sort of data storage as a way for spies to operate, or for governments to keep certain secrets.

Okay, onto more cheery stuff, like optical storage. I’ve always thought there was massive potential here, and am glad to see significant work has already been done to make this a reality. There are two technologies which are feasible, according to research that’s already been done:

  • HDSS (Holographic Data Storage Systems), which so far can store up to 1TB of data in a crystal the size of a sugar cube, but doesn’t yet allow rewrites
  • 3D optical data storage, which so far can store up to 1TB of data onto a 1.2mm thick optical disc

These developments are very encouraging. Optical storage is safe, and its potential capacities are huge, possibly endless. And when you think about computer hardware, and how manufacturers are looking at using optical technology in the bridges and buses and wires inside the hardware, because it’s incredibly fast, you start to see how optical makes sense. Let’s also not forget fiber optic cabling, and its incredible capacity to carry data. It certainly looks like optical is the future!

So what’s going to happen to the standard 3.5″ form factor of today’s hard drives? Well, it’s likely that it will stay the same, even though it the storage technology inside it might change. We’ll have crystals and lasers instead of platters and heads, but they’ll likely be able to fit them in there somehow. I don’t think we’ll need to start keeping crystal libraries on our desks, like in Superman’s Crystal Cave, and sticking various-sized crystals into our computers any time soon, although it did look pretty cool when Christopher Reeve did it in the movie.

It really all depends on how soon this new technology will come to market. Right now, there’s clearly enough vested interest in the 3.5″ and 2.5″ form factors to motivate drive manufacturers to shoehorn the new technologies into those shapes, but if optical hard drives won’t be here for the next 5-10 years, then it’s possible that the form factor will change as well. We are after all moving to smaller, sleeker shapes for most computers, notebooks and desktops alike.

Hardware preview: the ioSafe Rugged Portable

I’m a fan of ioSafe‘s rugged hard drives, which you can trust with your data through very rough conditions. I was glad to write about the ioSafe Solo (their first product), and the ioSafe SSD, because no other company on the market offered such unparalleled protection from destruction.

Now they’re launching a new model, the ioSafe Rugged Portable — their first drive made for travel.

As you can see, they’re using a new design, with a machined Aluminum or Titanium enclosure that can withstand up to 5,000 lbs (Ti) or 2,500 lbs of pressure (Al). The drive is suspended on all six axes of motion, and can withstand a drop from 20′ (SSD version) or 10′ (HDD version).

Of course, the drive can still withstand immersion in water — up to 10′ (Al) or 30′ (Ti), both up to 3 days in duration. And it’s got a whole other bunch of protections built in as well:

  • ChemSafeTM Technology – Full immersion in diesel fuel, oils, hydraulic fluids, aircraft fuels, 12” depth for 1 hour with no data loss per MIL-STD-810G Method 504
  • EnviroSafeTM Technology – Continuous exposure to UV, blowing sand, blowing dust, rain, salt fog, icing or freezing rain, 24 hours with no data loss per MIL-STD-810G Methods 505.4, 506.4, 509.4 and 510
  • AltiSafeTM Technology – High altitude operation. 15K ft. (Alum.) and 30K ft. (SSD and Ti.) rated altitudes per MIL-STD-810G Method 500.4
  • Theft Resistant Kensington® Lock compatible slot solid metal construction – theft protection

As you’ve no doubt gathered so far, there are multiple flavors of the drive, with HDDs or SSDs inside, and Aluminum or Titanium enclosures. And it ships with USB 3.0, FW800 and USB 2.0 connections — your choice.

Possibly the best feature — given the drive’s title — is its weight, which comes to 1 lbs for the Al enclosure and 1.5 lbs for the Ti enclosure.

Always check a SATA drive’s jumper settings

I made a quick video that shows you why it’s always important to check a SATA drive’s jumper settings. Many of us assume that since we’re dealing with SATA, not PATA/IDE drives, the jumper settings are no longer important. After all, the Master/Slave relationship no longer applies to the SATA model. Not so. The jumper settings on SATA drives control other important drive settings, such as their speed of operation.

Have a look at this Seagate 500GB SATA drive, the one in my video. I assumed (wrong) that it was operating at 3.0 Gb/sec all along. It wasn’t. For over 2 years, I had three of these drives in one of my Drobo units, and I thought I was getting 3.0 Gb/sec from them, when in fact I was only getting 1.5 Gb/sec. That’s because they shipped with a jumper set to limit their speed of operation at 1.5 Gb/sec from the factory, and I didn’t check it before sticking them in my Drobo and forgetting about them.

Only now, as I re-shuffled the drives between my Drobo units, did I realize I hadn’t been getting 3.0 Gb/sec from them, and corrected the situation right away.

I can understand why Seagate would want to ship the drives set to 1.5 Gb/sec. After all, some older computers might not be capable of 3.0 Gb/sec, and you’d run into compatibility issues. They assume IT geeks would know what to do, and they’re right, they would, if they’d only bother to look…