Permanent data storage

We need to focus our efforts on finding more permanent ways to store data. What we have now is inadequate. Hard drives are susceptible to failure, data corruption and data erasure (see effects of EM pulses for example). CDs and DVDs become unreadable after several years and archival-quality optical media also stops working after 10-15 years, not to mention that the hardware itself that reads and writes to media changes so fast that media written in the past may become unreadable in the future simply because there’s nothing to read it anymore. I don’t think digital bits and codecs are a future-proof solution, but I do think imagery (stills or sequences of stills) and text are the way to go. It’s the way past cultures and civilizations have passed on their knowledge. However, we need to move past pictographs on cave walls and cuneiform writing on stone tablets. Our data storage needs are quite large and we need systems that can accommodate these requirements.

We need to be able to read/write data to permanent media that stores it for hundreds, thousands and even tens of thousands of years, so that we don’t lose our collective knowledge, so that future generations can benefit from all our discoveries, study us, find out what worked and what didn’t.

We need to find ways to store our knowledge permanently in ways that can be easily accessed and read in the future. We need to start thinking long-term when it comes to inventing and marketing data storage devices. I hope this post spurs you on to do some thinking of your own about this topic. Who knows what you might invent?

Advertisements

A look at what’s ahead in terms of resources and the economy

The TED channel published two interesting videos recently which present two points of view about the Earth, in terms of its resources and economy. The first is from Paul Gilding, entitled “The Earth is full“, and the second is from Peter Diamandis, entitled “Abundance is our future“.

I invite you to watch both points of view, which are at first in seeming opposition but after some consideration, are both saying pretty much the same thing, namely this:

Our current economic models, based on carbon forms of energy, will soon reach their lifespan, and we have some choices to make ahead as we transition to other economic models and other ways of generating our energy and making our stuff.

We can have a smooth transition or we can have a rocky one, with elements of anarchy and possible energy and water wars.

What’s clear on both sides is that we need to something about it and we need to start doing it now.

The wonderful thing is there are solutions to our energy and pollution problems emerging now and if they’re implemented correctly, we will not only avert any potential crises but we will come out ahead of the curve.

What are we waiting for? Let’s do it!

What’s next in data storage?

My recent musings on high definition and the state of the technology behind it have spurred me to think about data storage (not that it’s a new subject for me). But so far, I’ve commented only on what’s already been developed, and didn’t take the time to think about what’s next.

What’s the motivation behind this post? It’s simple. For Ligia’s Kitchen, it costs me about 10.5 GB for 5 minutes of final, edited footage of show, with a one-camera setup. What goes into the 10.5GB? There’s the raw footage (and sound files, if I use a standalone mic), the edits, and the final, published footage. When I use two cameras, the space needed can easily go up by 1.5-2.5x, depending on the shots I need to get. I shoot and edit in 1080p, and output to 720p.

My storage needs are okay for now. I’ve got plenty of space, and if I keep going at this rate, I should be fine. But… and there’s always a but, isn’t there… I have more show ideas in mind. And there’s the hypothetical possibility of shooting with a RED camera at some point in the future, if certain factors come together to allow it. So I’m thinking ahead.

Current hard drive technology (bits of data on disks) has certainly come a long way. Those of us who’ve been in the business long enough know what prices used to be like for capacities that are laughable by today’s standards. Back in 1999, I paid $275 for a 27GB hard drive. My laptop’s drive in college could store a grand total of 120MB. And when I began to learn programming, I’d load the code into memory from tape…

I remember being really excited about Hitachi’s new Perpendicular Magnetic Recording Technology, which came out in early 2006. They even had an animation on their website, which they’ve taken down since. That technology is behind all of the new hard drives that are on the market today, by the way. Hitachi came up with a way to get the bits of data to stand up (hence the term perpendicular) instead of lying down on hard drive platters, thus doubling the amount of data that could be stored onto them.

There are two roads ahead when it comes to data storage, of which one is more likely to succeed:

  • Optical storage (this is probably the future of storage)
  • Biological storage

Let’s first look at biological storage. One particular article made the rounds lately: researchers at the Chinese University in Hong Kong have managed to store 90GB of data in 1g of bacteria. While it sounds exciting, the idea of storing my data in petri dishes on my desk doesn’t readily appeal to me, and certain complications come up:

  • 1g of bacteria is about 10 million cells (that’s a LOT); one must start thinking about the potential for bio hazards when you work with bacteria.
  • The data is stored in a bacteria’s DNA, which means it’s encrypted (a good thing), but it’s also subject to significant mutation (a bad thing) and it takes a long time to retrieve it because you need a gene sequencer, which is tedious and expensive (a bad thing).

I’m not against this. Hey, if they can make it safe and fast, okay. But I believe this is going to be relegated to special applications. The article suggests the technique is currently used to store copyright information for newly created organisms (I wonder how many new bacteria researchers as a whole have created, and is it any wonder antibiotics have such a hard time working against them when we keep playing God). I also see this sort of data storage as a way for spies to operate, or for governments to keep certain secrets.

Okay, onto more cheery stuff, like optical storage. I’ve always thought there was massive potential here, and am glad to see significant work has already been done to make this a reality. There are two technologies which are feasible, according to research that’s already been done:

  • HDSS (Holographic Data Storage Systems), which so far can store up to 1TB of data in a crystal the size of a sugar cube, but doesn’t yet allow rewrites
  • 3D optical data storage, which so far can store up to 1TB of data onto a 1.2mm thick optical disc

These developments are very encouraging. Optical storage is safe, and its potential capacities are huge, possibly endless. And when you think about computer hardware, and how manufacturers are looking at using optical technology in the bridges and buses and wires inside the hardware, because it’s incredibly fast, you start to see how optical makes sense. Let’s also not forget fiber optic cabling, and its incredible capacity to carry data. It certainly looks like optical is the future!

So what’s going to happen to the standard 3.5″ form factor of today’s hard drives? Well, it’s likely that it will stay the same, even though it the storage technology inside it might change. We’ll have crystals and lasers instead of platters and heads, but they’ll likely be able to fit them in there somehow. I don’t think we’ll need to start keeping crystal libraries on our desks, like in Superman’s Crystal Cave, and sticking various-sized crystals into our computers any time soon, although it did look pretty cool when Christopher Reeve did it in the movie.

It really all depends on how soon this new technology will come to market. Right now, there’s clearly enough vested interest in the 3.5″ and 2.5″ form factors to motivate drive manufacturers to shoehorn the new technologies into those shapes, but if optical hard drives won’t be here for the next 5-10 years, then it’s possible that the form factor will change as well. We are after all moving to smaller, sleeker shapes for most computers, notebooks and desktops alike.

Is Canon at work on a mirrorless “DSLR”?

Given the popularity of, and interest in mirrorless “DSLRs” like the Sony NEX-5 or NEX-3, Olympus E-P2 or E-PL1, Panasonic GF1 or G2, and the Samsung NX-10, I believe Canon is at work on their version of the “micro-DSLR”, and has been for at least a few months. (Why do I use quotes when I call them DSLRs? See this.)

Another hint of the upcoming launch is the restlessness in the Rebel line of cameras. Perhaps I’m wrong, but I’ve never seen so many new models launched in such a short span of time. It’s probable that Canon wanted to see what features consumers really want in an entry-level DSLR now that video and small size are the new must-haves, what cameras their competitors come up with, then take their time designing an entirely new body for the camera. Keep in mind they’re very entrenched in the classic SLR design, as their EOS line shows. A mirrorless DSLR without a viewfinder, or with a significantly different viewfinder, and a much smaller body, can be a serious challenge for any company.

Here’s what I think (and hope) will be in the new Canon EOS “Elph” (I made the name up, I don’t know what it’ll be called):

A 14 or 16-megapixel APS-C sensor — they’re going to choose less noise at high ISO, and that will mean less resolution, but that’s okay by me. I’m not too happy with the noise levels in the 7D. They might even downgrade the resolution to 12 megapixels, who knows…

High ISO capabilities — up to 12,800 natively, and possibly even up to 102,400 via ISO expansion. They’ll need to do this, because the Sony NEX line has raised the bar.

Fast frame rate — 5 fps, possibly even 7 fps. Again, this is because the NEX cameras have raised the bar. They can go up to 7 fps in burst mode. The mirrorless cameras can do about 3 fps.

14-bit image bit depth — pretty much all DSLRs are either there or going there.

Full HD Video Recording, with AF — it’s going to have to be full 1080p video, because Sony has raised the bar for all cameras in this category. I do hope though that Canon will make it 1080p, not 1080i, at selectable frame rates. AF is going to be tricky but will need to be implemented. Canon will have a choice of using the current USM AF built into its lenses, which can be slow, or build a new line of lenses, like Sony did for its NEX line, with fast, quiet AF that can’t be picked up by the microphone. I do hope we won’t have to invest in new lenses.

New lenses? This is possible, in which case they’ll be smaller and lighter, but my guess is Canon will either modify the mount or put out an adapter that will allow us to fully use existing EF or EF-S lenses.

Same 19-point AF system as in the 7D — I honestly hope it’s not the same old 9-point AF system used in the Rebel line and in my 1st gen 5D and the 5D Mark II, because it seems to be troublesome in cameras with higher resolutions.

In-camera panorama stitching — Canon, please put this in! I was blown away by Sony’s Sweep Panorama feature.

In-camera HDR — yes, HDR is overrated and I can’t stand most HDR photographs, but when used judiciously and for the right scenes, HDR can help you can get a properly exposed photograph without blown areas.

Optical viewfinder — I know I just said a mirrorless DSLR won’t have an optical viewfinder, but think how cool it would be if Canon could somehow fit a great pentaprism inside a mirrorless body!

Better dust reductionsome people seem to be having trouble with the dust reduction system in the 5D Mark II. Perhaps this warrants a closer look from Canon. I’m sure there are ways to improve things, especially in a camera that will have no mirror, and where the sensor will be exposed to the open air during every lens change.

SDHC, not CF cards — I would have loved to see Canon adopt SDHC cards for the 7D. SDHC cards are smaller and less expensive than CF cards, so why keep using CF cards?

Tiltable LCD — the Sony NEX line has them, the Canon G11 has it, so why shouldn’t this new camera have it as well?

External microphone input — this is a must, as the in-camera microphone is never enough for quality video sound.

Those are my thoughts. What do you think? Don’t ask me how it’ll look though. I don’t know, and I’d like to be pleasantly surprised when I see it. I’d like it to be flatter and smaller than a Rebel, but with a good grip, and a good selection of physical controls. It doesn’t have to be too light, but it should be light enough and sturdy enough so that it’s easily carried anywhere.

The Secret Powers of Time by Philip Zimbardo

This video is an epiphany. It explains how people’s conceptions of time affect their lives and societies — and vice-versa. My jaw kept dropping as I watched it. If you’re a sentient human being, you will think it’s the best 10 minutes you’ve spent in a long time.

RSA Animate – The Secret Powers of Time

So long, Contenture

According to their website and blog, Contenture is going out of business. That’s unfortunate. I signed up with them back in June of 2008, endorsed their services here on my site, and while I didn’t make a lot of money with them, I knew the idea of micropayments for web content was something that would eventually catch on, if only given enough time to take root. It’s sad to see them go, and I’m sorry they couldn’t hang on until they saw some profits.

I only know of one other company that offers similar monetization services, and it’s called CancelAds. Here’s hoping they stay in business, and the idea of micropayments/subscriptions for web content continues to gain momentum.

Using Contenture to handle micropayments

Back in April, I wrote about micropayments, and why I thought they were an equitable way to reward web publishers for their time and effort. I shared my thoughts on ad revenues, which, for most people, are minimal and not enough to live on, unless you are one of the relatively few websites that gets massive amounts of traffic.

In that same article, I also talked about how I thought micropayments should work, via a standard, instant way to charge readers a few cents per article, through protocols that were integrated into each browser. Instead of charging fees for each transaction, which would be impractical for such small amounts, micropayment processing centers would charge for bundles of transactions which exceed a set limit.

Fast forward a few months, and ZDNet publishes an interview with Barry Diller, one of the early Internet millionaires, where he talks about the very same thing: the need to go beyond falling ad revenues by charging small amounts for useful information. That article was hotly debated on FriendFeed, which is where I found out about it. Among the comments, I found one pointing me to Contenture, a newly launched micropayment system (it saw the light of day on 5/26).

contenture

Contenture’s method of handling micropayments is different from what I envisioned, in that it involves a monthly subscription. Here’s how they say it works:

“… a fully automated system that requires no user interaction. Users simply pay a set fee to Contenture on a monthly basis, and that money is automatically distributed to the sites they visit. How much each site gets from each user is determined by how often that user visits that web site, relative to all of the other Contenture sites they visit. Users get their seamless experience, and the site actually makes money. Everybody wins.”

So what’s involved on my end? I installed Contenture’s WordPress plugin and pasted in an extra line of code to customize the ad hiding behavior. Others might need to paste a Javascript snippet in their footer, which is basically what the plugin does for you.

On your end (the reader), the code will check to see if you’re a Contenture user, and will automatically distribute your subscription fee to the websites that you visit, based on how often you visit each site. As an immediate benefit, all ads on my site are hidden from you. They’ll load as the page loads, but they blink out of view and the ad space collapses unto itself as soon as the page finishes loading. It’s pretty cool.

Other benefits I might be able to offer you in the future are the ability to view exclusive content, or perhaps to even close off the archives to those who aren’t Contenture users, although I’m not too keen on that, since Google won’t be able to index me properly any more if I do it. At any rate, the ability to offer more benefits is built in, and that’s nice.

Contenture’s model is opt-in (also called freemium) and so it has some limitations, in that I don’t get paid for everyone that accesses my content. My micropayment model, the one I envisioned and the one Barry Diller and others are talking about, is standardized across platforms and browsers and works for everyone, all the time. Eventually, I think we’ll get there, but Contenture has made a good start of it, and they did it now, which is why I signed up with them.

To let site visitors know they can support my site, I placed a small link in the header, as you can see below. I also have a link in the footer, also shown below.

header-screenshot

footer-screenshot

I look forward to seeing Contenture’s user base grow, so I can tell whether I’ve made the right choice. I want to keep writing and publishing online for a long time to come, but the only feasible way for me to do that is if I get rewarded properly for my efforts. I believe micropayments are the way to do it. It’s affordable for the readers, and it scales up nicely for me, the web publisher. Time will tell exactly which micropayment method will work best, and you can be sure I’ll continue to monitor the options available out there.