decay of structures holding data…

I have photographs my grandmother had taken at a photo studio in San Francisco in 1927. I have other photographs of her dancing with a straw hat through fields in Italy taken in the late 1940’s. I don’t have the negatives, but I do have (black and white) prints, that have stood up pretty well.

Take that as a standard. Images still available and visible 81 years later. How many of the digital photographs we take are going to be available that long? A very few. Most of those images I’m pretty sure will be gone. To have it last that long you would have to print it in archival ink on acid-free paper now, and then it would have to survive through the gulf of time the same as my photo from 1927 has.

There are inks that are reputed to be archival, but they haven’t actually stood up to the time they have promised. Paper is a lesser problem, we’ve done archival paper.

Leave those images as digital, don’t print them and the loss will be staggering.

I keep notes and documentation of much of my job. Processes, procedures, configurations, commands, settings, notes on applications, more specific notes on rogue, unusual, non-standard applications. I’ve kept this information since I started.

My first repository for notes and docs was an Access (Microsoft) database. It worked. It was fragile. It corrupted easily. I had to keep multiple backups of the file holding the database. It wasn’t accessible except on the local computer. But it kept me from having to rediscover process and procedure, and in many cases at least gave me a launching point for solving a problem or creating a service.

Why Access? It was there, it had been installed on the windows system I was given. I didn’t know better.

I became more UNIX-centered in time. My workstation became a Sun Sparcstation. I had a laptop running windows for corporate access, but the Sparcstation was my primary tool. It would not run Access (surprise!). I needed to get the data out of Access, into some other format and available via a web browser.

I was doing some perl DBI database work at the time, so over a period of a little over a year – spare time for script and database, and then loads more spare time needed to convert and migrate data out – I put together a website using perl DBI to place and grab text data from a mysql database, then eventually the mysql database was replaced by Postgresql.

That database interface worked. It was quite a lot of program for what was really a set of notes and procedures. Each time I upgraded or rebuilt a server, part of the checklist was to install and bring up postgres, perlDBI, whatever module worked that time with that version of postgres, plus any other dependencies. Import a current sql backup. A lot of work for content that just wasn’t that complicated.

I started using a wiki as a documentation structure – and started using it personally as well. At one point my work switched from twikiwiki to MoinMoin, and I followed along. MoinMoin was written in python, used css and was relatively easy to customize. It was extensively searchable. And it used a directory structure and text-based files instead of a database.

That struck me as right. It really appealed to me. It could be run locally or on a server, and the structure was simple.

The wiki grew into wikis. It grew into a music database, a data CD and DVD database, a movie database, a systems administration database and a personal database. This was all in text and directory structure, really quite cool.

Toward the beginning of using it I wrote a perl script to parse out through the structure and convert pages to text, and to grab and store attachments with them. In other words to make the whole thing readable without any structure, no wiki, no special program needed. Remember, by this time I have converted this into different structures 6 times. This isn’t new.

MoinMoin wiki stayed in development over the last six years. The website is available today. At least, it is so far today.

It was NOT available much of last week, nor the week before that. I don’t know why. Maybe my repeated attempts to get to the site and find out what was happening with the project caused someone to see it was down…

Having the development site unavailable was a shock. I went back, found my MoinMoin-to-text-files script and tested it. Just in case…

Structure tends to decay. Structures holding digital information seem to do this alot.

There’s the personal entropy – having to maintain and keep running the computers that access and serve the digital data. There’s a new version of the operating system, or a different version of the webserver, or a patch that breaks the process. Upgrading a hard drive. I’ve gone from 40 GB to 60GB to 120GB to 250GB to 500GB drives in just my personal systems.

There’s format entropy. My notes went from “.mdb” format to mysql database format to postgresql database format to text embedded in directories and files referenced within MoinMoin to (via my perl script) text. There was a brief trip BACK to mysql format when I tried out tikiwiki, when for a couple of days it looked like MoinMoin was ended as a project.

Digital images will hit that format entropy fast – most consumer digitals take jpeg format images. Originally those images were 1 – 2 Mp, now a good camera is 8 – 12 Mp. I caught the digital wave at 2.1 megapixels. I have pictures of my daughter and my son, priceless to me, in digital format, in jpeg format, at that 2.1 megapixel resolution. Not very good today. Not very good even across the history of photography.

So there’s the degradation of the image just by using a digital camera at all. JPEG is a lossy, compressed image form, losing more information every time it is opened, changed and then saved. You can do the equivalent of acid-free paper – with a good digital SLR you can take RAW format, which gives you all of the data from the sensor, and you can preserve that, creating jpegs to be viewed but always archiving the original digital negative. How many people taking snapshots do this? And even RAW format support is questionable over time, better to convert to Adobe DNG.

Then there’s archival media format – to what? 5-1/4″ floppies? 3-1/2″ floppies? Zip drive cartridges? 4mm DAT tape? No – CDR and DVD optical disc. And then ten years from now, or sooner, you will need to move them to the next format. And multiple copies, because the media degrade as well. The amount of entropy and change you have to anticipate and account for is huge.

I started taking film photographs as well as digital in light of this. 35mm and medium format. They still become digital through scanning, but the source is proven archival – film. Film is just a high capacity sensor you use once…

I started keeping paper notebooks several years ago. Pocket-sized, moleskine-brand with a black cover and an elastic closer and lined pages. These replaced a couple of Palm Pilots.

I swear, I love technology – but…

Writings I would have committed to a journal page in the wiki now go on paper. I know it’ll be available in thirty or forty years. Someday my grandkids can find it. I know I found that kind of communication from my great-grandfather, in journals and letters. That was way more important in forming an idea of him as a person than anything else I could have from him.

I started writing by hand to keep up the practice of writing with a pen – I was finding that I wrote so seldom by hand, almost all communication was via a keyboard, that I was finding it very difficult to write more than a line or two. Just no practice, and the muscles were no longer used to it.

This black book is a place to put lists. Ideas around those lists, as well, but primarily it is “The Book That Tells Me What To Do”. There is an associated volume, a Day Timer spiral-bound compact-size date book in a leather cover, which would be “The Book That Tells Me When To Do What The Other Book Says”.

The structure here makes the data more useful – writing on paper, crossing out, adding notes next to the line in a list, reworking, sketching – all of that is intuitive on paper – and harder to accomplish in a form of that size any other way that is as effective. The structure creates a possibility of longevity and persistence. The books could get lost or destroyed. But they might not.

On the Palm Pilots, the lists took more time to maintain, and I found them somehow less real and less useful than the low-tech solution I use now. And more subject to entropy and decay.

My first computer had a hard drive. 20 MB. Running Dos 6.2, I had perhaps 11 MB free. Toward the end of its life I was doing a physical format, low level, on the hard drive and then restoring the data from floppy disks every three months. Backups were absolutely critical, because you could not determine when the hard drive would start to lose sectors and drop data out – it just happened. And a single copy of the data on a single set of floppies was dangerous – any one of them could become unreadable for some unknow reason at any time. That was a learnign experience – a more intense version of what happens over, what, five years today?

The data remains separate from the structure. Data decays differently. It falls out of use. It no longer has relevance to any purpose or dream you might have that it originally led to, or fed on. Structure seems to connect directly to immediate use, and connects directly to the long term need for the data, and to how portable it can be when the structure needs to be discarded.

Could always go way old school and print it all out and file it in a cabinet…

— dsm

source: The American Heritage® Dictionary of the English Language: Fourth Edition. 2000

back to article
back to top of article