intuitive engineering

I've been willing my daughter to breathe for the last 24 days. 

She came down with leukemia.   Acute, dangerous and sudden leukemia that tried very hard to kill her.  Right now she is on an oscillating ventilator, with two chest tubes, daily dialysis, and a wall of IVs you would have to see to believe.  And she is still breathing. 

I have a log rotation script I put together that works for my new employer, Constant Contact.  I'll put that up soon enough. 

As soon as she is out the other side of all of this.

–doug

Tags: doughter, script.

I used Firefox 2 with about seven plugins that I considered must-haves – things like Adblock Plus and Flashblock and Colorful Tabs and Tabbrowser and Remember Mismatched Domains. I recommended it highly, and if it had occasional crashes, they were few and far between enough that I never paid them a lot of attention.

Firefox 3. Not so much. I’ve been waiting since the product first came out for a resolution to whatever memory leak or looping code causes it to just lock up the entire desktop and churn away until I kill it off.

I’ve been using Opera, which I just find a bit odd, but mostly faster and mostly stable. I’ve downloaded Chromium for linux, Codeweaver’s proof-of-concept of running Chrome under wine on linux. And I downloaded and installed opera, chrome and even Apple’s safari on windows.

This firefox misbehavior seems to happen on BOTH operating systems. Here’s a shot of the current bahavior:


Tasks: 180 total, 2 running, 178 sleeping, 0 stopped, 0 zombie
Cpu(s): 70.1%us, 2.7%sy, 0.0%ni, 27.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2066104k total, 2023092k used, 43012k free, 296832k buffers
Swap: 1646620k total, 39812k used, 1606808k free, 972692k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17146 dsm 20 0 373m 180m 35m R 143 8.9 71:52.02 firefox
4995 root 20 0 387m 53m 4744 S 2 2.7 137:35.74 Xorg
7409 dsm 20 0 32592 15m 11m S 1 0.8 0:09.48 konsole
5343 root 20 0 3304 1044 908 S 0 0.1 2:40.47 hald-addon-stor
7793 dsm 20 0 53516 32m 2236 S 0 1.6 9:09.60 synergys
1 root 20 0 2844 1688 544 S 0 0.1 0:01.84 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:03.72 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.32 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:04.90 events/0
10 root 15 -5 0 0 0 S 0 0.0 0:03.88 events/1
11 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
46 root 15 -5 0 0 0 S 0 0.0 0:00.28 kblockd/0
47 root 15 -5 0 0 0 S 0 0.0 0:00.30 kblockd/1
50 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
51 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
160 root 20 0 0 0 0 S 0 0.0 0:07.48 pdflush
161 root 20 0 0 0 0 S 0 0.0 0:02.20 pdflush
162 root 15 -5 0 0 0 S 0 0.0 0:02.10 kswapd0


and then after killing it off and restarting:


root@dali:/home/dsm/Desktop/programmes# killall firefox
root@dali:/home/dsm/Desktop/programmes# top
top - 14:46:04 up 10 days, 23:08, 1 user, load average: 0.96, 1.31, 1.25
Tasks: 180 total, 3 running, 177 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.9%us, 0.6%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2066104k total, 1955112k used, 110992k free, 296928k buffers
Swap: 1646620k total, 39812k used, 1606808k free, 953204k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17650 dsm 20 0 248m 126m 28m R 17 6.2 0:25.31 firefox
4995 root 20 0 387m 53m 4520 S 1 2.7 137:42.25 Xorg
7366 dsm 20 0 29972 11m 8668 S 0 0.6 1:29.68 kwin
7370 dsm 20 0 36764 18m 13m S 0 0.9 11:19.00 kicker
7793 dsm 20 0 53516 32m 2236 S 0 1.6 9:11.19 synergys
1 root 20 0 2844 1688 544 S 0 0.1 0:01.84 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:03.72 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.32 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:04.92 events/0
10 root 15 -5 0 0 0 S 0 0.0 0:03.88 events/1
11 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
46 root 15 -5 0 0 0 S 0 0.0 0:00.28 kblockd/0
47 root 15 -5 0 0 0 S 0 0.0 0:00.30 kblockd/1
50 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
51 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
160 root 20 0 0 0 0 S 0 0.0 0:07.48 pdflush
161 root 20 0 0 0 0 S 0 0.0 0:02.20 pdflush
162 root 15 -5 0 0 0 S 0 0.0 0:02.10 kswapd0


Yes – that first figure – 143% of the cpu. Dropping back to 17% of the CPU with the same tabs and sites loaded.

Whatever. I don’t WANT to pay attention to the browser – the whole point of firefox was to NOT have to pay attention, and they have completely ruined that in version 3.

There is something seriously wrong with firefox, that causes it over time to start misbehaving and to require a reload from scratch. And this seems to occur on both windows and Linux. It is no longer useful. It also seems to being misbehavior once it has downloaded just about any kind of update whether for the browser or a plugin.

I’ve been waiting for firefox three to fix this. Recently firefox 3.07 came out. I just got the window informing me that it has upgraded me to the latest!!!. Just this morning. And yet… And yet – it is still BROKEN.

When you go onto the internet and search for Firefox 3 performance or cpu or memory issues – the first ten or fifteen google results laud firefox three for its blazing performance. True – for awhile. Eventually it will force you to pay attention to it by hanging and chewing up your cpu to where your computer is no longer working.

I wish they’d fix it. But whether they do or not,

I can’t wait for Chrome to come out for linux…

 

–doug

Tags: browsers, firefox 3, internet.

 

The world goes crazy because the money stops flowing. Why?

George Dyson wrote an essay on current finances in the world. In that essay, he recounts the history of “tallies”, sticks of wood carefully notched to indicate funds and dates. Money deposited in the Exchequer for the use of the king was tallied, accounted for on the stick and then the stick split. The tally was split and one half remained at the place of deposit (and was called the “stock”, thus the original of the term we have), the other half went with the depositor.

Take that essay, distill it into the New York Times and another article, this one an Op Ed piece by Richard Dooling. Quote:

 

Somehow the genius quants — the best and brightest geeks Wall Street firms could buy — fed $1 trillion in subprime mortgage debt into their supercomputers, added some derivatives, massaged the arrangements with computer algorithms and — poof! — created $62 trillion in imaginary wealth…

…Making money, it seems, is all about the velocity of moving it around, so that it can exist in Hong Kong one moment and Wall Street a split second later.

 

Go back to the tallies – a stock of wood representing money.

If you wave that stick back and forth REALLY REALLY fast, it looks like you have two of ‘em.

There’s your hallucinatory addition of wealth. If you move the money around fast enough, it looks like it is in two places at once, or more. Using computer and advanced made-up economic vehicles, you get 62 places at once.

That analogy, and, actually, metaphor, that visual of someone waving a stick back and forth in furious concentration, that works for me. Scale it up, make it much harder to see, make it shiny, call it a credit swap or a derivative, and you really are in that same place, waving a magic stick back and forth and insisting there are two of them.

 

 

Brilliant.

The plates twirling at the top of the sticks are fine as long as they are moving. The financial markets don’t come to an actual static accounting unless and until the movement stops. Visual economics.

Optical illusion. Optional Delusion. Or maybe not so optional – maybe just a matter of perception.

 

 

—dsm

 

Tags: economics, finances, hallucinations.

 

I have photographs my grandmother had taken at a photo studio in San Francisco in 1927. I have other photographs of her dancing with a straw hat through fields in Italy taken in the late 1940’s. I don’t have the negatives, but I do have (black and white) prints, that have stood up pretty well.

Take that as a standard. Images still available and visible 81 years later. How many of the digital photographs we take are going to be available that long? A very few. Most of those images I’m pretty sure will be gone. To have it last that long you would have to print it in archival ink on acid-free paper now, and then it would have to survive through the gulf of time the same as my photo from 1927 has.

There are inks that are reputed to be archival, but they haven’t actually stood up to the time they have promised. Paper is a lesser problem, we’ve done archival paper.

Leave those images as digital, don’t print them and the loss will be staggering.

I keep notes and documentation of much of my job. Processes, procedures, configurations, commands, settings, notes on applications, more specific notes on rogue, unusual, non-standard applications. I’ve kept this information since I started.

My first repository for notes and docs was an Access (Microsoft) database. It worked. It was fragile. It corrupted easily. I had to keep multiple backups of the file holding the database. It wasn’t accessible except on the local computer. But it kept me from having to rediscover process and procedure, and in many cases at least gave me a launching point for solving a problem or creating a service.

Why Access? It was there, it had been installed on the windows system I was given. I didn’t know better.

I became more UNIX-centered in time. My workstation became a Sun Sparcstation. I had a laptop running windows for corporate access, but the Sparcstation was my primary tool. It would not run Access (surprise!). I needed to get the data out of Access, into some other format and available via a web browser.

I was doing some perl DBI database work at the time, so over a period of a little over a year – spare time for script and database, and then loads more spare time needed to convert and migrate data out – I put together a website using perl DBI to place and grab text data from a mysql database, then eventually the mysql database was replaced by Postgresql.

That database interface worked. It was quite a lot of program for what was really a set of notes and procedures. Each time I upgraded or rebuilt a server, part of the checklist was to install and bring up postgres, perlDBI, whatever module worked that time with that version of postgres, plus any other dependencies. Import a current sql backup. A lot of work for content that just wasn’t that complicated.

I started using a wiki as a documentation structure – and started using it personally as well. At one point my work switched from twikiwiki to MoinMoin, and I followed along. MoinMoin was written in python, used css and was relatively easy to customize. It was extensively searchable. And it used a directory structure and text-based files instead of a database.

That struck me as right. It really appealed to me. It could be run locally or on a server, and the structure was simple.

The wiki grew into wikis. It grew into a music database, a data CD and DVD database, a movie database, a systems administration database and a personal database. This was all in text and directory structure, really quite cool.

Toward the beginning of using it I wrote a perl script to parse out through the structure and convert pages to text, and to grab and store attachments with them. In other words to make the whole thing readable without any structure, no wiki, no special program needed. Remember, by this time I have converted this into different structures 6 times. This isn’t new.

MoinMoin wiki stayed in development over the last six years. The website is available today. At least, it is so far today.

It was NOT available much of last week, nor the week before that. I don’t know why. Maybe my repeated attempts to get to the site and find out what was happening with the project caused someone to see it was down…

Having the development site unavailable was a shock. I went back, found my MoinMoin-to-text-files script and tested it. Just in case…

Structure tends to decay. Structures holding digital information seem to do this alot.

There’s the personal entropy – having to maintain and keep running the computers that access and serve the digital data. There’s a new version of the operating system, or a different version of the webserver, or a patch that breaks the process. Upgrading a hard drive. I’ve gone from 40 GB to 60GB to 120GB to 250GB to 500GB drives in just my personal systems.

There’s format entropy. My notes went from “.mdb” format to mysql database format to postgresql database format to text embedded in directories and files referenced within MoinMoin to (via my perl script) text. There was a brief trip BACK to mysql format when I tried out tikiwiki, when for a couple of days it looked like MoinMoin was ended as a project.

Digital images will hit that format entropy fast – most consumer digitals take jpeg format images. Originally those images were 1 – 2 Mp, now a good camera is 8 – 12 Mp. I caught the digital wave at 2.1 megapixels. I have pictures of my daughter and my son, priceless to me, in digital format, in jpeg format, at that 2.1 megapixel resolution. Not very good today. Not very good even across the history of photography.

So there’s the degradation of the image just by using a digital camera at all. JPEG is a lossy, compressed image form, losing more information every time it is opened, changed and then saved. You can do the equivalent of acid-free paper – with a good digital SLR you can take RAW format, which gives you all of the data from the sensor, and you can preserve that, creating jpegs to be viewed but always archiving the original digital negative. How many people taking snapshots do this? And even RAW format support is questionable over time, better to convert to Adobe DNG.

Then there’s archival media format – to what? 5-1/4″ floppies? 3-1/2″ floppies? Zip drive cartridges? 4mm DAT tape? No – CDR and DVD optical disc. And then ten years from now, or sooner, you will need to move them to the next format. And multiple copies, because the media degrade as well. The amount of entropy and change you have to anticipate and account for is huge.

I started taking film photographs as well as digital in light of this. 35mm and medium format. They still become digital through scanning, but the source is proven archival – film. Film is just a high capacity sensor you use once…

I started keeping paper notebooks several years ago. Pocket-sized, moleskine-brand with a black cover and an elastic closer and lined pages. These replaced a couple of Palm Pilots.

I swear, I love technology – but…

Writings I would have committed to a journal page in the wiki now go on paper. I know it’ll be available in thirty or forty years. Someday my grandkids can find it. I know I found that kind of communication from my great-grandfather, in journals and letters. That was way more important in forming an idea of him as a person than anything else I could have from him.

I started writing by hand to keep up the practice of writing with a pen – I was finding that I wrote so seldom by hand, almost all communication was via a keyboard, that I was finding it very difficult to write more than a line or two. Just no practice, and the muscles were no longer used to it.

This black book is a place to put lists. Ideas around those lists, as well, but primarily it is “The Book That Tells Me What To Do”. There is an associated volume, a Day Timer spiral-bound compact-size date book in a leather cover, which would be “The Book That Tells Me When To Do What The Other Book Says”.

The structure here makes the data more useful – writing on paper, crossing out, adding notes next to the line in a list, reworking, sketching – all of that is intuitive on paper – and harder to accomplish in a form of that size any other way that is as effective. The structure creates a possibility of longevity and persistence. The books could get lost or destroyed. But they might not.

On the Palm Pilots, the lists took more time to maintain, and I found them somehow less real and less useful than the low-tech solution I use now. And more subject to entropy and decay.

My first computer had a hard drive. 20 MB. Running Dos 6.2, I had perhaps 11 MB free. Toward the end of its life I was doing a physical format, low level, on the hard drive and then restoring the data from floppy disks every three months. Backups were absolutely critical, because you could not determine when the hard drive would start to lose sectors and drop data out – it just happened. And a single copy of the data on a single set of floppies was dangerous – any one of them could become unreadable for some unknow reason at any time. That was a learnign experience – a more intense version of what happens over, what, five years today?

The data remains separate from the structure. Data decays differently. It falls out of use. It no longer has relevance to any purpose or dream you might have that it originally led to, or fed on. Structure seems to connect directly to immediate use, and connects directly to the long term need for the data, and to how portable it can be when the structure needs to be discarded.

Could always go way old school and print it all out and file it in a cabinet…

— dsm

---

Definition: entropy

entropy

SYLLABICATION: en·tro·py
PRONUNCIATION: ntr-p
NOUN: Inflected forms: pl. en·tro·pies
1. Symbol S For a closed thermodynamic system, a quantitative measure of the amount of thermal energy not available to do work.
2. A measure of the disorder or randomness in a closed system.
3. A measure of the loss of information in a transmitted message.
4. The tendency for all matter and energy in the universe to evolve toward a state of inert uniformity.
5. Inevitable and steady deterioration of a system or society.

ETYMOLOGY: German Entropie : Greek en-, in; see en–2 + Greek trop, transformation; see trep- in Appendix I.
OTHER FORMS: en·tropic (n-trpk, -trpk) —ADJECTIVE
en·tropi·cal·ly —ADVERB

source: The American Heritage® Dictionary of the English Language: Fourth Edition. 2000

back to article
back to top of article

Tags: data, databases, entropy, moinmoin, structure.