The God of Backups.
==================

By Greg Rose.

There is a God whose prerogatives include backups.
I don't know this God's name, but I know He (or She,
or perhaps It, after all I wouldn't want to get
sexist when talking about this God...) is definitely
a vengeful God. Vengeful, angry, full of wrath,
innovative in devising punishments and with a warped
sense of humour. Altogether, He (She, It) is not a
fun Guy (Gal, Goo) to have around.

Take a case in point. This is a true story, I know
the names of the individuals and companies
involved, but I'm not going to tell you. Anyway,
there is this computer vendor, let's call them
Vendor Inc., who sold a big, expensive computer to
The Customer Corporation (say). This computer runs
Customer's absolutely vital, not to mention huge,
corporate everything database. 

This system and database have been installed for
over ten years, and the Data Processing department
of Customer Corp. has been doing its job quietly
and happily for that time. Every six months, in
addition to their regular backup schedule, they took
a full system image dump, ending up with more than
10 tapes. Then, being the professionals that they
are, Customer's Data Processing people sent the
tapes to Vendor and paid a lot of money to have them
loaded and verified. Vendor actually had to assemble
a big enough machine and restore and run the system
on it. Then the tapes were locked into a fireproof
safe at Vendor's headquarters in case they were
needed.

Well, late in December, Customer's mainframe lost
a cabinet full of drives to a small fire. "No
Worries," said the D.P. Manager, a Mr. Lamb.
"We'll just get new drives under maintenance and
reload from the backups!"

The replacement drives were installed within a day,
after all Customer Corp was big and important, and
they paid a lot for maintenance too.  The tapes were
retrieved from Vendor's safe, and the reload commenced.
As it happens, this major backup was done in
November, only a couple of weeks earlier.

Tape 7 was the problem. Tape 7 couldn't be read.
To be slightly more precise, what was there could
be read, but there wasn't anything there.  Nothing
at all. It had been a brand new tape before the
backup was done, of course, no expense was spared,
and it had all the characteristics of a brand new
tape now.  It looked like some sort of procedural
error had occured, and one of the other tapes had
been written twice, or some such irrelevant thing.
At this point, who cares what went wrong. Or, I
should say, Who cares?

It was, of course, recrimination time. Poor Mr.
Lamb was called up before the Board of Customer
Corp, and asked to explain why the system has been
down for two weeks. Fortunately, he was able to
point the finger; after all Vendor was supposed to
ensure that the backups were all there. The CEO of
Customer called the CEO of Vendor Inc, and after
the conversation had settled down a bit, asked why
the backups had been useless after customer had
spent the last 13 years paying (a lot) for them to be
verified? "I'll get back to you," said Vendor's
CEO, with a sort of quaver in his voice.

Vendor's CEO was noticeably more confident when he
called back. "I have some bad news for you -- I hope
you're sitting down," he said to Customer's Head
Honcho. "When the tapes came last month there was
a note attached. It said that you were fed up with
paying so much to have the backups verified, after
all they were never used, and you just wanted us to
store them. It was signed by a Mr. B. Counter. Of
course we did as requested. I'm sorry."

Mr. Lamb was fired for failing to verify that the
verification had happened. Somewhere, I'm sure,
there was a sound of Hysterical Laughter.


Many people think that "backup" is a noun. That it
is an object that you refer back to when you need
to. Others think that "backup" is a verb, that it
is the act of making a copy of data. That is a bit
closer.  Any noun can be verbed. But here is the
Truth.

A backup is a religious ceremony that propitiates
the God of Backups.

It's obvious really. Everyone knows that if you take
a good backup it will never be used.  Even Murphy
knew that.  (Archives get used a lot. They are not
backups.  If you ever let your users know how easily
you can restore files, they will delete important
ones whenever they need temporary storage, then just
ask for them back. That's an archive, not a backup.)

The converse is also true though. If you forget to
take a backup, that isn't so bad, the God has
other people He can Hassle, for a while anyway.
But if you take a backup onto bad media, or have a
power outage in the middle of one, or take a
backup with a script that expects to be run in "/"
when your current directory was "/tmp", or any of
these sorts of things, well, you asked for it.
KerPow. The God of Backups wants his ceremonies
done right, Or Else.

Just to establish my credentials (and show how old
I am) I'll give another example. We'd been running
Version 5 Unix (no, that is not UNIX System V)
for about six months when Version 6 came in. Doing
a disk-to-disk copy of an RK05 using the block
device took about 15 minutes. But V6 introduced
raw devices!

We had three drives, not surprisingly numbered 0,
1 and 2. No tape, but removable packs. All backups
were disk-to-disk. Well, we went to single user
mode, put the system backup pack in drive 1, and
started to dd from /dev/rk0 to /dev/rk1.  (I told
you about nouns and verbs, didn't I?) About halfway,
our administrator stopped the copying, and said
"I'll just use the raw devices to make it faster."
Of course, the moment the copying was interrupted,
the God's beeper went off, and He (She, It) was
watching in fascination.

Our admin person quickly mknoded three times to
create the raw devices.  When the dd was restarted,
this time on the raw devices, the lights on drives
0 and 2 (!) came on, and stayed on for a pleasing
amount of time as entire tracks were copied. It was
only about thirty seconds before we noticed that
the wrong drive was lighting up. The DecWriter
console still showed the typo that had mixed up the
drive minor numbers, but it was too late. The system
disk backup had the beginning of the current system
disk, but the end of the old one. The actual system
disk had the end of itself intact, but the beginning
of it was clobbered by the front of the user disk.
The situation might have been recovered at that
point, since the whole system disk image existed,
just in two pieces.

Of course, since the system was running off that
system disk, it promptly crashed. Maniacal Laughter.

So when you are performing a backup, remember the
proper sense of gravity in the situation.  Your
backup ceremony should be systematic. Label the
tapes neatly, immediately after they come out of
the drive. Store them properly. Don't interrupt a
backup, and don't let anything stop you from finishing
one. Establish a schedule of Worship, and stick to
it.

And don't ever start giggling during a backup, or
tell backup jokes...