Description of backup/restore procedures for EVN Archive at JIVE.

Introduction

The EVN Data Archive at JIVE is saved on LTO-3 400 GB and LTO-5 1500 GB magnetic tapes. The shift to LTO-5 occurred on 4 Feb 2015. On 2007-06-06 the whole archive was saved on 10 tapes and after that weekly backups were added to next tapes.

The backups are checked and written as tarfile of directory content or file content to tape:

The same directory may be saved more than once, because of additions to the data after first time of archiving.

Use of backupscripts

The backupprocedure is divided into two parts: the prepare and the backup part: prepareBackup.pl and backup2tape.pl.

During prepare a preparelist is created of all experiment directories to be saved.
The modification time of each directory is compared against a backuptime in mysql table backup_arch, which contains the backup administration. The preparelist may cover several tapes.
After that the backup2tape script must be executed for each tape in the list.

The backupArchive script can be executed by crontab, it contains calls to the prepareBackup and backup2tape script.
If a tape is full, the user will be notified by email.

Action:

   Replace the LTO tape by a new one: archtapexx

   > ssh jops@jop83
   jop83:~> backup2tape -init -f archtapexx
   jop83:~> backupArchive

If error message:

   backup2tape.pl: ERROR archtape57: tar: /dev/nst0: Cannot write: Input/output error
   tar: Error is not recoverable: exiting now

   Then clean the head of LTO drive, using a cleaning tape and try again.
   If that does not help: restart the archive computer.

The backup scripts

Collect filenames with length and last modification time for backupArchive.pl in database JIVE:archtime

archtime.pl

 function: Create a mysql table with timestamps of archivefiles or directories.
           timestamps for: experiment/standplots
                           experiment/pipe
                           experiment/fits/filename
           It is used to quickly detect which files need to be archived.

           Since 2007-12-22 the backupfiles are not anymore per experiment,
           but per experiment/standplots, etc. (see above).
           That's why 2007-12-22 us used as starting date for the checks.

 actions:  if table exists drop table;
           create table
           fill table.

Prepare a list of experiments to backup:

prepareBackup.pl

       -a prepare backup for all experiments in archive.
       -d prepare backup from archivedate, e.g.: -d 2006-06-01
          If no date, prepare backup for all experiments that are not on tape.
       -c "exp1 exp2 ... expn" => prepare backup from list in commandline.
       -m max nr of GBytes on tape [1500].
       -f Name of tape to start with, e.g. archtape12.
          If no tape given, default the latest backuptape will be used and
          label = lastlabel + 1
       -h Show this information

       prepareBackup.pl creates a file: prepare.log in dir /jop83_0/jops/backup.
       It contains a list of experiments to backup, ordered in a way that
       each tape is used as efficient as possible.

Write data from disk to tape:

 backup2tape.pl

       -f tapeName        => Check tapeName and start writing experiments
                             to tape from list in file prepared.log.
       -init - f tapeName => init this tape with tapeName
   
       backup2tape.pl reads the list of experiments that were prepared for this
       tape in file  /jop83_0/jops/backup/prepare.log.
       The logging of the backup procedure is written to mysql table:
       backup_arch.

Scheduled by crontab, currently each day:

 backupArchive.pl (Combined prepareBackup.pl and backup2tape.pl) 

       -f tapename  -> e.g. archtape12, default current tape.
       -d date      -> e.g. 2006-01-01, default no date (all experiments that are not on tape)
       -h           -> This information.

Restore data from tape to disk:

 restore2archive.pl

       -f tapeName
       -d destination directory
       -b beginlabel
          If no beginlabel, the whole tape will be restored.
       -e endlabel
          If no endlabel, endlabel = beginlabel
       -t test program without use of tape device.

       If a latest version of the experiment is on a higher label
       or on a next tape, the current label will be skipped and the
       position of the latest version will be mentioned.