archive - help show filename convention for fitsfiles
          - fits    -e expname_yymmdd fitsfile1 fitsfile2 ... exp.README
          - stnd    -e expname_yymmdd plotfile1 plotfile2 ... exp.piletter
          - pipe    -e expname_yymmdd [*]
          - feed    -e all *.hdr
          - auth    -e expname_yymmdd -n name -p password
          - remauth -e expname_yymmdd 

  where:

  expname  = experimentname.
  yymmdd   = observationdate (for unique expname)
  plotfile = zipped postscriptfile
  fits     = send fitsfiles to archive
  stnd     = send standardplots to archive
  pipe     = send pipeline files to archive from current dir.
  feed     = send station feedback pages to archive
  auth     = set authentication for P.I on experiment.
  remauth  = remove authentication from experiment.

  Notes:
  - Step in the directory of your files, then start archive script under jops.
  - Archive script uses sftp, it creates a batchfile which is filled
    with copy commands and then executed.
  - Archive script also uses a mysql table: daemon to prepare commands for user archive on jop83.
    It tells user archive on jop83 which data type of which experiment
    has to be copied from data buffer into the archive.
  - Archive user on jop83 polls by means of archdaemon.pl table daemon
    to see if files should be copied into the archive.

   Description:
   - Administration fields are copied from ~jops/Admin/Jexp/exp.jex files
     to mysql table jexp on jop83. 
     This table is a resource for the webinterface pages.
   - The items are: expcode, piname, wavel, obsdate, distribed,
                    status, completed, publarchiv, support, 
                    obstype, schedarray, exprdescr, schedsrc.
   - All the expfiles are checked for update, if an experiment
     entry does not yet exist in the table, it is created.
   - Timefields are converted from ddmmyy to yymmdd.
   - This script is scheduled when something is changed in the
     ~jops/Expadmin/Jexp/*.jex files by program: jexp_gui.pl

  
   Description
     Archdaemon.pl is a daemon which is continuously running on the
     archive, polling database table daemon each 5 seconds to see if a command
     is waiting.

     - If send2archive is scheduled, then files are copied
       from data buffer: /jop83_0/jops/stnd/*
                                      /pipe/*
                                      /fits/*
                                      /feed/*
       to the archive : /jop83_1/archive/exp/*
     - If setAccess.pl is scheduled, the authentication is set or unset.
     - If convertPipe.pl is scheduled then in the pipeline files are
       moved to subdir prot according the settings in tables archExp and
       archSrc. Besides pointers in the .html files are corrected.
 

   send2archive.pl -stnd | -fits | -pipe | -feed -e exp | all

   where: -stnd = standardplots
          -fits = fitsfiles
          -pipe = pipeline files
          -feed = station feedback pages
          -e    = exp or all.
                
   Description:
   - send2archive.pl copies files
     from the data buffer      : /jop83_0/jops/stnd/*
                                              /pipe/*
                                              /fits/*
                                              /feed/*
     to the archive destination: /jop83_1/archive/exp/*

   - If a file already exists it will be overwritten by the 
     newer file

   - If standardplots were copied, script createpng.pl is called
     to create also png files from the postscript files. Besides
     a gzipped tarfile is created from all the postscript files.

   - If pipeline files were copied, script convertPipe.pl is called.
     It converts the experiment.html file in a way that all the
     filereferences are checked and point to the correct directory.
     Protected pipeline files are moved to subdir prot.
  
   - After fitsfiles are copied, a checksum file will be created,
     using unix utility: md5sum 
     
   - If feedbackfiles were copied, script cpfeedback.pl copies each
     experiment.hdr file to its destination.

     createpng.pl -e Experiment [-d directory] [-r]
     where:
     -d = directory (Default directory = current directory).
     -r = Rotate +90 degr. if name contains "-ampphase" or "-weight"

   Description: - Create png files from postscript files.
                - Create also a zipped tarfile of all postscript files. 

   Description:
     Remove in experiment/standplots/.png and ps.tar.gz
     Call createpng.pl again. 

     Description:   
     - It converts the experiment.html file in a way that all the
       files are pointed to the correct directory.
     - Protected pipelinefiles are moved to subdir prot.
     - The original experiment.html is saved as experiment.html.orig
     - The left border of the page is adjusted.

     Description:
     - Remove experiment.html
     - Rename experiment.html.orig => experiment.html

   cpfeedback.pl

     feedback.pl -e experiment
     where experiment = all or exp

   Description:   
     cpfeedback.pl copies each experiment.hdr file to its destination.

     Program fitsverify of NASA was implemented in webtool fitsverify.php.
     It has a selection option to select each separate fitsfile in the
     archive.

     fitsverify 4.1 (CFITSIO V2.470)
       fitsverify - test if the input file(s) conform to the FITS format.

       Usage:  fitsverify filename ...   or   fitsverify @filelist.txt

       where 'filename' is a filename template (with optional wildcards), and
             'filelist.txt' is an ASCII text file with a list of
              FITS file names, one per line.

      Optional flags:
          -l  list all header keywords
          -q  quiet; print one-line pass/fail summary per file
          -e  only test for error conditions; don't issue warnings

      Help:   fitsverify -h

    Use parameters: - def      Set default name and password for experiment
                               Only fits files are protected. (pipefiles via admin.php)
                    - set      Set name and password for experiment
                    - unset    Unset name and password for experiment
                    - e exp    exp_yymmdd.
                    - n name   name of Pi.
                    - p pass   password of pi

    Examples:
     set  :   Set   exp. e.g.: setacces.pl -set -e exp -n name -p pass
     unset:   Unset exp. e.g.: setacces.pl -unset -e exp
     def  :   Set   exp. e.g.: setacces.pl -def -e exp

    Description:

      The script controls access for the fitsfiles and pipeline files.
        - def   .htaccess and .htpassw are created in fits directory
                with default name and password.
 
        - set   .htaccess and .htpassw are created in fits directory
                and in pipe/prot directory with specified name and
                password.
                In database table archExp name and password are also
                filled in, which makes the experiment entry visible
                in admin.php where access of pipeline files can be specified.                

        - unset .htaccess and .htpassw are removed from fits directory
                and protected pipelinefiles are moved back to pipe
                dir. Pointers in html files are set back to pipe dir.      
                Settings in database tables archExp and archSrc are removed.

    Description:
    - Create new expdirs if necessary and create initial protection,
      (i.e. indexfiles and default .htaccess and passwd files.)
    - Update authentication tables: archExp and archSrc.
    - Delete protection of experiment with "setAccess -unset -e exp" if publ.date reached.
    - Create entries for new fitsfiles in database table fitsPars, which is
      used by fitsfinder.php.

    cleanupArchiveBuffer.pl -stnd | -fits | -pipe -e exp | -e all -a maxAge

    where: -stnd  = standardplots
           -fits  = fitsfiles
           -pipe  = pipeline files
           -feed  = feedback files
           -e exp = exp or all.
           -a age = max. age in days

    Drupal needs to know which files are in the archive.
    /jop83_1/archive/scripts/util/archproducts.pl collects these files in a
    database table: archproducts, which is updated every night in dailyUpdate.pl.
    Besides the script is used in send2archive.pl whenever data has been sent
    to the archive.

    Update is done from directories: feedback, standplots, pipe and fits.

    archproducts.pl -h                show this help.
                    -e experiment     collect info for experiment. (fast)
                    -create           recreate the table archproducts.
    Without parameters:               update table. (slow)

#  Synchronize VLBI results on trantor with jop83.
#
   use strict;
   use Date::Parse;
   use Date::Format qw(time2str);
   use Getopt::Long;

   my $jop83Dir   = "/jop83_1/archive/exp";
   my $trantorDir = "trantor:/jive/archive/exp";
#  my $options    = "-vrptL --dry-run";
   my $options    = "-vrptL";

#
#    -t, --times                 preserve modification times
#    -v, --verbose               increase verbosity
#    -r, --recursive             recurse into directories
#    -p, --perms                 preserve permissions
#    -t, --times                 preserve modification times         
#    -L, --copy-links            transform symlink into referent file/dir
 
   doRsync("rsync $options $jop83Dir/* $trantorDir");

This script is scheduled by jexp_gui.pl each time when an experiment.jex file
is saved. Besides it is scheduled by crontab once a day. 

updateJexp.pl copies the content of the experiment.jex files to database table jexp.
Table jexp is a resource for the archive.

+--------------------+------------------+------+-----+---------+----------------+
| Field              | Type             | Null | Key | Default | Extra          |
+--------------------+------------------+------+-----+---------+----------------+
| id                 | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| expCode            | varchar(30)      | YES  |     | NULL    |                |
| shared_exp_logfile | varchar(80)      | YES  |     | NULL    |                |
| piName             | varchar(30)      | YES  |     | NULL    |                |
| wavel              | varchar(30)      | YES  |     | NULL    |                |
| obsDate            | date             | YES  |     | NULL    |                |
| completed          | date             | YES  |     | NULL    |                |
| released           | date             | YES  |     | NULL    |                |
| distributed        | date             | YES  |     | NULL    |                |
| publarchiv         | date             | YES  |     | NULL    |                |
| status             | varchar(30)      | YES  |     | NULL    |                |
| support            | varchar(30)      | YES  |     | NULL    |                |
| obsType            | varchar(30)      | YES  |     | NULL    |                |
| schedArray         | varchar(100)     | YES  |     | NULL    |                |
| exprDescr          | blob             | YES  |     | NULL    |                |
| schedSrc           | blob             | YES  |     | NULL    |                |
+--------------------+------------------+------+-----+---------+----------------+

/jop83_1/archive/scripts/user/updateJexp.pl -h

Update table jexp from Jexpdirs.
updateJexp.pl               : All experiments will be updated
updateJexp.pl -e Experiment : Only experiment will be updated.

Move experiment to different disk when current disk is almost full.

/jop83_1/archive/scripts/util> moveExp.pl
 
Use parameters: -s sourcedisk (e.g.: jop83_2)
                -d destinationdisk (e.g.: jop83_3)
                -e experiment (e.g.: EP037A_000920)

The experiment will be copied to the new disk and deleted from the old disk.
The old link at jop83_1 will be removed and a new link from the new disk at
jop83_1 will be created.

Create links of experiments on current disk to disk jop83_1 after experiments were restored.  

Step into the directory of experiments you want to link to jop83_1.
run /jop83_1/archive/scripts/util/newLink.pl

Now all the links will be created of the experiments on this disk that did not
already have a link to jop83_1.

    This script is scheduled by crontab once a day.

    baseDir = "/jop83_1/archive/scripts";
 
    Update table jexp with parameters from /export/jops/Expadmin/Jexp/*.jex
    system("$baseDir/user/updateJexp.pl");

    Update experiment directories and check permissions.
    system("$baseDir/util/initNewExp.php");

    Update table archproducts (collect all exising filenames for drupal)
    system("$baseDir/util/archproducts.pl");

    Collect length and archivetimes for backup to LTO tapes (backupArchive.pl)
    system("$baseDir/util/backup/archtime.pl");

    Backup data to backupsystem Trantor.
    system("$baseDir/util/backup/backup2trantor.pl");

    Create plot that shows amount of data in archive.
    system("$baseDir/util/createGrowplot.php");

    Update database for fitsfinder.php.
    system("$baseDir/avo/updateFitsDb.php");

    Registrate use of archive in database table: archuser.
    
    Description:
    - Show all experiments in the archive.
    - show per experiment the archive products.
    - show feedback pages.
    - show station logfiles from EVN archive in Bologna.
    - show show standard plots.
    - show pipeline plots.
    - show fitsfiles.

    Description:
    - Show registrated fields of archive use.
    - Give selection possibillities of the data.
    - A number of search engines is omitted from the list.

    Description:
    In mysql tables archExp and archSrc access information is kept about
    groups of filenames that contain a parameter mentioned in the list below.
    
    admin.php controls the contents of these tables:    
    - Access can be set separately for each experiment and for each source.  
    - Columns (parameter names) can be manually added to or deleted from a table.

    List of parameters which can be set:

      Fitsfiles       pipeline experiment plots        pipeline source plots
      ---------       -------------------------        ---------------------
      fits            BANDPASS                         CLPHS
                      CPOL                             ICLN
                      FRING_DELAY                      IMAP
                      FRING_PHAS                       IMAPN
                      FRING_RATE                       IMAPU
                      GAIN                             UVCOV 
                      POSSM_AUTOCORR                   UVPLT
                      POSSM_CAL                        VPLOT_MODEL
                      POSSM_UNCAL                      CALIB_AMP2
                      SENS                             CALIB_AMP4
                      SNR                              CALIB_PHAS1
                      TSYS                             CALIB_PHAS3
                      VPLOT_CAL
                      VPLOT_UNCAL

     
     When settings are sent to the database tables, also a command:
     "convertPipe.pl -e experiment" is executed, which means that
     files that need protection are moved from pipe to pipe/prot.
     The links in the .html files are adjusted to pipe/prot.

    Description:
 
    fitsfinder.php is a webbased userinterface which finds and displays source
    entries from mysql table: fitsPars. 
    Table fitsPars was filled with data by script: updateFitsDb.php.

    Fitsfinder.php contains extended selection and sort options to format a wanted
    listing.

    It also contains a plotoption which displays annotated source positions in
    a plotwindow.

    Description of updateFitsDb.php:

    - Update database table fitsPars with fitsinfo.
    - Check for each experiment: Are fitsfiles available?.
    - Check for each fitsfile:
        Already entries found in databasetable for this fitsfile?
        Does the timestamp of entry match the timestamp of file?
        If not: delete entry.

    - Read fitsfile using NASA's cfitsio programs:

        Program      Fitstable       Parameters
        ---------    --------------  ---------------------------------------------------
        listhead     UV_DATA         NO_STKD, STK_1 ,NO_BAND, NO_CHAN, REF_FREQ, CHAN_BW
        tabfreq      FREQUENCY       a list of freqparameters.
        tablist      SOURCE          Source, RaEpo, DecEpo, Equinox
        tablist      ARRAY_GEOMETRY  2 letter antenne names
        tabmap       UV_DATA         arrayId,  freqId,    sourceId, intTime, totalTime,
                                     startDay, startTime, endDay,   endTime.

    - Create entry in the database with a unique combination of: arrayId, freqId and sourceId
        
      So for a fitsfile there can be more than one entry with the same sourcename 
      but with a different frequency and/or a different array.
   
    - Read only IDI fits format. (UVF format cannot yet be read by cfitsio programs).