The data is stored on milo on
/solarm/FTP/pub/stokes
in individual
directories as detailed above. The data is archived using the tar
command to write all the files/directories into a single compressed
file on an exabyte tape. However, milo does not have its own exabyte
drive, so the tar file must be transferred over the network to a
machine that does have an exabyte. The basic command sequence for
tarring the data and transferring it to the exabyte drive on akala is
as follows (these commands are explained in the UNIX man pages for the
tar command):
akala>102% allocate st0 akala>103% rlogin milo milo>101% cd /solarm/FTP/pub/stokes milo>102% tar -cvfb - 20 * | rsh akala dd of=/dev/nrst0 obs=20bThis will take an hour or so. After the tar is complete, always rewind the tape and check it again with tar to make sure it can be read properly using the following commands:
akala>104% mt -f /dev/nrst0 rew akala>105% tar -tvf /dev/nrst0 > /solar/Stokes/tape_logs/stokes.tar.list.DATEwhere DATE is the current date in the DDmmmYY format (e.g. 06apr95).
In order to also assure that all the data in the directories was actually stored on the tape, one can compare the out from the "tar -t" command above with a general directory listing, produced with the following command:
milo>103% ls -lR > /solar/Stokes/tape_logs/stokes.ls.list.DATE
The numbers of files, their names, and their sizes can be compared between the two files to assure that no data was lost. Comparing the entire listing can be quite time consuming, so generally a spot check of several directories is all that is done. The above command will also store the output of the "ls -lR" command in the Stokes tape log directory (e.g. /solar/Stokes/tape_logs/stokes.ls.list.11oct95). This is simply done for the sake of completeness and to allow one to check in the future the actual contents of the directory with what was archived.
When the integrity of the tar file on exabyte has been checked, a listing of all the archived directories is made and sent to Elaine Kiernan (kiernan@koa), using a command such as:
milo>104% ls -l * > ~/stokes.dir.listThis lets Elaine knows that the data has been sucessfully transferred to exabyte and can also be deleted from the machine there.
After, the data has been archived, it can be deleted from /solarm. I actually like to leave the data for the past week or so on /solarm so that there is some recent data there to look at, should the need arise. To delete the data from /solarm, the following command can be used:
milo>105% \rm -r NN*where NN is the beginning two numbers of the directories you want to delete (e.g. 21*).
The exabyte tape is then labelled and put upstairs in the data archive. The command used to archive the data ("tar -xvf ... | dd ...") and what machine was used should be written on the card in the tape case. In addition, the days of the year included on that tape, and the range of dates that covers, should also be written on the tape.
If there is a problem in the transfer, or if /solarm fills up, then polsyn@koa will automatically send a message detailing any discrepencies between the directories for the data on koa and the directories on milo. If you clear space on /solarm, the missing files will automatically be re-transferred the following night.
In the past, a listing of the data that was archived off to tape was stored in a master file that showed the contents of each tape. This was useful and appropriate when the data was stored on nine-track tape (which only held two or three days) and the notification and backup procedures were more complex. However, the files in /solarm/Stokes/tape_logs (described above) take over most of this functionality. I mostly kept the list up to date, but I don't think it is necessary anymore. The file, which lists tapes back to May, 1991, is in /solar/daily/Mees/tape_logs/stokes.tapes.logs.
And that's all there is to it. One thing to watch out for is that new data may come in between the time you made the tar archive and when you go to delete the archived data. This could happen if you start the tar job in the evening, go home, and then delete the archived files the next day. In this case, be careful not to delete files that haven't been archived yet.