DATA SLURPING COOKBOOK

version 27 AUG 2014 by Jen Winters and John Subasavage

version 17 MAY 2016 by Todd Henry

You can find here details about how to:

1. Move data around at CTIO from the Torrent
machine to the ctio36 storage machine.

2. Retrieve data from the 0.9m at CTIO and
to do basic bias subtraction and
flat-fielding.

**********************************************************************
***** MOVE DATA FROM new-ctioa4 (Torrent computer) TO ctio36 *****
**********************************************************************

FROM BEYOND CTIO:

1. Type "huygens" to ensure that you are on the machine huygens, which
is the only ATL computer cleared for entry to computers at CTIO.

2. Type "ssh -l observer new-ctioa4.ctio.noao.edu" and use the
new-ctioa4 password to get into the Torrent computer (known as
new-ctioa4). observer@new-ctioa4 will be displayed on the screen.

3. You will enter in the /home/observer directory. The data you want
are typically found in the /home/data directory, so cd to there,
find your data, and note the path, typically /home/data/YYYYMMDD-09

4. leave new-ctioa4 by typing exit --- you are now back in ATL

5. Type "ssh v12@ctio36.ctio.noao.edu" and use the v12 password to get
in. v12% will be displayed on the screen.

AT CTIO ON ctio36:

6. Create a directory named /u363/v12/ctiopi/YYYYMMDD for the new
data, and cd into it.

7. ftp over to the new-ctioa4 machine by
/usr/bin/sftp -o User=observer new-ctioa4.ctio.noao.edu
and use the new-ctioa4 password to get in.

8. Move to where the new data are and copy the files to ctio36 with
mget *.fits .

**********************************************************************
***** SLURP DATA FROM CTIO TO ATL *****
**********************************************************************

1. cd /nfs/recons2/incoming.0.9m

2. mkdir YYYYMMDD corresponding to the observation date and cd into
that new directory

3. Type "huygens" to ensure that you are on the machine huygens, which
is the only one cleared for entry to computers at CTIO.

4. Type "ssh v12@ctio36.ctio.noao.edu" and use the v12 password to get
in. v12% will be displayed on the screen.

5. The data you want are typically found in the /u363/v12/ctiopi/
directory. Type "ls" to see where you are and verify that the data
you are looking for are there. If you don't find it there check
/u361.

***** INSERT SMARTS STEPS NOTED AT END OF FILE IF YOU ARE DOING SMARTS *****

6. cd into the directory for the night you want by typing cd YYYYMMDD

7. Type "ls" to see what's in that directory; there should be files
ending in ".fits" and two files of headers.

8. To see how many entries you will slurp, do a word count by typing
ls | wc

9. Type "rsync -avz YYYY* username@ssh.chara.gsu.edu:/nfs/recons2/incoming.0.9m/YYYYMMDD/"
i.e. "rsync -avz 2016* winters@ssh.chara.gsu.edu:/nfs/recons2/incoming.0.9m/20160314/"

The * will ensure that the headers get transferred, as well as the
.fits data files.

When asked, use your ATL password.

10. When done slurping, type "exit" to come back to ATL.

11. Type "ls -l" to get directory listing. Check the file sizes to
make sure they are all the same.

12. Type "ls * | wc" to make sure that all the files were successfully
transferred.

**********************************************************************
***** PROCESS THE DATA IN ATL *****
**********************************************************************

A) cd into the file that you created for the slurping process
(i.e. /nfs/recons3/SMARTS/20030327) type in "xgterm -sb &". In
xgterm, and type "cl". Then load the 'redpi' package. Log on to the
recons machine, as this will speed up the process significantly.

A1) Sometimes the observer will name the .fits files with *.CTIOPI.*
instead of *.09.*. If this is the case, you will need to rename
the image files.

A2) To do this, type 'ls *.fits >> old'. This will create a file
called 'old' that contains all the slurped image files. copy the
'old' file into another file called 'new'. Then edit the 'new'
file by replacing CTIOPI with 09.

A3) Use 'imrename' in iraf to change all image files names: 'imrename
@old @new'.

B) type "fixnameall"; it will ask you for the root name; the root name is
the string of eight characters that is the date you are working on
(ie, 20070303). This step will generate a list; wait until it's
done to go to next step.

C) type "gofall"

D) type "procall". You will then be asked to enter the bias section; trim
section, ccd readout noise, and ccd gain in turn; the numbers
should be provided. (The numbers in each column are identical all
the way down).

Make sure that the numbers given for you to enter match the ones in
the column. If they do, hit enter; otherwise, enter the numbers
manually in the form specified (including the brackets), and then
hit enter.

Then it will say "now reducing V band star images", "now reducing R
band star images", "now reducing I band star images". Once the
whole process is finished, you will see a "cl>" appear; this lets
you know that you're done with this set of data.

E) "ccdlist *.fits" to make sure that the filter is correct and
included.

**********************************************************************
***** SORT DATA INTO DIRECTORIES *****
**********************************************************************

PHOTOMETRY DIRECTORIES (found at: /nfs/recons3/phot.0.9m/)

A) If there is a photometry night, please make sure that you create a
directory for that night and then COPY all frames into that
directory BEFORE you sort the frames into the parallax directories.

A) If there are new objects and the frames are photometry frames,and
you don't know what the parallax filter is, just copy them to the
photometry directory (no need to create a new parallax directory
for them. A parallax filter probably has not yet been selected for
them.).

PARALLAX DIRECTORIES (found at: /nfs/recons4/CTIOPI/regions/)

A) hselect *.o.fits 'ra, $I, object' yes >> sort.file

B) If there are objects that are not ours (they are not on the
observing lists): EXO planet data go into exoplanet.dir under
special targets; BIN data go into the binary directory under
special targets.

C) Then, after you have sorted all the non-parallax frames where they
are supposed to go, type 'astrometry.processor sort.file'. This
will sort all the frames into the parallax directories.

D) If the directory does not exist, the code will ask you if you want
to create the directory and then will sort it into that newly
created directory if you reply 'yes'. If you reply 'no', it will
ask you for an alternative directory destination. If you simply
hit enter, it will do nothing.

E) The code might also complain if the name of the directory contains
capital letters. If this is the case, simply rename the directory
so that all letters are lower case.

Once the frames have been sorted, we move the entire directory (with
the raw frames only -- no .o.fits files should be left in it, unless a
photometry night was begun and then scrapped) to
/nfs/whiting1/CTIOPI/SMARTS.raw.'year'

**********************************************************************
***** UPDATE THE REDUCTION LOG *****
**********************************************************************

/nfs/morgan3/thenry/public_html/CTIOPI/protected.dir/master.astr.status

**********************************************************************
***** SMARTS SPECIFIC STEPS (somewhat obsolete) *****
**********************************************************************

7) Check to see that the data are in the expected directory. This
step can usually be omitted as observers typically store the data
here, but if there is ever a problem with the next step, come back
to this one.

cd /ua45/v12/"YYYYMMDD"-"PI NAME"/ (i.e., 20090402-Ciardi)

4) Return to original directory and run smartsproc script.

./smartsproc (the './' is not explicitly necessary)

5) Answer questions using all lower case.

PI name: ciardi
Date (YYYYMMDD): 20090402

6) If it doesn't barf, all is well. Log into IRAF from the current directory.

7) Change directory to where the input and output file names were
written (and where the fits data will be stored).

cd /u363/v12/SMARTS/"PI NAME"/"YYYYMMDD"/
(i.e., /u363.etc/ciardi/20090402/)

wfits @infile @outfile

8) Once all data are written to fits, generate an info file

hselect *.fits $I,object,filters yes > "YYYYMMDD"."PI NAME".inf
(i.e., 20090402.ciardi.inf).

9) Exit IRAF, go into directory data are stored, and ftp back to GSU server

logout

cd /u363/v12/SMARTS/"PI NAME"/"YYYYMMDD"/

rsync -avz 2009* "your
username"@ftp.chara.gsu.edu:/ftp/pub/SMARTS/"PI
NAME"/"YYYYMMDD"/

(i.e., rsync -avz 2009*
subasavage@ftp.chara.gsu.edu:/ftp/pub/SMARTS/ciardi/20090402/)

(NOTE: if the PI name does not exist on the GSU server, this will
barf. You'll need to log onto joy and change directory to
/ftp/pub/SMARTS/ and add the PI name in that directory).

Check the slurping process every few minutes as you will undoubtedly
lose the link and have to kill the process and up arrow to restart the
rsync command.

That's it!