CATALOG UPDATING COOKBOOK

version 08 MAY 2014 by Jen Winters

version 08 MAY 2015 by Todd Henry (slight modifications)

This is used to update the astrometry catalog for CTIOPI.

                                                     
*******************************************************************************
UPDATING THE CTIOPI ASTROMETRY CATALOG
*******************************************************************************

---------------------------------------------------------------------------
0. WHAT GOES IN THE CATALOG
---------------------------------------------------------------------------

   All parallax and photometry frames.

   All WIDEMS frames, even if defocused. Note as "ovo" instead of
   "ov." in the filter column.

   All GRKTRO/PLUTO frames, even if FULL chip.  Note as "xxo" instead
   of "xx." in the filter column.

---------------------------------------------------------------------------
1. PREPARE HEADERS (if needed)
---------------------------------------------------------------------------

A. Make sure you have all of the "long" headers for the nights of a
   run in a directory, typically named "2015.0416.long" (you will
   rename files into the format 20150416).

B. If you do not have the long headers, you can generate them by using
   the following command in IRAF: imhead 2015*.fits l+ > 'date'

C. If you obtained the headers via email, each one will have things
   added on top that you must get rid of at this point. Delete those
   rows all the way down to where the data begins.  Example: if you
   export the email to a directory, it will look like this:

     From v12@ctioa4.ctio.noao.edu Sat Jul 15 03:59:19 2006
     Date: Sat, 15 Jul 2006 03:58:58 -0400 (CST)
     From: Visitor #12 
     To: winters@astro.gsu.edu
     Subject: 20060714.09.headers.long

     20060714.09.001.fits[1118,1024][ushort]: bias
     No bad pixels, min=0., max=0. (old)
     Line storage mode, physdim [1118,1024], length of user area 2997 s.u.
     Created Sat 03:55:10 15-Jul-2006, Last modified Sat 03:55:10 15-Jul-2006
     Pixel file "20060714.09.001.fits" [ok]
     BSCALE  =       1.0000000000E0  /  REAL = TAPE*BSCALE + BZERO

   and so on. You must get rid of everything including blank rows so
   that the first line of the header reads:
   "20060714.09.001.fits[1118,1024][ushort]: bias"
   
D. Check that you have the frame counts for each night.

---------------------------------------------------------------------------
2. ADD NIGHTS FOR THE RUN TO THE OLD CATALOG
---------------------------------------------------------------------------

   All of the real work is going to happen in 

        /recons/disk4/CTIOPI/catalog.dir

   Open two terminal windows ready to work there.

   The file "ccdlog0.9" in the directory "data" is the fundamental
   form of the catalog from frame #1 until the present.

E. In terminal 1, 'cd data' to check that 'ccdlog0.9' is there
   (/nfs/recons4/CTIOPI/catalog.dir/data/), is sorted by frame number
   and that it has the correct number of old frames.

F. In terminal 2, copy the first (chronologically) night's headers
   into the directory /nfs/recons4/CTIOPI/catalog.dir/unprocessed.

   Make sure the name is just 20150416, not 2015.0416.long

   It is VERY IMPORTANT that you work chronologically because the
   program assigns frame numbers to the nights added and these must be
   in the correct sequence.

G. In terminal 1, convert the long headers into catalog format:

   cd /nfs/recons4/CTIOPI/catalog.dir and 
   type "./excatv3" --- v3 as of June 2016

   You should get a message that says:
   
   "This command takes a bit of time--please be patient.
   Splitting raw email files into single-frame-per-file format.
   Processing each frame.
   ... (many numbers here)
   Next framenumber for telescope 0.9 will be: 92131"

   After a few seconds, you should get another message:
 
   "Please open another terminal window and examine these files:

   /nfs/recons4/CTIOPI/catalog.dir/tmp/ccdtmp0.9

   Ready to insert this data into the CCD catalogs and append it to
   the CCDLOGS for this telescope(s)?  Type "y" to proceed or "n" to
   exit now:"

   !!!!!!!!!! DO NOT TYPE y OR n, YET !!!!!!!!!!
 
H. In terminal 2, check the format of catalog data for that night:

   cd /nfs/recons4/CTIOPI/catalog.dir/tmp
   emacs "ccdtmp0.9"

   1. Check the number of frames.

   2. Check names and types of frames (pi vs. phot vs. other) by
      comparing to short headers.

   3. Change "o" filters to "ov" filters (line up v with r and i).

   4. Next to the filter column, add a column to designate the frame
      type.  We generally adopt up to 10 focused frames nearest
      transit per night (e.g. 25 MANTIS frames have 10 with . and 15
      with o):

      . = good for pi
      b = bad frame (a terrible frame unusable for anything)
      d = dropped 
      o = other target (someone else's special target or a non-parallax
               object --- e.g. DELTAM, GRKTRO, MANTIS, WIDEMS, etc.)
      p = photometry
      r = refraction
      s = standard star/field for photometry
      x = different from a bad frame --- it might be useful in the
               future (e.g. done for pi, but in wrong filter)

   Note: If, in the process of adding the codes, the far right columns
   get moved to the right by one space, you need to make sure you
   remove this blank column.  Otherwise, the program will crash.

I. In terminal 1, type "y" (once everything is looking good) and hit
   enter.  You will then get a message that says:

     Update complete.
     run "./rebuildcat 0.9" *** AFTER EACH NIGHT ***

J. You may proceed with adding additional nights.  Start with the next
   (chronologically) night's header file and go through steps E
   through J.  Do this with each night until you are all done.

---------------------------------------------------------------------------
3. MAKE THE FINAL CATALOG ALL SHINY AND NEW
---------------------------------------------------------------------------

K. After all of the nights have been entered for the catalog update
   you are doing, you should cd into
   /nfs/recons4/CTIOPI/catalog.dir/data.  There is a file called
   'ccdlog0.9' and this is the catalog in its raw form (i.e.
   chronological order).  You should open that file and look at the
   nights you have added to make sure everything looks ok.  You will
   have already done this after processing each night, so here you are
   just glancing to make sure you don't have any major errors
   (e.g. nights accidentally added twice.)  Make any corrections in
   this file.

   This file MUST end in ONLY ONE BLANK LINE.  Otherwise, the
   rebuildcat command will die.  The ccdlog must also be sorted by the
   far left column (the frame number) in order for the numbering
   sequence to be correct and in order for there to be no duplicate
   numbers.

L. Make any tweaks to names or frame codes.  The link to the tweaks
   page is found directly beneath the link to the Astrometry Catalog
   on the protected page.  It helps to sort the 'ccdlog0.9' text file
   by object name to make these modifications.

   *** RE-SORT BY FRAME NUMBER WHEN YOU ARE DONE MODIFYING. ***

M. When you are ready to rebuild the catalog, you should go to the 1st
   terminal (/nfs/recons4/CTIOPI/catalog.dir), type "rebuildcat 0.9"
   (NOT the "./rebuildcat 0.9" that the code prompted you for earlier)
   and hit enter.  It will take a few seconds to rebuild the catalog.

N. You should now cd into data, then cd into ccd0.9
   (/nfs/recons4/CTIOPI/catalog.dir/data/ccd0.9)

O. Type in "cat ccd* >> catalog.astr.YYYY.MMDD".  This will
   concatenate the file and produce the final catalog for posting.

P. Copy this file into the protected directory, add the name at the
   top of the file, and link it to the weblink for the RECONS group to
   see.

*******************************************************************************
   NOTES
*******************************************************************************

1. For objects that have both parallax and photometry frames, you need
   to check to see which is the chosen parallax filter.  The
   astrometry observing list is a good place to find this information.

2. Check that the names for the stars are correct.  Star names should
   all be standardized (e.g. "LHS2112").  If you see a name like
   "E2mosit", make a note of it.  It will have to be changed later.
   If there are several "LHS2212AB" entries and then one "LHS2212" by
   itself, change it so that it says "LHS2212AB" also.

   For the most part, knowledge of star names will come with time, but
   in the meantime, things that may tip you off to the wrong names
   are:
   
   -- a short list of frames.  The stars will all appear (unless you
      sort them) with breaks in between the different stars.  (see
      above instructions for more on this).

   -- If you suspect that a name may be wrong, refer to the protected
      ctiopi page and check the astrometry and/or photometry observing
      list(s).

3. In each of the individual files for the data, the second to last
   column has a number that will be either negative or positive. A
   negative number tells you that the frame was taken in the evening.

*******************************************************************************
*****                   A TIDY CATALOG IS A HAPPY CATALOG                 *****
*******************************************************************************