.. _intro_proc_example:

Using :program:`sh_gamit` and :program:`sh_glred`
=================================================

Now that we've described the program flow and control files, we'll take you through a simple application of :program:`sh_gamit` and :program:`sh_glred` using the small network in Southern California previously provided (prior to 10.7) in :file:`~/gg/example/`.
The command for :program:`sh_gamit` issued at the project-directory level (:file:`2000/`) is

.. code-block:: console

   $ sh_gamit -expt scal -d 2000 034 035 036 -noftp -pres ELEV -copt x k p -dopt c ao >& sh_gamit.log

In this example, we start with all of the data to be processed already present in the :file:`rinex/` directory.
If you wanted to add additional data available from a global or regional archive, you would specify the site names in :file:`sites.defaults`, enter the URL information (if not already present) into :file:`~/gg/tables/ftp_info`, and remove the :option:`-noftp` option from the call to :program:`sh_gamit`.
As a result of the command shown, :program:`sh_gamit` will execute for each day the following steps, noted in the screen output redirected to :file:`sh_gamit.log`:

#. Assign parameters for program flow, giving precedence first to the command-line arguments, then to the parameters set in :file:`process.defaults` and :file:`sites.defaults`, and then to default assignments within :program:`sh_gamit` itself.
   In this case, the command-line entry :option:`-noftp` overrides the default to search archives for requested or updated observation, navigation, and EOP files;
   and the command-line entries for which files to compress or delete at the end of the run override those set in :file:`process.defaults`.
#. Create the day-directory and/or standard directories which do not yet exist (all of these are already present in the :file:`~/gg/example/` directory).
#. Link into the day directory (:file:`034/`) the standard tables (see script :program:`links.day`) and the RINEX files that contain data for the specified interval (00:00–24:00 as set in :file:`process.defaults`).
#. Download orbital SP3 files from a global data center and create GAMIT g-files using script :program:`sh_get_orbits`.
#. Run :program:`sh_upd_stnfo`, which invokes program :program:`mstinf` to update :file:`station.info` from the RINEX headers.
   (It is recommended that this step be skipped by setting :content:`xstinfo {<expt>} all` in :file:`sites.defaults`.)
#. Run :program:`makexp` to create the input files for :program:`makex` (:file:`scal.makex.batch`) and :program:`fixdrv` (:file:`dscal0.034`).
#. Run :program:`sh_check_sess` to make sure that all of the satellites included in the RINEX observation files are present in the navigation file (:file:`brdc/brdc0340.00n`, previously downloaded at MIT from an IGS archive) and in the g-file (created previously at MIT from an IGS SP3 file).
#. Run :program:`sh_makej` to create a j-file of satellite clock estimates from the SP3 file (default) or navigation file.
#. Run :program:`makex` to create x-files (observations) and k-files (receiver clock estimates) using phase and pseudorange data from the RINEX observation files, broadcast ephemeris from the navigation file, and satellite clocks from the j-file.
   A record of :program:`makex`, showing the data found and any problematic data encountered is written to :file:`scal.makex.infor`.
#. Run :program:`fixdrv` to create the batch file for GAMIT processing.
   Though not used directly, :program:`fixdrv` also reads the k-file of episodic clock values and fits a first-order polynomial to them as a crude check for jumps and rapid drifts in the receiver clock (:file:`fixdrv.out`).
#. Execute the batch run to generate a tabular orbital ephemeris (:program:`arc`), model the phase observations (:program:`model`), edit the data (:program:`autcln`), and estimate parameters (:program:`solve`), a sequence completed twice in order that :program:`autcln` may operate on flat residuals and that the final adjustments in :program:`solve` are well within a linear range.
   A record of this run is not written to :file:`sh_gamit.log` (to save space) but is recorded in :file:`GAMIT.status`, :file:`GAMIT.warning`, and :file:`GAMIT.fatal` in the day directory.
#. Save the cleaning summary (:file:`autcln.post.sum`) to :file:`autcln.post.sum.scal` (potentially for archiving, though that's not done for this example) and write key information from :program:`model` and :program:`solve` to the :file:`HISTORY` file, which, unlike all other files in the day directory, is appended rather than overwritten in reruns so that a record of previous runs is maintained.
#. Create sky plots of phase residuals and plots of phase vs elevation angle using the DPH files written by :program:`autcln`;
   if the Ghostscript program is available, translate the plots from PostScript to PNG and move them into :file:`figs/`.
#. Invoke :program:`sh_cleanup` to delete or compress files as specified by :option:`-dopt` and :option:`-copt`.

**The most common problems with** :program:`sh_gamit` **are missing or incorrect receiver and antenna information ("metadata") in** :file:`station.info` **and the loss of data due to bad tracking or bad coordinates.**

There are two approaches you can take to providing metadata to GAMIT.
If you have RINEX files for which you know all the headers are correct, you can have :program:`sh_gamit` invoke :program:`sh_upd_stnfo` to update :file:`station.info` for each day using the header entries (no update will be made if the :file:`station.info` entry is present and matches the RINEX header).
If the antenna height information is correct but the receiver and antenna names not IGS standard, you can still use this feature if you put into :file:`~/gg/tables/guess_rcvant.dat` a substring that will uniquely match what appears in the header.
If you know that all of your receivers and/or antennas are of the same type, you can force their use by specifying them as :content:`ant default` and :content:`rcv default` in :file:`guess_rcvant.dat`.
If you expect :program:`sh_gamit` to generate :file:`station.info` entries from the RINEX header information, you should review the antenna and receiver names used in all of the headers before you start (e.g., by :command:`grep`'ing on the RINEX files for :content:`REC #` and :content:`ANT #`).
Antenna heights can also be problematic.
In the RINEX standard, the header value is supposed to be a vertical height to antenna reference point (ARP), but often a slant height is actually given in the file.
If the latter is the case, then :content:`stinf_slthgt` in :file:`process.defaults` can be set to a height above which the value will be assumed to a slant height to the outside edge of the ground plane.
(Setting :content:`stinf_slthgt` to 0 or a large number will cause all values to be interpreted as direct height measurements.)

If you have *any* doubts about the validity of the RINEX headers, it is better to create (and check!) :file:`station.info` before you start the GAMIT processing.
You can create a file with all of the entries for your survey by running :program:`sh_upd_stnfo` manually with your RINEX files, then edit the file as appropriate.
In creating this file, it is best to start with a template from the current MIT or SOPAC global :file:`station.info` file (available in :file:`incremental_updates/tables/`) so that you can conveniently add entries for continuous stations from the global file.
This template can be header-only, but the preferred approach is to start with the full global :file:`station.info` file and use the :option:`-l {<sitelist>}` option of :program:`sh_upd_stnfo` to extract only the stations you need, either from an existing list or one created automatically by :program:`sh_upd_stnfo` from the sites with :content:`ftprnx` in :file:`sites.defaults` and/or the file names in your experiment :file:`rinex/` directory.
Hence, if you have a current copy of the MIT or SOPAC station.info in :file:`~/gg/tables/`, have run run :program:`sh_setup` to copy it into your experiment :file:`tables/` directory, and have edited :file:`sites.defaults` to specify the RINEX files you want from a remote archive (using :content:`ftprnx`), you can run (in :file:`tables/`):

.. code-block:: console

   $ sh_upd_stnfo -ref station.info -l sd

which will produce :file:`station.info.new` with the global entries for only the :content:`ftprnx` sites.
(If the sites you need from the global :file:`station.info` file are not in :file:`sites.defaults` with :content:`ftprnx`, but are present in the :file:`rinex/` directory, you can use :option:`-l sdrnx` or :option:`-l rnx` in the :program:`sh_upd_stnfo` command.)
Then make sure the RINEX files in the experiment :file:`rinex/` directory are not compressed, rename :file:`station.info.new` to :file:`station.info` and run (in :file:`tables/`):

.. code-block:: console

   $ sh_upd_stnfo -files ../rinex/*o

:program:`sh_upd_stnfo` also allows entries for :file:`station.info` to be created from IGS log files or SINEX files.
If you construct :file:`station.info` in advance of your processing, set :content:`xstinfo {<expt>} all` in :file:`sites.defaults` to block any attempt by :program:`sh_gamit` to update the file from RINEX headers.

Bad *a priori* coordinates will result in :program:`autcln` detecting too many cycle slips and deleting all of the data.
A clear indication that this has happened is :content:`Range rms` values greater than about 20 m at the top of the :file:`autcln.prefit.sum` file together with 0 for :content:`DATA AMOUNTS` in the same file.
When this happens, you should check the experiment :file:`tables/lfile.` for the source of the coordinates used.
If *a priori* coordinates for a station are not available in the L-file (or :file:`.apr` file) from previous processing, :program:`sh_gamit` will by default invoke the :program:`sh_rx2apr` script to perform a pseudorange solution.
Coordinates good to 10–20 m can usually be obtained from the data at the station of interest (better if selective availability is off), but the preferred approach is to perform the solution differentially, using also a RINEX file from an IGS station with known coordinates.
To make sure this happens, you should specify :content:`ftprnx` in :file:`sites.defaults` and have present in the :file:`rinex/` directory or available via download from an IGS data archive the RINEX files for each day from one or more IGS stations.
(You can use this setting to get a differential pseudorange solution even if you have :option:`-noftp` specified for :program:`sh_gamit` provided you copy the IGS data into the :file:`rinex/` directory in advance of the run.)
If a differential pseudorange solution was used and you still have bad coordinates, try executing :program:`sh_rx2apr` manually with data from each of several days and compare the coordinates to see if the day you used originally was an anomaly because it was short or had bad pseudorange data.
To by-pass :program:`sh_rx2apr` and use the coordinates from the RINEX header, set :content:`use_rxc = Y` in :file:`process.defaults`.
This option should be used only if you know that the header values are always present and accurate or if you have RINEX 3 files, which the programs :program:`svpos` and :program:`svdiff` invoked by :program:`sh_rx2apr` will not yet support.
Note that a large adjustment of coordinates, due to bad data or a short session, on one day can cause problems with the next day since :file:`lfile.` in the experiment :file:`tables/` directory will be updated.
You can avoid this update by copying into the :content:`aprf` file specified in :file:`process.defaults` any site coordinates that you know to be good;
the L-file will be initialized with these values as processing begins for each day.

:program:`sh_gamit` can also fail if it is unable to download required global RINEX files or orbital information from an IGS archive (CDDIS or SOPAC if not otherwise specified).
The :file:`GAMIT.fatal` message will usually make clear what file is missing.
In this case, check the connection to the archive manually and restart the processing.

For the most part, you can rerun a day after a failure with simply a remedy of the detected problem (e.g., a bad :file:`station.info` entry).
However, in some circumstances old corrupted files will be used:
(a) Unless you specify :option:`-remakex Y` in the command line, any existing x-file in a directory will be used again and the script assumes that there is a valid :file:`station.info` entry for this file (if not, the process will fail).
(b) Any existing RINEX file linked in the day directory will be assumed to exist.
If the link is now empty because you have renamed or removed the file in the remote directory, this may not be detected correctly on all systems.
(c) A previously added :file:`station.info` entry will be used (and not replaced) if it applies to the day being processed.
(d) Coordinates in the L-file will be used if they exist (so if the entry has been corrupted it should be removed and/or correct coordinates put in to the :file:`.apr` file).

Descriptions of how to run :program:`sh_gamit` with sessions crossing day boundaries can be found in Section 6.3 of the `GAMIT Reference Manual <http://geoweb.mit.edu/gg/docs/GAMIT_Ref.pdf>`_.

To generate time series from the GAMIT runs for the three days in the example, type at the project-directory level (:file:`2000/`)

.. code-block:: console

   $ sh_glred -expt scal -s 2000 034 2000 036 -opt H G T >&! sh_glred.log

The script will execute the following steps, noted in the screen output redirected to :file:`sh_glred.log`:

#. Search all day directories between :file:`034/` and :file:`036/` for GAMIT h-files containing the substring :content:`scal` and run :program:`htoglb` to convert these to binary h-files for :program:`glred`.
   Since each GAMIT h-file contains two solutions, one with ambiguities estimated as real numbers ("biases free") and one with ambiguities resolved ("biases fixed"), :program:`htoglb` will create two binary h-files, with extents :file:`.glr` (GAMIT loose free) and :file:`.glx` (GAMIT loose fixed), respectively.
   They are stored in the :file:`glbf/` directory and named with the year/month/day/hr/min, e.g :file:`h0002031200_scal.glx` for day 034.
#. Generate in the :file:`gsoln/` directory a :file:`.gdl` file (h-file list) for each day (:file:`.glx` is default).
#. Run :program:`glred` for each day, using commands of the form

   .. code:: console

      $ glred 6 globk_scat_00034.prt globk_scal_00034.log globk_scal_00034.gdl globk.cmd

   The :file:`.log` and :file:`.org` files provide a record of the output.
#. Run :program:`sh_plot_pos` to invoke program :program:`tssum` to create PBO-format :file:`.pos` files from the coordinates of each site on each day in the :file:`.org` files and invoke GMT to plot the time series.
   (Specifying :option:`E` instead of :option:`T` on the command line will use the older program :program:`ensum` to generate :file:`mb_` files for plotting with :program:`sh_plotcrd`.)

This task could be accomplished in individual steps by running :program:`htoglb` directly with wild cards specifying the day directories, creating a :file:`.gdl` file in :file:`gsoln/` using :console:`ls ../glbf/h*glx`, running the program :program:`glred` (rather than the script :program:`sh_glred`) with this :file:`.gdl` file, and running :program:`sh_plot_pos`.
In fact, for velocity solutions and time series using combined files, you will need to use this approach (see Chapter 4).
The advantages of the script are threefold:
1) fewer commands,
2) it allows easy daily combination of local h-files with those from an external processing center (e.g. MIT or SOPAC), using the :option:`LA`, :option:`LB` and :option:`LC` options (see :numref:`intro_prod_combine`); and
3) it allows easy aggregation of days into weekly or monthly combined h-files, using the :option:`-ncomb` option (see the :program:`sh_glred` help by typing the command without arguments).
You can mix using :program:`sh_glred` with running :program:`glred` directly as long as you keep in mind two differences:
1) :program:`sh_glred`, like :program:`sh_gamit`, is launched from the project directory but executes :program:`glred` within the :file:`gsoln/` directory whereas :program:`glred` is launched directly from within the :file:`gsoln/` directory itself; and
2) :program:`sh_glred` creates a :file:`.gdl` file for each day whereas running :program:`glred` itself efficiently requires creating a single :file:`.gdl` file containing the h-files for all of the days.
In a :file:`.gdl` file, the :content:`+` symbol is used to indicate to :program:`glred` that h-files are to be combined before performing the solution.
For example, using the file

.. code-block:: text

   ../glbf/h1211021200_pan1.glx 1.0 +
   ../glf/h1211021200_pan2.glx 1.0 +
   ../glbf/h1211021200_pcnw.glx 1.0 +
   ../glbf/nmt17125.e.glb 1.0
   ../glbf/h1211031200_pan1.glx 1.0 +
   ../glbf/h1211031200_pan2.glx 1.0 +
   ../glbf/h1211031200_pcnw.glx 1.0 +
   ../glbf/nmt17126.e.glb 1.0

:program:`glred` would combine all four h-files for day 2012-11-02 before performing a solution, and then do the same thing for the four files for day 2012-11-03.
(Running :program:`globk`, rather than :program:`glred`, would combine all of the files listed into a single solution whether or not the :content:`+` are present).

Other options of :program:`sh_glred` specify the downloading of data from a remote archive (:option:`F`), removal of old h-files from the :content:`glfpth` directory (:option:`R`), and compressing the h-files at the end of the run (:option:`C`).
Type :program:`sh_glred` without arguments to see all options.

To avoid overwriting useful h-files or using obsolete ones, it is important to keep in mind the precedence rules of the script.
For local data (:program:`sh_gamit` day directories), specifying the :option:`H` option will force :program:`htoglb` to be rerun for all directories within the time span indicated, whether or not a binary file or link exists in the :file:`glbf/` directory.
Omitting :option:`H` will cause no new binary files to be created, so it is not possible to retranslate only a selected group of ascii H-files.
This is not an important limitation, however, because :program:`htoglb` runs quickly.
For global ascii h-files downloaded from SOPAC, setting :option:`H` will also force :program:`htoglb` to be rerun on any ascii H-files present or linked (by :option:`LA`) in the H-file (:file:`glbf/`) directory, but you can safely set :option:`F` since the script will not re-download any remote (ascii) H-files that are present.

Note that most of the shell scripts called by :program:`sh_gamit` and :program:`sh_glred` can be run stand-alone for specific processing tasks.
The most useful of these are :program:`sh_make_rinex` (templates for running :program:`teqc` to translate raw data files), :program:`sh_get_nav`, :program:`sh_get_rinex`, :program:`sh_get_orbits`, :program:`sh_sp3fit`, :program:`sh_update_eop`, :program:`sh_link_rinex`, :program:`sh_oneway` (to get sky plots from :file:`DPH` files), and :program:`sh_get_hfiles`.
Type the name of the script without arguments to see the documentation.