VLBA data calibration pipeline processing (using AIPS)

VLBA data calibration pipeline procedure (VLBARUN)

General information on the VLBA, how to propose for observations with the VLBA, how to schedule VLBA observations, and how to reduce VLBA data can be obtained through the VLBA and AIPS homepages. Two extensive tutorials on VLBI data reduction in AIPS have been prepared, complete with data sets, introductory material about AIPS, and detailed instructions. They are simple VLBA project including self-calibration and spectral-line VLBA project plus astrometry. Users new to, or rusty in, VLBI data reduction are encouraged to try these tutorials. Appendix C and Chapter 9 of the AIPS CookBook are also recommended.

This web document is intended to provide information on the VLBA data calibration pipeline. This is a set of AIPS procedures found in a single `RUN file' which executes an automated process to provide a best effort a-priori calibration of the data. It includes instrumental calibration as well as some automated flagging of bad data points and atmospheric phase calibration (i.e. "fringe-fitting"). In order to asses the quality of the data calibration, plots and maps are produced but not distributed.

Topics in this document include:

  Pipeline ideology What is the VLBA data calibration pipeline?
  Copyrights, etc. Restrictions on the use of the VLBA data calibration pipeline
  Current status Current capabilities and limitations of the VLBA data calibration pipeline
  Scripts & releases Updates of the VLBA data calibration pipeline scripts
  How to pipeline How to pipeline the data for your project
  Multiple phase centers Pipelining secondary phase-stopping center data
  Last but not least The acknowledgments section

What is the VLBA data calibration pipeline?

The VLBA data calibration pipeline is a 'run file' named VLBARUN in AIPS which is primarily based on the AIPS VLBAUTIL procedures. The VLBAUTIL procedures are described in Appendix C of the AIPS Cookbook. The VLBA data calibration pipeline performs calibration tasks and makes lots of plots and some simple images to find out whether the calibration has worked. The plots may be made available on an html page of the user's choosing to enable easy review. The VLBA data calibration pipeline is not intended to produce high fidelity images. The VLBA data calibration pipeline will be available to AIPS users as part of the AIPS distribution and the Midnight Job.

At the moment the VLBA data calibration pipeline works reasonably well for simple VLBA-only experiments correlated in Socorro, at frequencies between about 1 and 15 GHz, both for continuum and spectral line experiments. It has been successfully run on higher and lower frequencies, but we're not advertising that for general use,

At the moment the VLBA data calibration pipeline works according to the flow chart as shown here in ps or in gif.


Restrictions on the use of the VLBA data calibration pipeline

The VLBA data calibration pipeline is an effort to streamline and simplify customary calibration of data from the Very Long Baseline Array (VLBA). The VLBA is operated by the National Radio Astronomy Observatory (NRAO) as a facility of the National Science Foundation (NSF) operated under cooperative agreement by Associated Universities, Inc. (AUI).

The Astronomical Imaging Processing System (AIPS) is NRAO's older, but still current, software package for calibration, data analysis, image display, plotting, and a variety of ancillary tasks on (mostly interferometric, radio-) astronomical data. The VLBA data calibration pipeline is an NRAO product (like AIPS itself), comes as an AIPS 'run file' (with an associated 'help file'), and therefore assumes the terms and conditions applicable to using AIPS; this is mostly the GNU General Public License (GNU GPL). AIPS is copyright (c) 1995 through 2019 by AUI.

To make use of the VLBA data calibration pipeline, you must have a relatively current installation of AIPS on your computer (AIPS version 31DEC19 or later), preferably run the AIPS 'Midnight Job' (MNJ), and follow the instructions as provided in the VLBA data calibration pipeline 'help file'. The `MNJ' is a script, usually in your home area, named do_daily.hostname, where hostname is the name of your computer or, perhaps, localhost. If you do not run the MNJ, you may be restricted to using an older version. Please report any bugs you may find after making sure you are using the latest versions (the bug may already have been fixed). Some options described here exist only in the 31DEC20 version of AIPS.

Note that the VLBA data calibration pipeline comes as is, without warranty or guarantee that it works (although we try to make it work)! [That is, we do not assume any responsibility for whatever happens (probably nothing malignant) in case you decide to use the scripts.]

When you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data were calibrated using NRAO's "VLBA data calibration pipeline" in AIPS.


Current capabilities and limitations of the VLBA data calibration pipeline

The current capabilities of VLBARUN are summarized and then described in detail below. Following that is a list of matters that VLBARUN cannot handle.
Some general operations that the VLBA data calibration pipeline does or can do:
  • Calibrate VLBA data at multiple intermediate radio frequencies for
    amplitude, bandpass, parallactic angle, some instrumental polarization,
    pulse-cal (continuum) or manual p-cal (spectral-line)
    on data sets that include calibration transfer files (post April 1999)
    [or use VLBALOAD first and append the tables with ANTAB before starting]
  • Including some flagging of bad data
  • Application of ionospheric correction with JPL's TEC-models
  • Fringe-fitting of calibrators and strong sources
  • Including phase-referencing schemes for up to 15 target sources
  • Imaging of continuum and spectral-line features
Specific operations performed by VLBARUN:
  • Examine inputs to insure that there is nothing obviously wrong
  • VLBALOAD (optional) loads FITS file from disk and merges redundant table files
  • VLBATECR applies total electron content correction to data < 12 GHz
  • VLBAEOPS applies the EOP (North Pole position) correction
  • UVFLG (optional) to apply flags from an a priori text file
  • TYSMO (optional) to clip bad values from the TY table, replacing them with smoothed good values
  • UVFLG (optional) to flag data below a selected elevation limit
  • VLBACCOR to calibrate amplitude from auto-correlations, smoothing away bad values
  • Calibrate bulk delays from a single scan or time range using either VLBAPCOR (task PCCOR) or VLBAMPCL (task FRING)
  • VLBABPSS to calibrate bandpass shape
  • VLBAAMP to re-calibrate with auto-correlations after application of bandpass (task ACSCL) and then to apply the TY table to the amplitudes (task APCAL)
  • VLBAPANG to correct for the parallactic angle
  • CLCOR (optional) to apply the atmospheric delay zenith dependence and clock error found by DELZN
  • VLBAFRNG to solve for delay and phase of calibrators as a function of time with subsequent application to as many as 15 phase-referencing target sources. Optionally, rates are clipped and time smoothed.
  • CLIP (optional) to flag data above user-specified flux levels
  • RFLAG (optional) one source at a time to flag RFI on a baseline basis
  • IMAGR and FRING (optional) to do self-calibration of calibration sources. Optionally, rates are clipped and time smoothed.
  • SPLIT to apply the calibration and flags generating single-source data sets
  • IMAGR (optional) to make continuum images of calibration sources and/or continuum or spectral cube images of target sources
(Watch this space for updates of capabilities)

Wish list for future implementation, roughly in order of priority:
  • Polarization calibration
  • Apply external calibration tables to file after FITLD with ANTAB (aka 'manual calibration transfer')
Of course any other suggestions for the wish list are very welcome as well.

The VLBA data calibration pipeline is not supposed to handle, or cannot do:
  • Correct for bad observing weather/phase-coherence/RFI/tactic/techniques!
  • Data sets with multiple frequency IDs. They must be split apart before running VLBARUN.
  • Data sets with multiple subarrays. They should be separated before using VLBARUN if that is appropriate. Complex subarray situations, such as are found in astrometric observations,
    are not suitable for VLBARUN.
  • Data from other correlators than the VLBA correlator (ie. in Socorro NM, USA)
    Note: data from the JIVE correlator can be processed with an EVN pipeline.
  • High-fidelity flagging of visibilities in single (and multi-) source files
  • Advanced imaging (multi-field & -resolution, deep cleaning, MEM, etc)
  • Advanced hybrid-mapping (high-fidelity self-cal iterations, tricks & kicks)
  • Apply doppler-tracking for individual spectral line sources (aka CVEL)

Updates of the VLBA data calibration pipeline scripts

VLBARUN received numerous corrections and additions in the 31DEC19 version of AIPS, so we recommend that you obtain this version of the script. The 31DEC19 script might work with older versions of AIPS although certain options (and corrections) in RFLAG will not be present. We do strongly recommend keeping your version of AIPS relatively current by installing the new development release in January and then running the MNJ periodically. Conservative sites might wish to stay a year behind, but start that next year. Some otions described below exist only in the 31DEC20 and later versions of AIPS.

How to pipeline the data for your project

To set up the AIPS environment in your terminal process, source the AIPS LOGIN.[C]SH file found in the AIPS_ROOT directory. Then type $CDTST to find a large number of environment variables including $RUNSYS. If your version of AIPS is up to date, then the RUN file VLBARUN is found in $RUNSYS and its help file in $HLPFIL. You may use these distributed versions simply by invoking them in your AIPS session. If you want to modify them, copy them to some directory of yours and edit away. Then, before starting AIPS, set an environment variable such as MYAREA to point at your disk directory. Then in AIPS enter VERSION = 'MYAREA'. For more information about the use of private disk areas, See section 3.10 (page 3-16) of the AIPS Cookbook.

You may also get your version of VLBARUN from the links below. Put the two files in your disk directory and set the VERSION adverb to use them.

For both the MNJ and newly fetched files, you have to type RUN VLBAUTIL and RUN VLBARUN (in that order and on separate command line) in your AIPS window to make sure you are using the latest 'run files'. If you forget this, either you might end up using an older version, or it will complain about not finding or recognizing the command. The same will happen if you forget to place the 'help file' or 'run files' in the appropriate directories (or misplace them).

After defining the procedure in AIPS, read the 'help file' for its use (type EXPLAIN VLBARUN) and type INPUTS VLBARUN to review its current parameter settings. Invoke the VLBA data calibration pipeline by simply typing VLBARUN; it will check the inputs for possible problems, and if none are found, it will proceed with the operations listed above in that order. Do not forget to check the plots and images it has made, as they may point to specific problems with your data set (dead IF channels, RFI, failing fringe-fitting, etc). If something is definitely wrong, usually the problems are solved by flagging the bad data (in flag/FG table one!) and rerunning the VLBA data calibration pipeline. Other common problems are that you selected a time range for your bandpass calibrator in which it has too few visibilities (e.g. when it is late on source), in which the source is below the horizon for one or more antennas, or the time range does not correspond to the scan on your bandpass calibrator - change either or both. For other problems contact designated AIP. The table below allows you to download the procedures and help file. To view the help file in your browser with links to other help files, click on VLBARUN.HLP

Current VLBA pipeline run file script:VLBARUN.001Put this in your $MYAREA directory
Current VLBA pipeline help file text: VLBARUN.HLPPut this in your $MYAREA directory
Current VLBAUTIL procedures:VLBAUTIL.001You may need this too (in $MYAREA)
A sample html file from a run at L band is shown in file 19OCT291452/vlbarun.html

Note that the html file, message file, and plots are placed in a subdirectory labeled with the date and local time. Note also that the results are generally okay but the data from SC really need closer scrutiny. There are two calibration sources and one target which is not detected in the data from individual days. Note that the full message file from the run is available, on a separate page in your browser, by clicking on the link at the top of the vlbarun.html file.
The inputs to VLBARUN:
-- Load the data from disk --
DATAIN Disk file name
DOUVCOMP Compress data? Use -1, 0 or 1
OUTNAME File name (name)
OUTDISK Working disk with ample space
-- OR existing file --
INNAME Input file name
INCLASS Input file class
INSEQ Input file sequence number
INDISK Disk number for input file
-----------------------------
OPTYPE For CONT, PSEU or SPEC (LINE)
-----------------------------
CLINT CL table interval in minutes.
CHREFANT Reference antenna NAME
TIMERANG Time range of a good bandpass scan, or use
scan# in TIMERANG(1)+rest=0
INVERS PC table to use
-1 => don't use Pulse cals, do manual phase cal
CALSOUR List ALL Fringe fit & CAL sour
FIRST is bandpass calib
SOURCES ONLY FOR PHASE REFERENCING:
Source *PAIRS* to calibrate, START each pair with phase referencing calibrator
If 2nd='*': all non-CALSOUR are phase-referenced to 1st
INFILE Set to DELZN correction file
TECRFILE Local copy of TEC file; '' => download automatically
EOPSFILE Local copy of EOP file; '' => download automatically
---------------------------------------------------
SOLINT Time interval for fringe-fit
---------------------------------------------------
IMSIZE Image size for target SOURCES; =-1 for no images
FACTOR CALSOUR IMSIZE (FACTORx128)
---------------------------------------------------
DOCLIP > 0 -> use CLIP on all sources: gives clip levels
DOFLAG > 0 -> use RFLAG on all sources - clip level is
max (100, DOFLAG) if a clip limit > 100 Jy is desired
TYSMOLEV > 0 -> clip level in TYSMO
ELEVLIM > 0 -> flag elevation < ELEVLIM degrees
INTEXT Input flag commands
SELFCAL > 0 -> do self-cal with IMAGR and FRING on CALSOUR
SMRATES (1) > 0 do SNSMO on rates with SMRATES(1) boxcar (hours)
(2) MWF for clipping rates of width SMRATES(2) (hours)
(3) flag rates > SMRATES(3) (mHz) from median
this edit and smooth on rates is done after the long FRING operations
Hint: when setting directory and e-mail, lower-case letters
can be retained by NOT using the close quote.
DOPLOT <= 0 no plots; =1 some plots; >1 huge number of plots
OUTFILE Directory for html and plot files: if you want an html file with
plots please set the dir where the html and plot files can be put.
OUTTEXT E-MAIL ADDRESS: if you want an e-mail when the script is complete.
BADDISK Disks to avoid for scratch

Pipelining secondary phase-stopping center data

The DiFX correlator is capable of correlating a single antenna pointing position at numerous phase-stopping positions within the beam. These data are written into one "primary" file containing the target's central phase-stopping position plus all the calibration data which are taken with only one phase stopping position. "Secondary" FITS-IDI files are written for each of the additional phase-stopping positions. A pipeline RUN file called VLBAMPHC has been developed in 31DEC19 to use the output of VLBARUN on the primary file to assist in calibrating the secondary files.

Many of the inputs to VLBAMPHC are very similar to those of VLBARUN. The main difference is that one specifies the primary data set (assumed to be on disk) in the IN2NAME et al. adverbs. The steps done by VLBAMPHC are then:

Specific operations performed by VLBAMPHC:
  • Examine inputs to insure that there is nothing obviously wrong
  • VLBALOAD (optional) loads FITS file from disk and merges redundant table files
  • TACOP copies the highest version bandpass (BP) and calibration (CL) tables plus the version 1 flag (FG) table from the primary file. The source number in each of these tables is reset to 0 to have the table values apply to any source number.
  • CLIP (optional) to flag data above user-specified flux levels
  • RFLAG (optional) to flag RFI on a baseline basis
  • SPLIT to apply the calibration and flags generating A single-source data set
  • IMAGR (optional) to make continuum or spectral cube images of the target source
VLBAMPHC run and help files are distributed with AIPS and updated by the MNJ. The help file may be viewed in your browser with links to other AIPS symbols by clicking on VLBAMPHC.HLP. Copies of the run and help files are also available for download from the following links.
Current VLBA secondary pipeline run file script:VLBAMPHC.001Put this in your $MYAREA directory
Current VLBA pipeline secondary help file text: VLBAMPHC.HLPPut this in your $MYAREA directory

The acknowledgments section

Again, when you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data were calibrated using NRAO's VLBA data calibration pipeline in AIPS." Thanks!

We on our side acknowledge the efforts of the Cast of AIPS, in particular Eric Greisen, for making and maintaining the AIPS-tasks and environment needed to reduce VLBI data in AIPS over the years and Bill Cotton for writing many of the tasks used on VLBI data. Chris Flatters started collecting the tasks into the VLBAUTIL procedures. This project was continued and completed by Amy Mioduszewski. Lorant Sjouwerman then developed the VLBAPIPE pipeline script making heavy use of the VLBAUTIL procedures. Amy Mioduszewski then developed VLBARUN using VLBAPIPE as a starting point. Most recently Eric Greisen cleaned up minor problems in VLBARUN and added a variety of options. He also developed the secondary pipeline VLBAMPHC
Comments, suggestions, and bug reports should be sent to the designated AIP.


Pipeline ideology / Copyrights, etc. / Current status / Scripts & releases / How to use / Multiple phase stopping centers / Acknowledgments

VLBA Home Page
The VLBA data calibration pipeline processing webpage originated June 5th, 2002 by Loránt Sjouwerman.
October 2019 version written and maintained by Eric Greisen daip@nrao.edu.