VLBA Pipeline

VLBAPIPE and VLBARUN are two similar procedures which use the VLBA calibration procedures (from VLBAUTIL) and some logic to calibrate and image VLBA data. They both attempt to make intelligent decisions on defaults, so they can be run fairly automatically if the names of the sources are known. If desired, VLBAPIPE and VLBARUN will produce diagnostic plot files and write them to disk, creating an HTML file to ease examination of these files. Images will be produced but only one phase-only self-cal is done (if requested), so the images should be considered diagnostic in nature.

VLBAPIPE and VLBARUN do all the calibration steps described in §C.1 and the next few sections but in a different order. They do not do polarization calibration. They offers options to use CLIP and/or RFLAG to do flagging. They also offer options to flag data at low elevations and to try to clean up bad values in the system temperature table. Although VLBAPIPE and VLBARUN are intended for simple VLBI observations, the addition of non-VLBA antennas should work if all the non-VLBA system temperature and gain curve information are loaded into the first TY and GC tables.

The two piplines use essentially the same input adverbs, except for the meaning of DOFLAG parameter. Sample inputs for procedure VLBAPIPE are:

> RUN VLBAUTIL C R

to acquire the procedures used by VLBAPIPE.

> RUN VLBAPIPE C R

to acquire VLBAPIPE.

> INDISK n ; GETN ctn C R

to specify the input data file.

> OPTYPE ’CONT’ C R

to say this is a continuum data set.

> CLINT 0 C R

to use default.

> CHREFANT ’FD’ C R

to set reference antenna to Fort Davis.

> CHINC -1 C R

to have the procedure determine an appropriate channel increment for FRING.

> TIMERANG 0 C R

to have VLBAPIPE determine a good instrumental delay calibration scan.

> INVER 0 C R

PC table to use, -1 if you want to force manual phase cal.

> CALSOUR ’CAL-BAND’, ’CAL-PHASE’, ’CAL-AMP’ C R

to list calibrators, bandpass and initial FRINGcalibrator must be first.

> SOURCES ’CAL-PHASE’, ’TARGET’, … C R

to list phase referencing and target pairs.

> INFILE ” C R

set if you want to apply a zenith delay file produced by DELZN.

> SOLINT 0 C R

to use 1 minute averages in FRING.

> IMSIZE 512 C R

to make images and specify size of target images.

> FACTOR 0 C R

to make calibrator images 128x128.

> DOCLIP fp,fc C R

to run CLIP to flag a sample when the parallel-hand flux is above |fp| or the cross-hand flux is above |fc|. If fp is positive, the clip is done after amplitude calibration; if negative, it is done before the bandpass and amplitude calibration. The appropriate values for fp and fc are different in these two cases.

> DOFLAG 1 C R

to run RFLAG on each source individually.

> TYSMOLEV 100 C R

to clip system temperatures above 100 K with median-window filtering at 10 K and replacing the clipped values with nearby good ones.

> ELEVLIM 15 C R

to flag out all data for which the elevation is < 15 degrees.

> SELFCAL 1 C R

to do one pass of self-cal, phase only on each calibrator prior to the final SPLIT¿

> GET2N ctn2 ; NMAPS=1 C R

to use a model for the bandpass calibration.

> BPASS5 -2 C R

to normalize bandpass scans before the fit. Default is +1 to normalize the bandpass functions themselves without normalization of the data.

> BPSOLINT 0 C R

to make a solution for each calibration scan. BPSOLINT=-1 averages all calibration scans.

> DOBAND 3 C R

to interpolate the bandpass over time. DOBAND=1 (the default) averages all bandpass records.

> CALTASK ’CALIBS’ C R

to use FRING for the initial time series calibration but use CALIB for any self-cal. Other choices are ’CALIB’ and ’FRING’ (the default) to use the specified task for all time series calibrations.

> DOPLOT 1 C R

to make some diagnostic plots. Use level 2 only if there are serious issues; it generates a lot of plots.

> OUTFILE ’/directory/name C R

to specify directory for output plots. If this is set, plots are written out from and organized in an HTML file for easy viewing. The plots, HTML file, and message file contents are put in a dated subdirectory below the specified directory.

> OUTTEXT ’email@somewhere.edu C R

to specify an e-mail address if the user wants to be notified when the pipeline is done.

> BADDISK 0 C R

to specify which disks to avoid for scratch.

> VLBAPIPE C R

to run the procedure.

The order of operations differs between the two pipelines. The most significant difference is that FRING is run before BPASS in VLBAPIPE. This should insure that the bandpass solutions, when there are multiple primary calirator scans, remain phase coherent. The only adverb that differs between the two pipeline scripts is DOFLAG. In VLBAPIPE, the pipeline will shut down after running RFLAG if 0 < DOFLAG < 10 to allow you to rerun the pipeline using the RFLAG flag tables. Presumably, the calibration in the second pass should be better. If DOFLAG≥ 10 in VLBAPIPE or DOFLAG> 0 in VLBARUN the pipeline invokes RFLAG and then continues to the optional self-cal step followed by SPLIT and IMAGR. VLBARUN uses the value of DOFLAG as an initial clip level in RFLAG but only if it is greater than 100. VLBAPIPE does not use the initial clip option in RFLAG, leaving that to the DOCLIP adverb.

It is a good idea to examine your TY table with SNPLT before running VLBAPIPE; this will tell you what a reasonble upper limit to T_sys will be and whether there are bad values in need of correction. SNRMS provides a summary of the statistics of tables such as the TY table. It’s maxima and rmses will make problems with the table very clear. Note that editing with CLIP is always a good idea. RFLAG appears to work well at low frequencies but to be overly enthusiastic at C-band and above. Note that the flagging is done after all calibration steps, so you may examine your data with and without the new flags to determine their correctness. RFLAG is run using baseline-dependent clip levels and so should work with arrays containing high-sensitivity telescopes as well as the VLBA. You may wish to use VLBAPIPE without RFLAG (or to discard the flags produced by RFLAG) and then run SPFLG to flag the visibility data interactively. Then use VLBAPIPE over again applying the manually produced flags. Procedure R_RESTART (P_RESTART in VLBARUN) will clear away all plot and calibration tables from your data set, leaving any flag tables. Do not delete version one of the CL and FG tables since they contain the initial calibration and on-line flagging information.

VLBAPIPE and VLBARUN will then run and produce the requested number of diagnostic plots. For details on the plots produced for each level of DOPLOT see EXPLAIN VLBAPIPE. If an e-mail address is specified then a VLBAPIPE DONE or VLBAPIPE FAILED message will be sent. However, the VLBAPIPE FAILED message will only be sent if VLBAPIPE failed because of problems with the inputs, if the pipeline failed because a task it was running failed this message is not sent. It is highly recommended that the user read the explain file for VLBAPIPE.

A similar RUN file named VLBAMPHC is available to calibrate, flag, and image data sets for secondary phase stopping centers when DiFX has been used to correlate multiple centers from a single pointing. A pipeline called VLBARFI appeared in 31DEC21. It calibrates a data set concentrating on the autocorrelation data on which it performs a statistical analysis.

C.5 VLBA Pipeline