Pipeline for the VLBA

VLBAPIPE and VLBARUN are two similar pipeline procedures which use the VLBA calibration procedures (from VLBAUTIL) and some logic to calibrate and image VLBA data. They both attempt to make intelligent decisions on defaults, so they can be run fairly automatically, if the names of the sources are known. VLBAPIPE and VLBARUN have been given options to flag data on elevation, to flag excessive fluxes with CLIP, and to apply auto-flagging with RFLAG. The option to clip out bad system temperatures, replacing them with good ones, with TYSMO was also added. The two pipelines differ primarily in the order in which they carry out their operations. Most importantly, VLBAPIPE runs FRING on all calibration scans before computing the bandpass. If the primary calibrator was onserved with multiple scans, this order should help to make the scans phase coherent. If desired, VLBAPIPE and VLBARUN will produce diagnostic plot files and write them to disk creating an HTML file to ease examination of these files. Images will be produced, optionally including self-calibration of the calibration sources. Since only one self-cal, phase only iteration is run, the images should be considered diagnostic in nature.

The two piplines use essentially the same input adverbs, except for the meaning of the DOFLAG parameter. Sample inputs for procedure VLBAPIPE are:

> RUN VLBAUTIL C R

to acquire the procedures used by VLBAPIPE.

> RUN VLBAPIPE C R

to acquire VLBAPIPE.

> DATAIN /dirname/data.fits C R

to set data file to load from disk.

> OUTDISK n C R

to set disk for output.

> OPTYPE ’CONT’ C R

to say this is a continuum data set.

> CLINT 0 C R

to use default.

> CHREFANT ’FD’ C R

to set reference antenna to Fort Davis.

> TIMERANG 0 C R

to have VLBARUN determine a good instrumental delay calibration scan.

> INVER 0 C R

PC table to use, -1 if you want to force manual phase cal.

> CALSOUR ’CAL-BAND’, ’CAL-PHASE’, ’CAL-AMP’ C R

to list calibrators, bandpass calibrator must be first.

> SOURCES ’CAL-PHASE’, ’TARGET’, … C R

to list phase referencing and target pairs.

> CHINC -1 C R

to determine channel increment based on the data channel width.

> INFILE ” C R

set if you want to apply a zenith delay file produced by DELZN.

> SOLINT 0 C R

to use default.

> IMSIZE 512 C R

to make images and specify size of target images.

> FACTOR 0 C R

to make calibrator images 128x128.

> DOCLIP fp,fc C R

to run CLIP to flag a sample when the parallel-hand flux is above |fp| or the cross-hand flux is above |fc|. If fp is positive, the clip is done after amplitude calibration; if negative, it is done before the bandpass and amplitude calibration. The values for fp and fc are different in these two cases.

> DOFLAG 1 C R

to run RFLAG on each source individually.

> TYSMOLEV 100 C R

to clip system temperatures above 100 K with median-window filtering at 10 K and replacing the clipped values with nearby good ones.

> ELEVLIM 15 C R

to flag out all data for which the elevation is < 15 degrees.

> GET2N ctn2 ; NMAPS=1 C R

to use a model for the bandpass calibration.

> SELFCAL 1 C R

to do one pass of self-cal, phase only on each calibrator prior to the final SPLIT¿

> BPASS5 -2 C R

to normalize bandpass scans before the fit. Default is +1 to normalize the bandpass functions themselves without normalization of the data.

> BPSOLINT 0 C R

to make a solution for each calibration scan. BPSOLINT=-1 averages all calibration scans.

> DOBAND 3 C R

to interpolate the bandpass over time. DOBAND=1 (the default) averages all bandpass records.

> CALTASK ’CALIBS’ C R

to use FRING for the initial time series calibration but use CALIB for any self-cal. Other choices are ’CALIB’ and ’FRING’ (the default) to use the specified task for all time series calibrations.

> DOPLOT 1 C R

to make some diagnostic plots; level 2 makes an awful lot of plots.

> OUTFILE ’/directory/name C R

to specify directory for output plots. If this is set, plots are written out from and organized in an HTML file for easy viewing. The plots, HTML file, and message file contents are put in a dated subdirectory below the specified directory.

> OUTTEXT ’email@somewhere.edu C R

to specify an e-mail address if the user wants to be notified when the pipeline is done.

> BADDISK 0 C R

to specify which disks to avoid for scratch.

> VLBAPIPE C R

to run the procedure.

VLBAPIPE or VLBARUN will then run and produce the requested number of diagnostic plots. For details on the plots produced for each level of DOPLOT, see EXPLAIN VLBAPIPE. If an e-mail address is specified then a VLBAPIPE DONE or VLBAPIPE FAILED message will be sent. However, the VLBAPIPE FAILED message will only be sent if the pipeline failed because of problems with the inputs, if it failed because a task it was running failed this message is not sent. It is highly recommended that the user read the explain file for VLBAPIPE before use.

The only adverb that differs between the two pipeline scripts is DOFLAG. In VLBAPIPE, the pipeline will shut down after running RFLAG if 0 < DOFLAG < 10 to allow you to rerun the pipeline using the RFLAG flag tables. Presumably, the calibration in the second pass should be better. If DOFLAG≥ 10 in VLBAPIPE or DOFLAG> 0 in VLBARUN the pipeline invokes RFLAG and then continues to the optional self-cal step followed by SPLIT and IMAGR. VLBARUN uses the value of DOFLAG as an initial clip level in RFLAG but only if it is greater than 100. VLBAPIPE does not use the initial clip option in RFLAG, leaving that to the DOCLIP adverb.

Note that editing with CLIP is always a good idea. RFLAG appears to work well at low frequencies but to be overly enthusiastic at C-band and above. Note that the flagging is done after all calibration steps, so you may examine your data with and without the new flags to determine their correctness. RFLAG is run using baseline-dependent clip levels and so should work with arrays containing high-sensitivity telescopes as well as the VLBA. You may wish to use VLBAPIPE without RFLAG (or to discard the flags produced by RFLAG) and then run SPFLG to flag the visibility data interactively. Then use VLBAPIPE over again applying the manually produced flags. Procedure R_RESTART (P_RESTART in VLBARUN) will clear away all plot and calibration tables from your data set, leaving any flag tables. Do not delete version one of the CL and FG tables since they contain the initial calibration and on-line flagging information.

A similar RUN file named VLBAMPHC is available to calibrate, flag, and image data sets for secondary phase stopping centers when DiFX has been used to correlate multiple centers from a single pointing.

In 31DEC21, a similar pipeline was constructed to inspect autocorrelation data for RFI. VLBARFI reduces the data set using many of the steps of VLBARUN. After the amplitude calibration, the pipeline writes out the autocorrelation data with SPLAT and then runs VBRFI to analyze the statistics of the autocorrelations and to write then into a text file and optionally to plot them. Task PLRFI may be used later to plot one or more of these output text files. In 31DEC22, VLBARFI may be run with no calibration and statistics determined with medians using VLBRF rather than VBRFI.

9.2 Pipeline for the VLBA