AIPS Benchmarks
(Dirty Dozen Tasks)

---

Most recent change on $Date: 2024/04/10 21:18:59 $ (UT) [Y2K test]

NOTE: a benchmark test, called Y2K, has been developed. It is an adjustment of the existing DDT with an altered set of tasks (UVMAP and MX replaced by IMAGR), and a 10-fold change in the size of the problems. Its goal was to take on the order of an hour to run on fairly new desktop hardware. However, it now takes all of 29 seconds to run on the new record holder.

The Y2K and DDT (Dirty Dozen) suites are reliable ways of determining two things:

• Does a given AIPS installation give accurate results, and
• How fast does it do it?

The AIPS Memo 104 describes the new test and gives initial performance results. It was developed on the 31DEC00 version of AIPS. See the 31DEC24 and 31DEC23 pages for information on obtaining and maintaining your very own copy of the current releases. For more details on the earlier DDT, see AIPS Memo 85 which has the information on the last iteration of the DDT test. This was modified for the 15JAN94 release of AIPS (this is not the most recent version; see the AIPS Home Page for info on recent releases). There are other documents available too. Note that the DDT test is no longer available; it uses the task MX which is itself no longer available.

The AIPSMark⁰⁰ is defined as:

A_m⁽⁰⁰⁾ = (40000 ÷ T_Y2K(L))

where T_Y2K(L) is the total run time in seconds of the Y2K LARGE test. This run time was defined initially by Glendenning and Hunt as the time in seconds from RUN DDTEXEC through PRINTING ANSWERS, ERRORS, OTHER IMPORTANT MESSAGES). The multiplier 40000 was chosen to make the new AIPSMark⁰⁰ about the same as the old AIPSMark⁹³ on many machines.

The AIPSMark⁹³ is defined as:

A_m⁽⁹³⁾ = (4000 ÷ T_DDT(L))

where T_DDT(L) is the total run time in seconds for the LARGE DDT suite as defined by Glendenning and Hunt. Chris Flatters devised a formula for predicting a computer's AIPSMarks based on its measured SPECfp(95) and SPECint(95) performance. An extensive list of SPEC results is maintained on the Web by John DiMarco at the University of Toronto. Chris finds that the following relation gives a fair, ball-park estimate of the AIPSmark that can be expected on a machine with known SPEC(95) benchmarks.

AIPSMark⁹³ = 0.898 × SPECfp(95) + 0.11 × SPECint(95) - 1.665

How do I run Y2K?

Install the latest version of AIPS; see for example 31DEC22. Start running AIPS and enter EXPLAIN Y2K. This is available on the web for the impatient as Y2K.HLP. Note that DDT is no longer available since it uses tasks that have been removed from AIPS. The Y2K process consists of several steps:

• Install AIPS
• Copy the relevant data files (see below) to your computer.
• Compile Y2K (RUN Y2KLOAD).
• Set adverbs in Y2K to load the data (TPUT Y2K) and load them (RUN Y2KEXEC).
• Set adverbs in Y2K to run the test (TPUT Y2K) and then actually run the test (RUN Y2KEXEC).
• Examine the printfile for timing and agreement information

Where can I get the Y2K data files?

Don't panic; 31DEC22 versions of the Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp. For manual use of ftp, the address for anonymous ftp is ftp.aoc.nrao.edu and the data files are located in directory pub/software/aips/FITS/31DEC22. The older Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp. But please look at the file sizes (use the ls -l or dir command in your ftp program) before pulling these across. Some of them are large: the aggregate size of the 19 Y2K (large) files is 319 Megabytes. The RUN and HELP files needed to compile and execute the tests are provided with the 31DEC00 and later versions of AIPS. Note that DDT does not run on modern version of AIPS since MX and other tasks involved have been removed from the system.

DISCLAIMER

The results published in these pages do NOT constitute any sort of an endorsement by NRAO or Associated Universities, Inc. (NRAO's parent organization) of any particular vendor. The AIPS web pages at NRAO/Socorro do not and will not contain links to any hardware vendors' web pages.

In addition, many of the results reported here were relayed to us by others, and while all of these were carried out in a uniform way to the best of our knowledge, there will inevitably be variations introduced by factors such as changes in the AIPS code over time, system loads, different disk controllers (e.g. IDE vs. SCSI), and so on. The intent here is to share AIPS users' experiences with various platforms among the Astronomical Community.

New Y2K HUGE test

As reported above, the Y2K LARGE test can now run in as little as 29 seconds on some computers. Therefore, we have put together a HUGE version of the Y2K test. It uses Carilli/Perley Cynus-A X-Band data from D, C, B, and 4 frequencies of A configurations of the VLA. The 2674490 visbilities are Cleaned in four resolutions by IMAGR before the self-cal (replacing APCLN since it does not do multiple resolutions). The self-cal is amplitude and phase with the square root of the normal weights. The following Clean is also four resolutions done with IMAGR. The VTESS (and the one in LARGE) use the trick of setting FLUX = -F/10 where F is about the correct flux. With this trick VTESS is quite convergent rather than being completely erratic. The 42 files of Y2K HUGE add up to 1.2 Gbytes.

The use of versions of Clean in these tests is somewhat problematical. When there are numerous image pixels at the same value as seen by a human, there will likely be one a single bit higher than the others as seem by the computer, Which pixel that is will depend on slight differences in the round-off and other algorithms of the host arithmetic unit. And, once Clean has chosen a different pixel, the algorithm will head off in a different direction. This explains why, with data of high signal-to-noise, the peak error in Cleans is usually no better than 9 bits. The Huge test saves the 4 residual images from the initial Clean and they usually have an rms difference around 6 bits and a peak difference around 0.3 - 3 bits. Nonetheless, the Huge test is useful. On the top-speed Mac mini M1 machine, Huge takee 390 seconds real time. On my latest Dell desktop, Huge takes twice as long (803 seconds).

Initial test results with Y2K

The test results from AIPS Memo 104 are summarized in a separate table.

Ancient Mac OS/X results

    Machine   Machine         Compiler    DDTL      Y2KL     Y2KL cpu  AMark
    run on    compiled on     type        sec       sec      min        2000
    -------   -----------     --------   -------   --------  --------  -----

1      4         4  .           A GNU    121-124     1126     16.28      35
2.     4         4              GNU      115-120     1060     15.14      38
3.     4         4 (b)          IBM       81- 83      748      9.58      53
5.     4         5 (gen)        IBM       77- 79      748      9.55      53
7.     4         4 (gen)        IBM       78- 83      759      9.57      53

11.    5         5              A GNU     69- 73      582      9.29      69
12.    5         5              GNU       68- 70      558      7.77      72
13.    5         5 (a)          IBM       33- 36      310      4.60     129
13.    5         5 (b)          IBM       32- 38      305      4.42     131
15.    5         5 (gen)        IBM       33- 34      300      4.46     133
16.    5         5 (gen -O5)    IBM       34- 39   313-322  4.66-4.82  >124

The G5 used is hakone a 2.0 GHz, dual-headed machine with 8 Gbyte ram,
       3 Terabytes disk!
The G4 used is pippin a 1.33 GHz, single-headed, rack-mounted machine
       with 1 Gbyte of ram.

Putting aips on a "disk image" slows things down (e.g. 47 sec DDT on
the G5 with the IBM compiler, 90 sec on G4)

IBM options (a) are -qthreaded -qunroll=auto -qtune=auto -qarch=auto
            (b) are -qtune=auto -qarch=auto
    all including (gen) use -O3.

The A GNU is the Apple-provided version of the GNU compilers while GNU
is one installed by Wes (~ 3.4.1).

Really old boxes per DDT

This table is an attempt to show the best results for the most advanced hardware on which we or others have been able to run the DDT and AIPS back in the day. It is obsolete.

Recent DDT Benchmark Results

A full table of results is available along with the initial 1993 results.

Pointers to sections ordered by manufacturer.

DEC ALPHA AXP (Digital Unix, OSF/1; and Linux)

Hewlett Packard (HP-UX 10)

IBM Power PC (AIX)

Intel/Linux and others, including Pentium III, Athlon, P-Pro, P-II, P-I, i486, i386

Silicon Graphics

Sun SPARC Ultra including the /tmp trick, the Weitek Upgrade