source: git/Tst/README @ 64daec

Last change on this file since 64daec was b16a305, checked in by Hans Schoenemann <hannes@…>, 14 years ago
recommend tests git-svn-id: file:///usr/local/Singular/svn/trunk@13088 2c84dea3-7e68-4137-9b89-c4e89433aadc
  • Property mode set to 100644
File size: 6.2 KB
1README file for Singular test-suite
4This is the test-suite for Singular. This directory contains
6regress.cmd     -- perl script which drives tests
7Short           -- directory containing "short" tests
8Long            -- directory containing "long" tests
9Old             -- directory containing "old" (TST) tests
10recommended set of tests:
11./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst
14  perl ./regress.cmd -h
15for a summary of the comaand-line options of regress.cmd.
17To test(s) correctness of Singular:
191.) Put the Singular binary in this directory
20    (prefered solution: ln -s <path_to_Singular_binary> Singular)
212.) Run
22     perl ./regress.cmd [*.tst] [*.lst]
23    with the desired tst (i.e., Singular test scripts) or lst (list
24    of tst files -- see Old/universal.lst for an example) file(s).
25    (prefered test scenario:
26    perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst
27    )
283.) If no differences between the new results and the
29    previously obtained results are found, regress.cmd exits with
30    status 0.
314.) If differences between the new results and the
32    previously obtained results are found, regress.cmd exits with
33    status != 0, and, for each file xx.tst which lead to a
34    difference, the coresponding files, xx.res, and xx.diff
35    are kept, and differences are written to stdout.
365.) The Singular executable which is used for testing can
37    alternatively be specified using the -s <Singular> option.
40To test memory/timing performance of Singular:
421.) Put the Singular binary in this directory
432.) Run
44     perl ./regress.cmd -e [*.tst] [*.lst]
45    with the desired tst (i.e., Singular test scripts) or lst (list
46    of tst files -- see Old/universal.lst for an example) file(s).
473.) If no result differences occur, and if timing/memory performance
48    differences are less than $error_val (currently set to 5)
49    per-cent, then regress.cmd exits with status 0.
504.) Otherwise, exit status is != 0, and the corresponding files
51, xx.res, xx.diff, and xx.stat.diff (which logs the
52    differences of timing/memory performance) are kept, and
53    differences are written to stdout.
545.) Using the -r instead of the -e option only reports about
55    memory/timings differences, i.e., it does not cause an exit with
56    status != 0 if timing/memory performance is above specified
57    per-cent.
586.) The timing/memory performance differenes which trigger an error
59    (or, report) can be also be set: Either use
60      ./regress.cmd -[e,r] all%<val>
61    which sets tolerance of all criteria to <val> per-cent, or, use
62      ./regress.cmd -[e,r] <crit>%<val>
63    which sets tolerance of all <crit> to <val> per-cent.
65To generate new result file(s):
68    perl ./regress.cmd -g [*.tst] [*.lst]
69with the desired tst (i.e., Singular test scripts) or lst (list
70of tst files).
71NOTE: Running regress.cmd with the -g option re-generates the result
72and stat files, i.e., all previous results and statisitics are
73overwritten and lost!
75To include timing/memory statistics for a machine:
78    perl ./regress.cmd -m [*.tst] [*.lst]
79with the desired tst (i.e., Singular test scripts) or lst (list
80of tst files).
81NOTE:  Running regress.cmd with the -m option merges the
82timings/memory results of the current machine into the respective
83xx.stat file and overwrites previous timings/memory results of the
84current machine.
87To add a new test for a library/commando to the test-suite:
89* The following files need to be provided:
90        short/xx_s.tst : Singular code for short and basic tests
91        long/xx_l.tst : Singular code for long and extended tests
92  or, alternatively:
93        short/xx.tst: Singular code for short tests, only
94        short/xx.res : Result output of xx.tst
96* xx_s.tst should test the essential functionality of the
97  library/commando in a relatively short time (say, in no more than
98  30s).
100* xx_l.tst should test the functionality of the
101  library/commando in detail so that, if possible, all relevant
102  cases/results are tested. Nevertheless, such a test should not run
103  longer than, say, 10 minutes.
105* If useful tests do generally execute in a short time, have xx.tst,
106  only.
108* Rules for providing tst files:
109  1.) tst files always start with the following three commandos as
110  preamble:
111         LIB "tst.lib";
112         tst_init();
113         tst_ignore("CVS ID $Id$"); // or, write own version number here
114  tst_init() writes some general info to stdout (like date,
115  uname, hostname, version, etc.). The library tst.lib (contained in
116  the Singular distribution) provides, among others, the routines
117  tst_init() and tst_ignore().
119  2.) tst files should end with the following statements:
120         tst_status(1); $
121  which enables (automatic) checks of the timing/memory performance of
122  Singular.
124  3.) All system-dependent output (like run-times, memory usages,
125  pathnames, dates, etc.) should generally be avoided.
127  4.) After time/memory critical sections of the tst files, the
128  command
129        tst_status();
130  should be inserted. This enables (automatic) checks of the
131  timing/memory performance of Singular since the last call to
132  tst_status (resp. since the start-up of Singular).
134  5.) If system-dependent output can not be avodied, the routine
135  tst_ignore(...) should be used:
136  tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
137                               a string conversion exists;
138                               if present, keyword must be one of the
139                               following strings: "time", "memory"
140  tst_ignore() outputs 'val' by prepending the following prefix:
141  no keyword --       // tst_ignore:
142  "time" keyword --   // tst_ignore: time:
143  "memory" keyword -- // tst_ignore: memory:
144  which causes automatic tests to ignore these lines when doing a diff
145  on result files.
147* The command for generating xx.res.gz.uu and xx.stat is:
148  cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
150* You can download the newest version of tst.lib from
Note: See TracBrowser for help on using the repository browser.