source: git/Tst/README @ a1b40a

Last change on this file since a1b40a was 4bde6b, checked in by Hans Schoenemann <hannes@…>, 4 years ago
spelling p1
  • Property mode set to 100644
File size: 6.2 KB
1README file for Singular test-suite
4This is the test-suite for Singular. This directory contains
6regress.cmd     -- perl script which drives tests
7Short           -- directory containing "short" tests
8Long            -- directory containing "long" tests
9Old             -- directory containing "old" (TST) tests
10Manual          -- directory containing manual examples
11New             -- directory containing test for new features
12Buch            -- examples from the Singular book
13BuchDL          -- examples from another Singular book
15recommended set of tests:
16./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst
19  perl ./regress.cmd -h
20for a summary of the command-line options of regress.cmd.
22To test(s) correctness of Singular:
241.) Put the Singular binary in this directory
25    (preferred solution: ln -s <path_to_Singular_binary> Singular)
262.) Run
27     perl ./regress.cmd [*.tst] [*.lst]
28    with the desired tst (i.e., Singular test scripts) or lst (list
29    of tst files -- see Old/universal.lst for an example) file(s).
30    (preferred test scenario:
31    perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst
32    )
333.) If no differences between the new results and the
34    previously obtained results are found, regress.cmd exits with
35    status 0.
364.) If differences between the new results and the
37    previously obtained results are found, regress.cmd exits with
38    status != 0, and, for each file xx.tst which lead to a
39    difference, the corresponding files, xx.res, and xx.diff
40    are kept, and differences are written to stdout.
415.) The Singular executable which is used for testing can
42    alternatively be specified using the -s <Singular> option.
45To test memory/timing performance of Singular:
471.) Put the Singular binary in this directory
482.) Run
49     perl ./regress.cmd -e [*.tst] [*.lst]
50    with the desired tst (i.e., Singular test scripts) or lst (list
51    of tst files -- see Old/universal.lst for an example) file(s).
523.) If no result differences occur, and if timing/memory performance
53    differences are less than $error_val (currently set to 5)
54    per-cent, then regress.cmd exits with status 0.
554.) Otherwise, exit status is != 0, and the corresponding files
56, xx.res, xx.diff, and xx.stat.diff (which logs the
57    differences of timing/memory performance) are kept, and
58    differences are written to stdout.
595.) Using the -r instead of the -e option only reports about
60    memory/timings differences, i.e., it does not cause an exit with
61    status != 0 if timing/memory performance is above specified
62    per-cent.
636.) The timing/memory performance differences which trigger an error
64    (or, report) can be also be set: Either use
65      ./regress.cmd -[e,r] all%<val>
66    which sets tolerance of all criteria to <val> per-cent, or, use
67      ./regress.cmd -[e,r] <crit>%<val>
68    which sets tolerance of all <crit> to <val> per-cent.
70To generate new result file(s):
73    perl ./regress.cmd -g [*.tst] [*.lst]
74with the desired tst (i.e., Singular test scripts) or lst (list
75of tst files).
76NOTE: Running regress.cmd with the -g option re-generates the result
77and stat files, i.e., all previous results and statistics are
78overwritten and lost!
80To include timing/memory statistics for a machine:
83    perl ./regress.cmd -m [*.tst] [*.lst]
84with the desired tst (i.e., Singular test scripts) or lst (list
85of tst files).
86NOTE:  Running regress.cmd with the -m option merges the
87timings/memory results of the current machine into the respective
88xx.stat file and overwrites previous timings/memory results of the
89current machine.
92To add a new test for a library/commando to the test-suite:
94* The following files need to be provided:
95        short/xx_s.tst : Singular code for short and basic tests
96        long/xx_l.tst : Singular code for long and extended tests
97  or, alternatively:
98        short/xx.tst: Singular code for short tests, only
99        short/xx.res : Result output of xx.tst
101* xx_s.tst should test the essential functionality of the
102  library/commando in a relatively short time (say, in no more than
103  30s).
105* xx_l.tst should test the functionality of the
106  library/commando in detail so that, if possible, all relevant
107  cases/results are tested. Nevertheless, such a test should not run
108  longer than, say, 10 minutes.
110* If useful tests do generally execute in a short time, have xx.tst,
111  only.
113* Rules for providing tst files:
114  1.) tst files always start with the following three commandos as
115  preamble:
116         LIB "tst.lib";
117         tst_init();
118  tst_init() writes some general info to stdout (like date,
119  uname, hostname, version, etc.). The library tst.lib (contained in
120  the Singular distribution) provides, among others, the routines
121  tst_init() and tst_ignore().
123  2.) tst files should end with the following statements:
124         tst_status(1); $
125  which enables (automatic) checks of the timing/memory performance of
126  Singular.
128  3.) All system-dependent output (like run-times, memory usages,
129  pathnames, dates, etc.) should generally be avoided.
131  4.) After time/memory critical sections of the tst files, the
132  command
133        tst_status();
134  should be inserted. This enables (automatic) checks of the
135  timing/memory performance of Singular since the last call to
136  tst_status (resp. since the start-up of Singular).
138  5.) If system-dependent output can not be avodied, the routine
139  tst_ignore(...) should be used:
140  tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
141                               a string conversion exists;
142                               if present, keyword must be one of the
143                               following strings: "time", "memory"
144  tst_ignore() outputs 'val' by prepending the following prefix:
145  no keyword --       // tst_ignore:
146  "time" keyword --   // tst_ignore: time:
147  "memory" keyword -- // tst_ignore: memory:
148  which causes automatic tests to ignore these lines when doing a diff
149  on result files.
151* The command for generating xx.res.gz.uu and xx.stat is:
152  cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
Note: See TracBrowser for help on using the repository browser.