source: git/Tst/README @ e35f0b

spielwiese
Last change on this file since e35f0b was e2114af, checked in by Olaf Bachmann <obachman@…>, 26 years ago
* incoporatred status checks git-svn-id: file:///usr/local/Singular/svn/trunk@2317 2c84dea3-7e68-4137-9b89-c4e89433aadc
  • Property mode set to 100644
File size: 5.8 KB
Line 
1README file for Singular test-suite
2===================================
3
4This is the test-suite for Singular. This directory contains
5
6regress.cmd     -- perl script which drives tests
7Short           -- directory containing "short" tests
8Long            -- directory containing "long" tests
9Old             -- directory containing "old" (TST) tests
10
11Run
12  perl ./regress.cmd -h
13for a summary of the comaand-line options of regress.cmd.
14
15To test(s) correctness of Singular:
16-----------------------------------
171.) Put the Singular binary in this directory.
182.) Run
19     perl ./regress.cmd [*.tst] [*.lst]
20    with the desired tst (i.e., Singular test scripts) or lst (list
21    of tst files -- see Old/universal.lst for an example) file(s).
223.) If no differences between the new results and the
23    previously obtained results are found, regress.cmd exits with
24    status 0.
254.) If differences between the new results and the
26    previously obtained results are found, regress.cmd exits with
27    status != 0, and, for each file xx.tst which lead to a
28    difference, the coresponding files xx.new.res, xx.res, and xx.diff
29    are kept, and differences are written to stdout.
305.) The Singular executable which is used for testing can
31    alternatively be specified using the -s <Singular> option.
32
33
34To test memory/timing performance of Singular:
35----------------------------------------------
361.) Put the Singular binary in this directory
372.) Run
38     perl ./regress.cmd -e [*.tst] [*.lst]
39    with the desired tst (i.e., Singular test scripts) or lst (list
40    of tst files -- see Old/universal.lst for an example) file(s).
413.) If no result differences occur, and if timing/memory performance
42    differences are less than $error_val (currently set to 5)
43    per-cent, then regress.cmd exits with status 0.
444.) Otherwise, exit status is != 0, and the corresponding files
45    xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the
46    differences of timing/memory performance) are kept, and
47    differences are written to stdout.
485.) Using the -r instead of the -e option only reports about
49    memory/timings differences, i.e., it does not cause an exit with
50    status != 0 if timing/memory performance is above specified
51    per-cent.
526.) The timing/memory performance differenes which trigger an error
53    (or, report) can be also be set: Either use
54      ./regress.cmd -[e,r] all%<val>
55    which sets tolerance of all criteria to <val> per-cent, or, use
56      ./regress.cmd -[e,r] <crit>%<val>
57    which sets tolerance of all <crit> to <val> per-cent.
58
59To generate new result file(s):
60-------------------------------
61Run
62    perl ./regress.cmd -g [*.tst] [*.lst]
63with the desired tst (i.e., Singular test scripts) or lst (list
64of tst files).
65NOTE: Running regress.cmd with the -g option re-generates the result
66and stat files, i.e., all previous results and statisitics are
67overwritten and lost!
68
69To include timing/memory statistics for a machine:
70---------------------------------------------------
71Run
72    perl ./regress.cmd -m [*.tst] [*.lst]
73with the desired tst (i.e., Singular test scripts) or lst (list
74of tst files).
75NOTE:  Running regress.cmd with the -m option merges the
76timings/memory results of the current machine into the respective
77xx.stat file and overwrites previous timings/memory results of the
78current machine.
79
80
81To add a new test for a library/commando to the test-suite:
82-----------------------------------------------------------
83* The following files need to be provided:
84        short/xx_s.tst : Singular code for short and basic tests
85        long/xx_l.tst : Singular code for long and extended tests
86  or, alternatively:
87        short/xx.tst: Singular code for short tests, only
88        short/xx.res : Result output of xx.tst
89
90* xx_s.tst should test the essential functionality of the
91  library/commando in a relatively short time (say, in no more than
92  30s).
93
94* xx_l.tst should test the functionality of the
95  library/commando in detail so that, if possible, all relevant
96  cases/results are tested. Nevertheless, such a test should not run
97  longer than, say, 10 minutes.
98
99* If useful tests do generally execute in a short time, have xx.tst,
100  only.
101
102* Rules for providing tst files:
103  1.) tst files always start with the following three commandos as
104  preamble:
105         LIB "tst.lib";
106         tst_init();
107         tst_ignore("CVS ID $Id:$"); // or, write own version number here
108  tst_init() writes some general info to stdout (like date,
109  uname, hostname, version, etc.). The library tst.lib (contained in
110  the Singular distribution) provides, among others, the routines
111  tst_init() and tst_ignore().
112
113  2.) tst files should end with the following statements:
114         tst_status(1); $
115  which enables (automatic) checks of the timing/memory performance of
116  Singular.
117
118  3.) All system-dependent output (like run-times, memory usages,
119  pathnames, dates, etc.) should generally be avoided.
120
121  4.) After time/memory critical sections of the tst files, the
122  command
123        tst_status();
124  should be inserted. This enables (automatic) checks of the
125  timing/memory performance of Singular since the last call to
126  tst_status (resp. since the start-up of Singular).
127
128  5.) If system-dependent output can not be avodied, the routine
129  tst_ignore(...) should be used:
130  tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
131                               a string conversion exists;
132                               if present, keyword must be one of the
133                               following strings: "time", "memory"
134  tst_ignore() outputs 'val' by prepending the following prefix:
135  no keyword --       // tst_ignore:
136  "time" keyword --   // tst_ignore: time:
137  "memory" keyword -- // tst_ignore: memory:
138  which causes automatic tests to ignore these lines when doing a diff
139  on result files.
140 
141* The command for generating xx.res.gz.uu and xx.stat is:
142  cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
143
144
Note: See TracBrowser for help on using the repository browser.