source: git/Tst/README @ d8ad37

spielwiese
Last change on this file since d8ad37 was d8ad37, checked in by Hans Schoenemann <hannes@…>, 6 years ago
update Tst/README
  • Property mode set to 100644
File size: 6.2 KB
Line 
1README file for Singular test-suite
2===================================
3
4This is the test-suite for Singular. This directory contains
5
6regress.cmd     -- perl script which drives tests
7Short           -- directory containing "short" tests
8Long            -- directory containing "long" tests
9Old             -- directory containing "old" (TST) tests
10Manual          -- directory containing manual examples
11New             -- directory containing test for new features
12Buch            -- examples from the Singular book
13BuchDL          -- examples from another Singular book
14
15recommended set of tests:
16./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst
17
18Run
19  perl ./regress.cmd -h
20for a summary of the comand-line options of regress.cmd.
21
22To test(s) correctness of Singular:
23-----------------------------------
241.) Put the Singular binary in this directory
25    (prefered solution: ln -s <path_to_Singular_binary> Singular)
262.) Run
27     perl ./regress.cmd [*.tst] [*.lst]
28    with the desired tst (i.e., Singular test scripts) or lst (list
29    of tst files -- see Old/universal.lst for an example) file(s).
30    (prefered test scenario:
31    perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst
32    )
333.) If no differences between the new results and the
34    previously obtained results are found, regress.cmd exits with
35    status 0.
364.) If differences between the new results and the
37    previously obtained results are found, regress.cmd exits with
38    status != 0, and, for each file xx.tst which lead to a
39    difference, the coresponding files xx.new.res, xx.res, and xx.diff
40    are kept, and differences are written to stdout.
415.) The Singular executable which is used for testing can
42    alternatively be specified using the -s <Singular> option.
43
44
45To test memory/timing performance of Singular:
46----------------------------------------------
471.) Put the Singular binary in this directory
482.) Run
49     perl ./regress.cmd -e [*.tst] [*.lst]
50    with the desired tst (i.e., Singular test scripts) or lst (list
51    of tst files -- see Old/universal.lst for an example) file(s).
523.) If no result differences occur, and if timing/memory performance
53    differences are less than $error_val (currently set to 5)
54    per-cent, then regress.cmd exits with status 0.
554.) Otherwise, exit status is != 0, and the corresponding files
56    xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the
57    differences of timing/memory performance) are kept, and
58    differences are written to stdout.
595.) Using the -r instead of the -e option only reports about
60    memory/timings differences, i.e., it does not cause an exit with
61    status != 0 if timing/memory performance is above specified
62    per-cent.
636.) The timing/memory performance differenes which trigger an error
64    (or, report) can be also be set: Either use
65      ./regress.cmd -[e,r] all%<val>
66    which sets tolerance of all criteria to <val> per-cent, or, use
67      ./regress.cmd -[e,r] <crit>%<val>
68    which sets tolerance of all <crit> to <val> per-cent.
69
70To generate new result file(s):
71-------------------------------
72Run
73    perl ./regress.cmd -g [*.tst] [*.lst]
74with the desired tst (i.e., Singular test scripts) or lst (list
75of tst files).
76NOTE: Running regress.cmd with the -g option re-generates the result
77and stat files, i.e., all previous results and statisitics are
78overwritten and lost!
79
80To include timing/memory statistics for a machine:
81---------------------------------------------------
82Run
83    perl ./regress.cmd -m [*.tst] [*.lst]
84with the desired tst (i.e., Singular test scripts) or lst (list
85of tst files).
86NOTE:  Running regress.cmd with the -m option merges the
87timings/memory results of the current machine into the respective
88xx.stat file and overwrites previous timings/memory results of the
89current machine.
90
91
92To add a new test for a library/commando to the test-suite:
93-----------------------------------------------------------
94* The following files need to be provided:
95        short/xx_s.tst : Singular code for short and basic tests
96        long/xx_l.tst : Singular code for long and extended tests
97  or, alternatively:
98        short/xx.tst: Singular code for short tests, only
99        short/xx.res : Result output of xx.tst
100
101* xx_s.tst should test the essential functionality of the
102  library/commando in a relatively short time (say, in no more than
103  30s).
104
105* xx_l.tst should test the functionality of the
106  library/commando in detail so that, if possible, all relevant
107  cases/results are tested. Nevertheless, such a test should not run
108  longer than, say, 10 minutes.
109
110* If useful tests do generally execute in a short time, have xx.tst,
111  only.
112
113* Rules for providing tst files:
114  1.) tst files always start with the following three commandos as
115  preamble:
116         LIB "tst.lib";
117         tst_init();
118  tst_init() writes some general info to stdout (like date,
119  uname, hostname, version, etc.). The library tst.lib (contained in
120  the Singular distribution) provides, among others, the routines
121  tst_init() and tst_ignore().
122
123  2.) tst files should end with the following statements:
124         tst_status(1); $
125  which enables (automatic) checks of the timing/memory performance of
126  Singular.
127
128  3.) All system-dependent output (like run-times, memory usages,
129  pathnames, dates, etc.) should generally be avoided.
130
131  4.) After time/memory critical sections of the tst files, the
132  command
133        tst_status();
134  should be inserted. This enables (automatic) checks of the
135  timing/memory performance of Singular since the last call to
136  tst_status (resp. since the start-up of Singular).
137
138  5.) If system-dependent output can not be avodied, the routine
139  tst_ignore(...) should be used:
140  tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
141                               a string conversion exists;
142                               if present, keyword must be one of the
143                               following strings: "time", "memory"
144  tst_ignore() outputs 'val' by prepending the following prefix:
145  no keyword --       // tst_ignore:
146  "time" keyword --   // tst_ignore: time:
147  "memory" keyword -- // tst_ignore: memory:
148  which causes automatic tests to ignore these lines when doing a diff
149  on result files.
150
151* The command for generating xx.res.gz.uu and xx.stat is:
152  cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
153
154
Note: See TracBrowser for help on using the repository browser.