source: git/Tst/README @ 8e0ad0

spielwiese Release-4-3-2p8
Last change on this file since 8e0ad0 was 4bde6b, checked in by Hans Schoenemann <hannes@…>, 4 years ago
spelling p1
  • Property mode set to 100644
File size: 6.2 KB
RevLine 
[b35b93]1README file for Singular test-suite
2===================================
3
[75f460]4This is the test-suite for Singular. This directory contains
[b35b93]5
[75f460]6regress.cmd     -- perl script which drives tests
[b35b93]7Short           -- directory containing "short" tests
8Long            -- directory containing "long" tests
[d8ad37]9Old             -- directory containing "old" (TST) tests
10Manual          -- directory containing manual examples
11New             -- directory containing test for new features
12Buch            -- examples from the Singular book
13BuchDL          -- examples from another Singular book
14
[b16a305]15recommended set of tests:
16./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst
[b35b93]17
[75f460]18Run
[e2114af]19  perl ./regress.cmd -h
[4bde6b]20for a summary of the command-line options of regress.cmd.
[e2114af]21
22To test(s) correctness of Singular:
23-----------------------------------
[c1d386]241.) Put the Singular binary in this directory
[4bde6b]25    (preferred solution: ln -s <path_to_Singular_binary> Singular)
[75f460]262.) Run
[e2114af]27     perl ./regress.cmd [*.tst] [*.lst]
[75f460]28    with the desired tst (i.e., Singular test scripts) or lst (list
[e2114af]29    of tst files -- see Old/universal.lst for an example) file(s).
[4bde6b]30    (preferred test scenario:
[c1d386]31    perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst
32    )
[e2114af]333.) If no differences between the new results and the
34    previously obtained results are found, regress.cmd exits with
35    status 0.
364.) If differences between the new results and the
37    previously obtained results are found, regress.cmd exits with
[75f460]38    status != 0, and, for each file xx.tst which lead to a
[4bde6b]39    difference, the corresponding files xx.new.res, xx.res, and xx.diff
[e2114af]40    are kept, and differences are written to stdout.
415.) The Singular executable which is used for testing can
42    alternatively be specified using the -s <Singular> option.
43
44
45To test memory/timing performance of Singular:
46----------------------------------------------
[b35b93]471.) Put the Singular binary in this directory
[75f460]482.) Run
[e2114af]49     perl ./regress.cmd -e [*.tst] [*.lst]
[75f460]50    with the desired tst (i.e., Singular test scripts) or lst (list
[e2114af]51    of tst files -- see Old/universal.lst for an example) file(s).
523.) If no result differences occur, and if timing/memory performance
53    differences are less than $error_val (currently set to 5)
[75f460]54    per-cent, then regress.cmd exits with status 0.
[e2114af]554.) Otherwise, exit status is != 0, and the corresponding files
56    xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the
57    differences of timing/memory performance) are kept, and
58    differences are written to stdout.
595.) Using the -r instead of the -e option only reports about
60    memory/timings differences, i.e., it does not cause an exit with
61    status != 0 if timing/memory performance is above specified
62    per-cent.
[4bde6b]636.) The timing/memory performance differences which trigger an error
[75f460]64    (or, report) can be also be set: Either use
65      ./regress.cmd -[e,r] all%<val>
[e2114af]66    which sets tolerance of all criteria to <val> per-cent, or, use
[75f460]67      ./regress.cmd -[e,r] <crit>%<val>
[e2114af]68    which sets tolerance of all <crit> to <val> per-cent.
69
70To generate new result file(s):
71-------------------------------
[75f460]72Run
[e2114af]73    perl ./regress.cmd -g [*.tst] [*.lst]
[75f460]74with the desired tst (i.e., Singular test scripts) or lst (list
75of tst files).
[e2114af]76NOTE: Running regress.cmd with the -g option re-generates the result
[4bde6b]77and stat files, i.e., all previous results and statistics are
[e2114af]78overwritten and lost!
79
80To include timing/memory statistics for a machine:
81---------------------------------------------------
[75f460]82Run
[e2114af]83    perl ./regress.cmd -m [*.tst] [*.lst]
[75f460]84with the desired tst (i.e., Singular test scripts) or lst (list
85of tst files).
[e2114af]86NOTE:  Running regress.cmd with the -m option merges the
87timings/memory results of the current machine into the respective
88xx.stat file and overwrites previous timings/memory results of the
89current machine.
[779b9b7]90
91
[b35b93]92To add a new test for a library/commando to the test-suite:
93-----------------------------------------------------------
94* The following files need to be provided:
[75f460]95        short/xx_s.tst : Singular code for short and basic tests
[b35b93]96        long/xx_l.tst : Singular code for long and extended tests
97  or, alternatively:
98        short/xx.tst: Singular code for short tests, only
99        short/xx.res : Result output of xx.tst
100
101* xx_s.tst should test the essential functionality of the
102  library/commando in a relatively short time (say, in no more than
103  30s).
104
105* xx_l.tst should test the functionality of the
106  library/commando in detail so that, if possible, all relevant
107  cases/results are tested. Nevertheless, such a test should not run
108  longer than, say, 10 minutes.
109
[779b9b7]110* If useful tests do generally execute in a short time, have xx.tst,
[b35b93]111  only.
112
113* Rules for providing tst files:
114  1.) tst files always start with the following three commandos as
[75f460]115  preamble:
[b35b93]116         LIB "tst.lib";
117         tst_init();
118  tst_init() writes some general info to stdout (like date,
[e2114af]119  uname, hostname, version, etc.). The library tst.lib (contained in
120  the Singular distribution) provides, among others, the routines
[75f460]121  tst_init() and tst_ignore().
[e2114af]122
123  2.) tst files should end with the following statements:
124         tst_status(1); $
125  which enables (automatic) checks of the timing/memory performance of
[75f460]126  Singular.
[e2114af]127
128  3.) All system-dependent output (like run-times, memory usages,
[75f460]129  pathnames, dates, etc.) should generally be avoided.
[e2114af]130
131  4.) After time/memory critical sections of the tst files, the
132  command
133        tst_status();
134  should be inserted. This enables (automatic) checks of the
135  timing/memory performance of Singular since the last call to
136  tst_status (resp. since the start-up of Singular).
137
[75f460]138  5.) If system-dependent output can not be avodied, the routine
[e2114af]139  tst_ignore(...) should be used:
[b35b93]140  tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
141                               a string conversion exists;
142                               if present, keyword must be one of the
143                               following strings: "time", "memory"
144  tst_ignore() outputs 'val' by prepending the following prefix:
[75f460]145  no keyword --       // tst_ignore:
[e2114af]146  "time" keyword --   // tst_ignore: time:
147  "memory" keyword -- // tst_ignore: memory:
[75f460]148  which causes automatic tests to ignore these lines when doing a diff
149  on result files.
150
151* The command for generating xx.res.gz.uu and xx.stat is:
[e2114af]152  cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
[b35b93]153
154
Note: See TracBrowser for help on using the repository browser.