1 | README file for Singular test-suite |
---|
2 | =================================== |
---|
3 | |
---|
4 | This is the test-suite for Singular. This directory contains |
---|
5 | |
---|
6 | regress.cmd -- perl script which drives tests |
---|
7 | Short -- directory containing "short" tests |
---|
8 | Long -- directory containing "long" tests |
---|
9 | Old -- directory containing "old" (TST) tests |
---|
10 | recommended set of tests: |
---|
11 | ./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst |
---|
12 | |
---|
13 | Run |
---|
14 | perl ./regress.cmd -h |
---|
15 | for a summary of the comaand-line options of regress.cmd. |
---|
16 | |
---|
17 | To test(s) correctness of Singular: |
---|
18 | ----------------------------------- |
---|
19 | 1.) Put the Singular binary in this directory |
---|
20 | (prefered solution: ln -s <path_to_Singular_binary> Singular) |
---|
21 | 2.) Run |
---|
22 | perl ./regress.cmd [*.tst] [*.lst] |
---|
23 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
24 | of tst files -- see Old/universal.lst for an example) file(s). |
---|
25 | (prefered test scenario: |
---|
26 | perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst |
---|
27 | ) |
---|
28 | 3.) If no differences between the new results and the |
---|
29 | previously obtained results are found, regress.cmd exits with |
---|
30 | status 0. |
---|
31 | 4.) If differences between the new results and the |
---|
32 | previously obtained results are found, regress.cmd exits with |
---|
33 | status != 0, and, for each file xx.tst which lead to a |
---|
34 | difference, the coresponding files xx.new.res, xx.res, and xx.diff |
---|
35 | are kept, and differences are written to stdout. |
---|
36 | 5.) The Singular executable which is used for testing can |
---|
37 | alternatively be specified using the -s <Singular> option. |
---|
38 | |
---|
39 | |
---|
40 | To test memory/timing performance of Singular: |
---|
41 | ---------------------------------------------- |
---|
42 | 1.) Put the Singular binary in this directory |
---|
43 | 2.) Run |
---|
44 | perl ./regress.cmd -e [*.tst] [*.lst] |
---|
45 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
46 | of tst files -- see Old/universal.lst for an example) file(s). |
---|
47 | 3.) If no result differences occur, and if timing/memory performance |
---|
48 | differences are less than $error_val (currently set to 5) |
---|
49 | per-cent, then regress.cmd exits with status 0. |
---|
50 | 4.) Otherwise, exit status is != 0, and the corresponding files |
---|
51 | xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the |
---|
52 | differences of timing/memory performance) are kept, and |
---|
53 | differences are written to stdout. |
---|
54 | 5.) Using the -r instead of the -e option only reports about |
---|
55 | memory/timings differences, i.e., it does not cause an exit with |
---|
56 | status != 0 if timing/memory performance is above specified |
---|
57 | per-cent. |
---|
58 | 6.) The timing/memory performance differenes which trigger an error |
---|
59 | (or, report) can be also be set: Either use |
---|
60 | ./regress.cmd -[e,r] all%<val> |
---|
61 | which sets tolerance of all criteria to <val> per-cent, or, use |
---|
62 | ./regress.cmd -[e,r] <crit>%<val> |
---|
63 | which sets tolerance of all <crit> to <val> per-cent. |
---|
64 | |
---|
65 | To generate new result file(s): |
---|
66 | ------------------------------- |
---|
67 | Run |
---|
68 | perl ./regress.cmd -g [*.tst] [*.lst] |
---|
69 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
70 | of tst files). |
---|
71 | NOTE: Running regress.cmd with the -g option re-generates the result |
---|
72 | and stat files, i.e., all previous results and statisitics are |
---|
73 | overwritten and lost! |
---|
74 | |
---|
75 | To include timing/memory statistics for a machine: |
---|
76 | --------------------------------------------------- |
---|
77 | Run |
---|
78 | perl ./regress.cmd -m [*.tst] [*.lst] |
---|
79 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
80 | of tst files). |
---|
81 | NOTE: Running regress.cmd with the -m option merges the |
---|
82 | timings/memory results of the current machine into the respective |
---|
83 | xx.stat file and overwrites previous timings/memory results of the |
---|
84 | current machine. |
---|
85 | |
---|
86 | |
---|
87 | To add a new test for a library/commando to the test-suite: |
---|
88 | ----------------------------------------------------------- |
---|
89 | * The following files need to be provided: |
---|
90 | short/xx_s.tst : Singular code for short and basic tests |
---|
91 | long/xx_l.tst : Singular code for long and extended tests |
---|
92 | or, alternatively: |
---|
93 | short/xx.tst: Singular code for short tests, only |
---|
94 | short/xx.res : Result output of xx.tst |
---|
95 | |
---|
96 | * xx_s.tst should test the essential functionality of the |
---|
97 | library/commando in a relatively short time (say, in no more than |
---|
98 | 30s). |
---|
99 | |
---|
100 | * xx_l.tst should test the functionality of the |
---|
101 | library/commando in detail so that, if possible, all relevant |
---|
102 | cases/results are tested. Nevertheless, such a test should not run |
---|
103 | longer than, say, 10 minutes. |
---|
104 | |
---|
105 | * If useful tests do generally execute in a short time, have xx.tst, |
---|
106 | only. |
---|
107 | |
---|
108 | * Rules for providing tst files: |
---|
109 | 1.) tst files always start with the following three commandos as |
---|
110 | preamble: |
---|
111 | LIB "tst.lib"; |
---|
112 | tst_init(); |
---|
113 | tst_ignore("CVS ID $Id$"); // or, write own version number here |
---|
114 | tst_init() writes some general info to stdout (like date, |
---|
115 | uname, hostname, version, etc.). The library tst.lib (contained in |
---|
116 | the Singular distribution) provides, among others, the routines |
---|
117 | tst_init() and tst_ignore(). |
---|
118 | |
---|
119 | 2.) tst files should end with the following statements: |
---|
120 | tst_status(1); $ |
---|
121 | which enables (automatic) checks of the timing/memory performance of |
---|
122 | Singular. |
---|
123 | |
---|
124 | 3.) All system-dependent output (like run-times, memory usages, |
---|
125 | pathnames, dates, etc.) should generally be avoided. |
---|
126 | |
---|
127 | 4.) After time/memory critical sections of the tst files, the |
---|
128 | command |
---|
129 | tst_status(); |
---|
130 | should be inserted. This enables (automatic) checks of the |
---|
131 | timing/memory performance of Singular since the last call to |
---|
132 | tst_status (resp. since the start-up of Singular). |
---|
133 | |
---|
134 | 5.) If system-dependent output can not be avodied, the routine |
---|
135 | tst_ignore(...) should be used: |
---|
136 | tst_ignore(val [, keyword]): 'val' can have arbitrary type for which |
---|
137 | a string conversion exists; |
---|
138 | if present, keyword must be one of the |
---|
139 | following strings: "time", "memory" |
---|
140 | tst_ignore() outputs 'val' by prepending the following prefix: |
---|
141 | no keyword -- // tst_ignore: |
---|
142 | "time" keyword -- // tst_ignore: time: |
---|
143 | "memory" keyword -- // tst_ignore: memory: |
---|
144 | which causes automatic tests to ignore these lines when doing a diff |
---|
145 | on result files. |
---|
146 | |
---|
147 | * The command for generating xx.res.gz.uu and xx.stat is: |
---|
148 | cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst |
---|
149 | |
---|
150 | * You can download the newest version of tst.lib from |
---|
151 | ftp://www.mathematik.uni-kl.de/pub/Math/Singular/singular-cd/LIBRARIES/tst.lib |
---|
152 | |
---|
153 | |
---|