[b35b93] | 1 | README file for Singular test-suite |
---|
| 2 | =================================== |
---|
| 3 | |
---|
| 4 | This is the test-suite for Singular. This directory contains |
---|
| 5 | |
---|
[e2114af] | 6 | regress.cmd -- perl script which drives tests |
---|
[b35b93] | 7 | Short -- directory containing "short" tests |
---|
| 8 | Long -- directory containing "long" tests |
---|
| 9 | Old -- directory containing "old" (TST) tests |
---|
| 10 | |
---|
[e2114af] | 11 | Run |
---|
| 12 | perl ./regress.cmd -h |
---|
| 13 | for a summary of the comaand-line options of regress.cmd. |
---|
| 14 | |
---|
| 15 | To test(s) correctness of Singular: |
---|
| 16 | ----------------------------------- |
---|
| 17 | 1.) Put the Singular binary in this directory. |
---|
| 18 | 2.) Run |
---|
| 19 | perl ./regress.cmd [*.tst] [*.lst] |
---|
| 20 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
| 21 | of tst files -- see Old/universal.lst for an example) file(s). |
---|
| 22 | 3.) If no differences between the new results and the |
---|
| 23 | previously obtained results are found, regress.cmd exits with |
---|
| 24 | status 0. |
---|
| 25 | 4.) If differences between the new results and the |
---|
| 26 | previously obtained results are found, regress.cmd exits with |
---|
| 27 | status != 0, and, for each file xx.tst which lead to a |
---|
| 28 | difference, the coresponding files xx.new.res, xx.res, and xx.diff |
---|
| 29 | are kept, and differences are written to stdout. |
---|
| 30 | 5.) The Singular executable which is used for testing can |
---|
| 31 | alternatively be specified using the -s <Singular> option. |
---|
| 32 | |
---|
| 33 | |
---|
| 34 | To test memory/timing performance of Singular: |
---|
| 35 | ---------------------------------------------- |
---|
[b35b93] | 36 | 1.) Put the Singular binary in this directory |
---|
[e2114af] | 37 | 2.) Run |
---|
| 38 | perl ./regress.cmd -e [*.tst] [*.lst] |
---|
[b35b93] | 39 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
[e2114af] | 40 | of tst files -- see Old/universal.lst for an example) file(s). |
---|
| 41 | 3.) If no result differences occur, and if timing/memory performance |
---|
| 42 | differences are less than $error_val (currently set to 5) |
---|
| 43 | per-cent, then regress.cmd exits with status 0. |
---|
| 44 | 4.) Otherwise, exit status is != 0, and the corresponding files |
---|
| 45 | xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the |
---|
| 46 | differences of timing/memory performance) are kept, and |
---|
| 47 | differences are written to stdout. |
---|
| 48 | 5.) Using the -r instead of the -e option only reports about |
---|
| 49 | memory/timings differences, i.e., it does not cause an exit with |
---|
| 50 | status != 0 if timing/memory performance is above specified |
---|
| 51 | per-cent. |
---|
| 52 | 6.) The timing/memory performance differenes which trigger an error |
---|
| 53 | (or, report) can be also be set: Either use |
---|
| 54 | ./regress.cmd -[e,r] all%<val> |
---|
| 55 | which sets tolerance of all criteria to <val> per-cent, or, use |
---|
| 56 | ./regress.cmd -[e,r] <crit>%<val> |
---|
| 57 | which sets tolerance of all <crit> to <val> per-cent. |
---|
| 58 | |
---|
| 59 | To generate new result file(s): |
---|
| 60 | ------------------------------- |
---|
| 61 | Run |
---|
| 62 | perl ./regress.cmd -g [*.tst] [*.lst] |
---|
| 63 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
| 64 | of tst files). |
---|
| 65 | NOTE: Running regress.cmd with the -g option re-generates the result |
---|
| 66 | and stat files, i.e., all previous results and statisitics are |
---|
| 67 | overwritten and lost! |
---|
| 68 | |
---|
| 69 | To include timing/memory statistics for a machine: |
---|
| 70 | --------------------------------------------------- |
---|
| 71 | Run |
---|
| 72 | perl ./regress.cmd -m [*.tst] [*.lst] |
---|
| 73 | with the desired tst (i.e., Singular test scripts) or lst (list |
---|
| 74 | of tst files). |
---|
| 75 | NOTE: Running regress.cmd with the -m option merges the |
---|
| 76 | timings/memory results of the current machine into the respective |
---|
| 77 | xx.stat file and overwrites previous timings/memory results of the |
---|
| 78 | current machine. |
---|
[779b9b7] | 79 | |
---|
| 80 | |
---|
[b35b93] | 81 | To add a new test for a library/commando to the test-suite: |
---|
| 82 | ----------------------------------------------------------- |
---|
| 83 | * The following files need to be provided: |
---|
| 84 | short/xx_s.tst : Singular code for short and basic tests |
---|
| 85 | long/xx_l.tst : Singular code for long and extended tests |
---|
| 86 | or, alternatively: |
---|
| 87 | short/xx.tst: Singular code for short tests, only |
---|
| 88 | short/xx.res : Result output of xx.tst |
---|
| 89 | |
---|
| 90 | * xx_s.tst should test the essential functionality of the |
---|
| 91 | library/commando in a relatively short time (say, in no more than |
---|
| 92 | 30s). |
---|
| 93 | |
---|
| 94 | * xx_l.tst should test the functionality of the |
---|
| 95 | library/commando in detail so that, if possible, all relevant |
---|
| 96 | cases/results are tested. Nevertheless, such a test should not run |
---|
| 97 | longer than, say, 10 minutes. |
---|
| 98 | |
---|
[779b9b7] | 99 | * If useful tests do generally execute in a short time, have xx.tst, |
---|
[b35b93] | 100 | only. |
---|
| 101 | |
---|
| 102 | * Rules for providing tst files: |
---|
| 103 | 1.) tst files always start with the following three commandos as |
---|
| 104 | preamble: |
---|
| 105 | LIB "tst.lib"; |
---|
| 106 | tst_init(); |
---|
[e2114af] | 107 | tst_ignore("CVS ID $Id:$"); // or, write own version number here |
---|
[b35b93] | 108 | tst_init() writes some general info to stdout (like date, |
---|
[e2114af] | 109 | uname, hostname, version, etc.). The library tst.lib (contained in |
---|
| 110 | the Singular distribution) provides, among others, the routines |
---|
| 111 | tst_init() and tst_ignore(). |
---|
| 112 | |
---|
| 113 | 2.) tst files should end with the following statements: |
---|
| 114 | tst_status(1); $ |
---|
| 115 | which enables (automatic) checks of the timing/memory performance of |
---|
| 116 | Singular. |
---|
| 117 | |
---|
| 118 | 3.) All system-dependent output (like run-times, memory usages, |
---|
| 119 | pathnames, dates, etc.) should generally be avoided. |
---|
| 120 | |
---|
| 121 | 4.) After time/memory critical sections of the tst files, the |
---|
| 122 | command |
---|
| 123 | tst_status(); |
---|
| 124 | should be inserted. This enables (automatic) checks of the |
---|
| 125 | timing/memory performance of Singular since the last call to |
---|
| 126 | tst_status (resp. since the start-up of Singular). |
---|
| 127 | |
---|
| 128 | 5.) If system-dependent output can not be avodied, the routine |
---|
| 129 | tst_ignore(...) should be used: |
---|
[b35b93] | 130 | tst_ignore(val [, keyword]): 'val' can have arbitrary type for which |
---|
| 131 | a string conversion exists; |
---|
| 132 | if present, keyword must be one of the |
---|
| 133 | following strings: "time", "memory" |
---|
| 134 | tst_ignore() outputs 'val' by prepending the following prefix: |
---|
[e2114af] | 135 | no keyword -- // tst_ignore: |
---|
| 136 | "time" keyword -- // tst_ignore: time: |
---|
| 137 | "memory" keyword -- // tst_ignore: memory: |
---|
[b35b93] | 138 | which causes automatic tests to ignore these lines when doing a diff |
---|
[863a27] | 139 | on result files. |
---|
| 140 | |
---|
[e2114af] | 141 | * The command for generating xx.res.gz.uu and xx.stat is: |
---|
| 142 | cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst |
---|
[b35b93] | 143 | |
---|
| 144 | |
---|