README file for Singular test-suite =================================== This is the test-suite for Singular. This directory contains regress.cmd -- perl script which drives tests Short -- directory containing "short" tests Long -- directory containing "long" tests Old -- directory containing "old" (TST) tests recommended set of tests: ./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst Run perl ./regress.cmd -h for a summary of the comaand-line options of regress.cmd. To test(s) correctness of Singular: ----------------------------------- 1.) Put the Singular binary in this directory (prefered solution: ln -s Singular) 2.) Run perl ./regress.cmd [*.tst] [*.lst] with the desired tst (i.e., Singular test scripts) or lst (list of tst files -- see Old/universal.lst for an example) file(s). (prefered test scenario: perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst ) 3.) If no differences between the new results and the previously obtained results are found, regress.cmd exits with status 0. 4.) If differences between the new results and the previously obtained results are found, regress.cmd exits with status != 0, and, for each file xx.tst which lead to a difference, the coresponding files xx.new.res, xx.res, and xx.diff are kept, and differences are written to stdout. 5.) The Singular executable which is used for testing can alternatively be specified using the -s option. To test memory/timing performance of Singular: ---------------------------------------------- 1.) Put the Singular binary in this directory 2.) Run perl ./regress.cmd -e [*.tst] [*.lst] with the desired tst (i.e., Singular test scripts) or lst (list of tst files -- see Old/universal.lst for an example) file(s). 3.) If no result differences occur, and if timing/memory performance differences are less than $error_val (currently set to 5) per-cent, then regress.cmd exits with status 0. 4.) Otherwise, exit status is != 0, and the corresponding files xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the differences of timing/memory performance) are kept, and differences are written to stdout. 5.) Using the -r instead of the -e option only reports about memory/timings differences, i.e., it does not cause an exit with status != 0 if timing/memory performance is above specified per-cent. 6.) The timing/memory performance differenes which trigger an error (or, report) can be also be set: Either use ./regress.cmd -[e,r] all% which sets tolerance of all criteria to per-cent, or, use ./regress.cmd -[e,r] % which sets tolerance of all to per-cent. To generate new result file(s): ------------------------------- Run perl ./regress.cmd -g [*.tst] [*.lst] with the desired tst (i.e., Singular test scripts) or lst (list of tst files). NOTE: Running regress.cmd with the -g option re-generates the result and stat files, i.e., all previous results and statisitics are overwritten and lost! To include timing/memory statistics for a machine: --------------------------------------------------- Run perl ./regress.cmd -m [*.tst] [*.lst] with the desired tst (i.e., Singular test scripts) or lst (list of tst files). NOTE: Running regress.cmd with the -m option merges the timings/memory results of the current machine into the respective xx.stat file and overwrites previous timings/memory results of the current machine. To add a new test for a library/commando to the test-suite: ----------------------------------------------------------- * The following files need to be provided: short/xx_s.tst : Singular code for short and basic tests long/xx_l.tst : Singular code for long and extended tests or, alternatively: short/xx.tst: Singular code for short tests, only short/xx.res : Result output of xx.tst * xx_s.tst should test the essential functionality of the library/commando in a relatively short time (say, in no more than 30s). * xx_l.tst should test the functionality of the library/commando in detail so that, if possible, all relevant cases/results are tested. Nevertheless, such a test should not run longer than, say, 10 minutes. * If useful tests do generally execute in a short time, have xx.tst, only. * Rules for providing tst files: 1.) tst files always start with the following three commandos as preamble: LIB "tst.lib"; tst_init(); tst_ignore("CVS ID $Id$"); // or, write own version number here tst_init() writes some general info to stdout (like date, uname, hostname, version, etc.). The library tst.lib (contained in the Singular distribution) provides, among others, the routines tst_init() and tst_ignore(). 2.) tst files should end with the following statements: tst_status(1); $ which enables (automatic) checks of the timing/memory performance of Singular. 3.) All system-dependent output (like run-times, memory usages, pathnames, dates, etc.) should generally be avoided. 4.) After time/memory critical sections of the tst files, the command tst_status(); should be inserted. This enables (automatic) checks of the timing/memory performance of Singular since the last call to tst_status (resp. since the start-up of Singular). 5.) If system-dependent output can not be avodied, the routine tst_ignore(...) should be used: tst_ignore(val [, keyword]): 'val' can have arbitrary type for which a string conversion exists; if present, keyword must be one of the following strings: "time", "memory" tst_ignore() outputs 'val' by prepending the following prefix: no keyword -- // tst_ignore: "time" keyword -- // tst_ignore: time: "memory" keyword -- // tst_ignore: memory: which causes automatic tests to ignore these lines when doing a diff on result files. * The command for generating xx.res.gz.uu and xx.stat is: cd ; perl ../regress.cmd -g xx.tst * You can download the newest version of tst.lib from ftp://www.mathematik.uni-kl.de/pub/Math/Singular/singular-cd/LIBRARIES/tst.lib