ie succie succ.\}

testie

simple test harness

SYNOPSIS

  testie [OPTIONS] [FILE]...

DESCRIPTION

Testie is a simple test harness. Each testie test file incorporates a shell script to be run and, optionally, input and expected output files for that script. Testie runs the script; the test fails if any of the script commands fail, or if the script generates unexpected output.

To run testie, pass it one or more test filenames. It will print useful error messages for failed tests. Alternatively, give it directory names; the directories are recursively searched for ’*.testie’ files.

Return status is 0 if all tests succeed, 1 if any test fails, and 2 if a test fails due to an internal error. Tests whose %require prerequisites fail do not affect the return status, except that if all tests’ prerequisites fail, the return status is 1 instead of 0.

OPTIONS

VARIABLE=VALUE
Provide an environment variable setting for VARIABLE within the script.
-V, --verbose
Print information to standard error about successful tests as well as unsuccessful tests.
-VV, --superverbose
Like --verbose, but use a slightly different format, and additionally print every test’s %info section before the test results.
-q, --quiet
Don’t print information to the terminal while running multiple tests.
-v, --version
Print version number information and exit.
--help
Print help information and exit.
--preserve-temporaries
Preserve the temporary directory created for the test.
-s, --show FILE
Echo the contents of FILE on completion. FILE should be one of the filenames specified by %file or %expect*, or ’stdout’ or ’stderr’. Leaves out any ignored lines.
-S, --show-raw FILE
Like --show, but includes any ignored lines.
--show-all
Like ’--show’ for all filenames specified by any %expect*, plus ’stdout’ and ’stderr’. Leaves out any ignored lines.
--show-all-raw
Like ’--show-raw’ for all filenames specified by any %expect*, plus ’stdout’ and ’stderr’. Includes any ignored lines.
-e, --expand
Don’t run the given test; instead, expand its files into the current directory. The script is stored in a file called ’+script+’.
-jN, --jobs=N
Run up to N tests simultaneously. Like Make’s ’-j’ option.

FILE FORMAT

Testie test files consist of several sections, each introduced by a line starting with %. There must be, at least, a %script section.

The %file and %expect* sections define input and/or output files by name. Testie runs its script in a private directory in /tmp; any files mentioned in %file or %expect* are placed in that directory.

%script
The shell script (in sh syntax) that controls the test. Testie will run each command in sequence. Every command in the script must succeed, with exit status 0, or the test will fail. Use %file sections to define script input files and %expect* sections to check script output files for expected values.

The %script section can contain multiple subtests. To start a new subtest, execute a command like “testie_subtest SECTIONNAME”. Testie will report the offending SECTIONNAME when standard output or error doesn’t match an expected value.
%require [-q]
A shell script (in sh syntax) defining prerequisites that must be satisfied before the test can run. Every command in the script must succeed, with exit status 0, for the test to run. Standard output and error are not checked, however. The -q flag tells testie not to print an error message if a requirement fails.

Testie runs the requirement script before creating any other test files. For example, contents of %file sections are not available.
%info
A short description of the test. In --superverbose mode, the first paragraph of its contents is printed before the test results.
%cut
This section is ignored. It is intended to comment out obsolete parts of the test.
%file [-d] [+LENGTH] FILENAME...
Create an input file for the script. FILENAME can be ’stdin’, which sets the script’s standard input. If LENGTH is provided, the file data consists of the LENGTH bytes following this line. Otherwise, it consists of the data up to the next section. The -d flag tells testie to delete the first character of each line in the section; this makes it possible to include files that have lines that start with %.
%expectv [-ad] [+LENGTH] FILENAME...
An expected output file for the script. FILENAME can be ’stdout’, for standard output. If LENGTH is provided, the file data consists of the LENGTH bytes following this line; otherwise, it consists of the data up to the next section.

Testie will run the script, then compare the script’s output file with the provided data. They must match exactly or the test fails.

The -a flag marks this expected output as an alternate. Testie will compare the script’s output file with each provided alternate; the test succeeds if any of the alternates match. The -d flag behaves as in %file.
%expect [-adiw] [+LENGTH] FILENAME...
An expected output file for the script. Arguments are as for %expectv.

Testie will run the script, then compare the file generated by script with the provided data. The files are compared line-by-line. Testie ignores blank lines and trailing whitespace on each line. It also ignores lines in the script output that match %ignore patterns (see below). %expect lines can contain Perl regular expressions, enclosed by two sets of braces; so the %expect line

    foo{{(bar)?}}
matches either ’foo’ or ’foobar’.

Document an %expect line with “{{?comment}}” blocks. For example:

    foo                {{? the sort was in the right order}}
Testie ignores whitespace before and after the “{{?comment}}” block, and if the actual output differs from this expected line, it prints the comment in addition to the line differences.

The -a and -d flags may also be used for %expect sections. Also, the -i flag makes any regular expressions case-insensitive (text outside of regular expressions must match case), and the -w flag ignores any differences in amount of whitespace within a line.
%expectx [-adiw] [+LENGTH] FILENAME...
%expectx is just like %expect, except that every line is treated as a regular expression. The input is parsed for “{{?comment}}” blocks, but other brace pairs are treated according to the normal regular expression rules.
%stdin [+LENGTH]
Same as ’%file stdin [ARGS]’.
%stdout [-adiw] [+LENGTH]
Same as ’%expect stdout’.
%stderr [-adiw] [+LENGTH]
Same as ’%expect stderr’.
%ignorex [-di] [+LENGTH] [FILENAME]
Each line in the %ignorex section is a Perl regular expression. Lines in the supplied FILENAME that match any of those regular expressions will not be considered when comparing files with %expect data. The regular expression must match the whole line. FILENAME may be ’all’, in which case the regular expressions will apply to all %expect files. “{{?comment}}” blocks are ignored.
%ignore, %ignorev
Like ’%ignorex’, but ’%ignore’ parses regular expressions only inside double braces (“{{ }}”), and ’%ignorev’ lines must match exactly.
%include FILENAME
Interpolate the contents of another testie file.
%eot
Marks the end of the current test. The rest of the file will be parsed for additional tests.
%eof
The rest of the file is ignored.

EXAMPLE

This simple testie script checks that ’grep -c’ works for a simple output file.

  %script
  grep -c B.
  %stdin
  Bfoo
  B
  %stdout
  1

ENVIRONMENT

By default, testie sets the LC_ALL environment variable to “C”; without this setting commands like ’sort’ have unpredictable effects. To set LC_ALL to another value, set it in the %script section.

AUTHOR

Eddie Kohler, <[email protected]>