tests/README
author Danek Duvall <danek.duvall@sun.com>
Mon, 21 Aug 2006 14:13:27 -0700
changeset 2985 4cdb68d7eb92
parent 2935 1c66aad252f9
child 2991 f63667f694de
permissions -rw-r--r--
patch queue: portability.notes
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
331
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
     1
A simple testing framework
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
     2
332
6c869059beb4 testing fixups
mpm@selenic.com
parents: 331
diff changeset
     3
To run the tests, do:
6c869059beb4 testing fixups
mpm@selenic.com
parents: 331
diff changeset
     4
6c869059beb4 testing fixups
mpm@selenic.com
parents: 331
diff changeset
     5
cd tests/
2207
8a2a7f7d9df6 Delete the shell version of run-tests
Stephen Darnell <stephen@darnell.plus.com>
parents: 1933
diff changeset
     6
python run-tests.py
332
6c869059beb4 testing fixups
mpm@selenic.com
parents: 331
diff changeset
     7
331
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
     8
This finds all scripts in the test directory named test-* and executes
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
     9
them. The scripts can be either shell scripts or Python. Each test is
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    10
run in a temporary directory that is removed when the test is complete.
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    11
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    12
A test-<x> succeeds if the script returns success and its output
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    13
matches test-<x>.out. If the new output doesn't match, it is stored in
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    14
test-<x>.err.
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    15
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    16
There are some tricky points here that you should be aware of when
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    17
writing tests:
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    18
2283
e506c14382fd deprecate 'update -m'. use 'merge' instead.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2207
diff changeset
    19
- hg commit and hg merge want user interaction
331
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    20
761
0fb498458905 Change all references to -t --text commit message to -m and --message.
Andrew Thompson <andrewkt@aktzero.com>
parents: 350
diff changeset
    21
  for commit use -m "text"
2283
e506c14382fd deprecate 'update -m'. use 'merge' instead.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2207
diff changeset
    22
  for hg merge, set HGMERGE to something noninteractive (like true or merge)
331
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    23
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    24
- changeset hashes will change based on user and date which make
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    25
  things like hg history output change
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    26
1933
7544700fd931 Use 'hg ci -d "1000000 0"' in tests to circumvent problem with leading zero.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 814
diff changeset
    27
  use commit -m "test" -u test -d "1000000 0"
331
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    28
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    29
- diff will show the current time
55f63f3b6a54 Add a simple testing framework
mpm@selenic.com
parents:
diff changeset
    30
2935
1c66aad252f9 sed portability fixes
Brendan Cully <brendan@kublai.com>
parents: 2283
diff changeset
    31
  use hg diff | sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
1c66aad252f9 sed portability fixes
Brendan Cully <brendan@kublai.com>
parents: 2283
diff changeset
    32
                    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/"
1c66aad252f9 sed portability fixes
Brendan Cully <brendan@kublai.com>
parents: 2283
diff changeset
    33
  to strip dates
2985
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    34
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    35
You also need to be careful that the tests are portable from one platform
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    36
to another.  You're probably working on Linux, where the GNU toolchain has
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    37
more (or different) functionality than on MacOS, *BSD, Solaris, AIX, etc.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    38
While testing on all platforms is the only sure-fire way to make sure that
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    39
you've written portable code, here's a list of problems that have been
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    40
found and fixed in the tests.  Another, more comprehensive list may be
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    41
found in the GNU Autoconf manual, online here:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    42
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    43
    http://www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    44
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    45
sh:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    46
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    47
The Bourne shell is a very basic shell.  /bin/sh on Linux is typically
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    48
bash, which even in Bourne-shell mode has many features that Bourne shells
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    49
on other Unix systems don't have (and even on Linux /bin/sh isn't
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    50
guaranteed to be bash).  You'll need to be careful about constructs that
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    51
seem ubiquitous, but are actually not available in the least common
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    52
denominator.  While using another shell (ksh, bash explicitly, posix shell,
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    53
etc.) explicitly may seem like another option, these may not exist in a
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    54
portable location, and so are generally probably not a good idea.  You may
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    55
find that rewriting the test in python will be easier.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    56
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    57
- don't use pushd/popd; save the output of "pwd" and use "cd" in place of
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    58
  the pushd, and cd back to the saved pwd instead of popd.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    59
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    60
- don't use math expressions like let, (( ... )), or $(( ... )); use "expr"
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    61
  instead.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    62
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    63
grep:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    64
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    65
- don't use the -q option; redirect stdout to /dev/null instead.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    66
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    67
- don't use extended regular expressions with grep; use egrep instead, and
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    68
  don't escape any regex operators.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    69
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    70
sed:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    71
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    72
- make sure that the beginning-of-line matcher ("^") is at the very
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    73
  beginning of the expression -- it may not be supported inside parens.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    74
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    75
echo:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    76
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    77
- echo may interpret "\n" and print a newline; use printf instead if you
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    78
  want a literal "\n" (backslash + n).
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    79
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    80
false:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    81
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    82
- false is guaranteed only to return a non-zero value; you cannot depend on
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    83
  it being 1.  On Solaris in particular, /bin/false returns 255.  Rewrite
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    84
  your test to not depend on a particular return value, or create a
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    85
  temporary "false" executable, and call that instead.
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    86
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    87
diff:
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    88
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    89
- don't use the -N option.  There's no particularly good workaround short
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    90
  of writing a reasonably complicated replacement script, but substituting
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    91
  gdiff for diff if you can't rewrite the test not to need -N will probably
4cdb68d7eb92 patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents: 2935
diff changeset
    92
  do.