tests/hypothesishelpers.py
author David R. MacIver <david@drmaciver.com>
Sat, 24 Oct 2015 12:46:03 +0100
changeset 26842 0f76c64f5cc3
child 27998 84513a4fcc3a
permissions -rw-r--r--
testing: add hypothesis fuzz testing Hypothesis a library for adding fuzzing over a range of structure data to your test suite: http://hypothesis.readthedocs.org/en/latest/ This adds the ability to build tests using Hypothesis within the Mercurial test suite. New tests and fixes using this helpers comes in later changesets.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
26842
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     1
# Helper module to use the Hypothesis tool in tests
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     2
#
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     3
# Copyright 2015 David R. MacIver
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     4
#
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     5
# For details see http://hypothesis.readthedocs.org
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     6
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     7
import os
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     8
import sys
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
     9
import traceback
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    10
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    11
from hypothesis.settings import set_hypothesis_home_dir
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    12
import hypothesis.strategies as st
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    13
from hypothesis import given, Settings
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    14
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    15
# hypothesis store data regarding generate example and code
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    16
set_hypothesis_home_dir(os.path.join(
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    17
    os.getenv('TESTTMP'), ".hypothesis"
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    18
))
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    19
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    20
def check(*args, **kwargs):
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    21
    """decorator to make a function a hypothesis test
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    22
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    23
    Decorated function are run immediately (to be used doctest style)"""
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    24
    def accept(f):
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    25
        # Workaround for https://github.com/DRMacIver/hypothesis/issues/206
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    26
        # Fixed in version 1.13 (released 2015 october 29th)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    27
        f.__module__ = '__anon__'
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    28
        try:
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    29
            given(*args, settings=Settings(max_examples=2000), **kwargs)(f)()
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    30
        except Exception:
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    31
            traceback.print_exc(file=sys.stdout)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    32
            sys.exit(1)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    33
    return accept
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    34
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    35
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    36
def roundtrips(data, decode, encode):
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    37
    """helper to tests function that must do proper encode/decode roundtripping
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    38
    """
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    39
    @given(data)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    40
    def testroundtrips(value):
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    41
        encoded = encode(value)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    42
        decoded = decode(encoded)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    43
        if decoded != value:
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    44
            raise ValueError(
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    45
                "Round trip failed: %s(%r) -> %s(%r) -> %r" % (
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    46
                    encode.__name__, value, decode.__name__, encoded,
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    47
                    decoded
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    48
                ))
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    49
    try:
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    50
        testroundtrips()
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    51
    except Exception:
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    52
        # heredoc swallow traceback, we work around it
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    53
        traceback.print_exc(file=sys.stdout)
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    54
        raise
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    55
    print("Round trip OK")
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    56
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    57
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    58
# strategy for generating bytestring that might be an issue for Mercurial
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    59
bytestrings = (
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    60
    st.builds(lambda s, e: s.encode(e), st.text(), st.sampled_from([
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    61
        'utf-8', 'utf-16',
0f76c64f5cc3 testing: add hypothesis fuzz testing
David R. MacIver <david@drmaciver.com>
parents:
diff changeset
    62
    ]))) | st.binary()