OPEN

Test exhaustiveness depends on the developer
Edge case coverage
Time consuming
Non-robust test
- Unclear or ambiguous function requirement
- What kind of input our function should expect?
- How the function should behave in unexpected input scenarios?

def merge_sort(list1, list2):
    merged_list = list1 + list2
    return sorted(list(dict.fromkeys(merged_list)))

What data types in list should the function support?
Are mixed data types a valid input?
How the function should behave if it gets null as input?

Define the possible inputs
Define the properties of the output
Generate random examples
Test the properties are met.

Generalize the scenarios
Find essential features
Run many different inputs per property
Find a failing edge case
Automates writing tests
Focus on the general claim

Approach to address the test oracle problem and test case generation problem
Intorduced by T.Y. Chen in 1998
Approach to test Machine learning (Xie et al., 2010)
Predict the output
Metamorphic relations

Xie et al. (2010) defined the following MRs for machine learning:
- MR-0: Consistence with affine transformation
- MR-1.1: Permutation of class labels
- MR-1.2: Permutation of the attribute
- MR-2.1: Addition of uninformative attributes
- MR-2.2: Addition of informative attributes
- MR-3.1: Consistence with re-prediction
- MR-3.2: Additional training sample
- MR-4.1: Additional of classes by duplicating samples
- MR-4.2: Additional of classes by re-labeling samples
- MR-5.1: Removal of classes
- MR-5.2: Removal of samples

A library designed to help write property-based tests.
The key object of Hypothesis is a strategy.
A strategy - a recipe for generating data
Easy to use predefined strategies
Focuses on finding edge cases
Custom strategies.

@given(st.integers(), st.integers())
def test_given_integers_add_is_commutative(x, y):
    assert x + y == y + x

@given(st.floats(allow_nan=False, allow_infinity=False), st.floast(allow_nan=False, allow_infinity=False))
def test_given_floats_add_is_commutative(x, y):
    assert x + y == y + x

booleans(), text(), integers(), none(), one_of(), dates(), dictionaries(), just(), permutations() and more.

@given(arrays(int, st.shared(array_shapes(min_dims=3, max_dims=5), key="shape")), arrays(int, st.shared(array_shapes(min_dims=3, max_dims=5), key="shape")))
def test_given_arrays_multiply_is_commutative(arr1, arr2):
    np.array_equal(arr1 * arr2, arr2 * arr1)

data()

@given(array_shapes(min_dims=3, max_dims=5), st.data())
def test_given_arrays_multiply_is_commutative(arr_shape, data):
    arr1 = data.draw(arrays(int, arr_shape))
    arr2 = data.draw(arrays(int, arr_shape))
    np.array_equal(arr1 * arr2, arr2 * arr1)

@given(st.lists(st.integers() | st.floats(allow_nan=False)), st.lists(st.integers() | st.floats(allow_nan=False)))
def test_commutativity(list1, list2):
    assert merge_sort(list1, list2) == merge_sort(list2, list1)

builds(target, /, *args, **kwargs) ~ pass generated args to callable
flatmap ~ chain strategies together
Composite strategies ~ create custom strategies
Transforming data functions:
- Mapping
- Filtering

class Rectangle:
    """ A class of Python object that describe the properties of a rectangle """
    def __init__(self, width, height, center=(0, 0)):
        self.width = width
        self.height = height
        self.center = center
 
    def __repr__(self):
        return "Rectangle(width={w}, height={h}, center={c})".format(h=self.height, w=self.widht, c=self.center)
 
    def __lt__(self, other):
        return self.get_area() < other.get_area()
 
    def get_area(self):
        return self.with * self.height
 
def rectangle_list_strategy():
    return st.lists(st.builds(Rectangle, st.integers(min_value=0), st.integers(min_value=0), st.tuples(st.integers(), st.integers())))
 
@given(rectangle_list_strategy())
def test_given_rectangle_list_sort_is_distinct(rectangle_list):
    assert sorted(rectangle_list) == sorted(sorted(rectangle_list))

@given(st.integers().filter(lambda num: num % 2 ==0))
def test_given_even_number_transform_is_even(num):
    assert (num + 2) % 2 == 0

@given(st.integers().map(lambda num: num * 2))
def test_given_even_numbers_transform_is_even(num):
    assert (num + 2) % == 0

example() - return example value of strategy.

def list_strategy():
    return st.lists(st.one_of(st.integers(), st.floats(allow_nan=False)))

note() ~ print additional information on failure
@settings decorator ~ tweak hypothesis defults
- max_examples ~ the default is 100.
- database ~ the default storage is a directory structure, located in .hypothesis/examples
- verbosity
~~hypothesis~show~statistics

A test can run successfully a couple of times before finding a failing example
Repeat failing with the same example
Local test database
A bug will never go away by chance

Product human readable examples
Turn complex example into a simpler one
Each strategy defines an order in which it shrinks
Example: booleans() shrink towards False

from scipy import ndimage
from hypothesis.extra.numpy import arrays, array_shapes
 
@composite
def add_additional_blobs_to_prediction_strategy(draw, blob_prediction):
    dilated_blob_mask = ...
    ...
    return prediction
 
@given(arrays(bool, array_shape(min_dims=3, min_side=10)), st.data())  
def test_given_prediction_adding_blobs_return_include_features(raw_prediction, data):
    modified_prediction = data.draw(add_additional_blobs_to_prediction_strategy(raw_prediction))
    assert np.all(np.isin(extract_blob_mask_features(raw_prediction), extract_blob_mask_features(modified_prediction)))