Pytest

Introduction

testing with pytest pytest can be used for all types and levels of software testing. Many projects – amongst them Mozilla and Dropbox - switched from unittest or nose to pytest.

A Simple First Example with Pytest

Test files which pytest will use for testing have to start with test_ or end with _test.py We will demonstrate the way of working by writing a test file test_fibonacci.py for a file fibonacci.py. Both files are in one directory:

The first file is the file which should be tested. We assume that it is saved as fibonacci.py:
def fib(n):
    old, new = 0, 1
    for _ in range(n):
        old, new = new, old + new
    return old


Now, we have to provide the code for the file test_fibonacci.py. This file will be used by 'pytest':
from fibonacci import fib


def test_fib():
    assert fib(0) == 0
    assert fib(1) == 1
    assert fib(10) == 55
We call pytest in a command shell in the directory where the two file shown above reside:
(base) bernd@moon:~$ pytest
The result of this code can be seen in the following:
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/pytest_ex1, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 1 item                                                               

test_fibonacci.py .                                                      [100%]

=========================== 1 passed in 0.01 seconds ===========================
We create now an erroneous version of fib. We change the two start values from 0 amd 1 to the values 2 and 1. This is the beginning of the Lucas sequence, but as we want to implement the Fibonacci sequence this is faulty. This way, we can study how pytest behaves in this case:
def fib(n):
    old, new = 2, 1
    for _ in range(n):
        old, new = new, old + new
    return old
Calling 'pytest' with this erroneous implementation of fibonacci gives us the following results:
$ pytest  ============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/pytest_ex1, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 1 item                                                               

test_fibonacci.py F                                                      [100%]

=================================== FAILURES ===================================
___________________________________ test_fib ___________________________________

    def test_fib():
>       assert fib(0) == 0
E       assert 2 == 0
E        +  where 2 = fib(0)

test_fibonacci.py:5: AssertionError
=========================== 1 failed in 0.03 seconds ===========================


Another Pytest Example

We will get closer to 'reality' in our next example. In a real life scenario, we will usually have more than one file and for each file we may have a corresponding test file. Each test file may contain various tests. We have various files in our example folder ex2:

The files to be tested:

The test files: We start 'pytest' in the directory 'ex2' and get the following results:
$ pytest 
==================== test session starts ======================
platform linux -- Python 3.7.3, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/bernd/Dropbox (Bodenseo)/websites/python-course.eu/examples/pytest/ex2, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
collected 4 items                                                              

test_fibonacci.py .                                                      [ 25%]
test_foobar.py ..                                                        [ 75%]
test_foobar_plus.py .                                                    [100%]

==================== 4 passed in 0.05 seconds =================


Another Pytest Example

It is possible to execute only tests, which contain a given substring in their name. The substring is determined by a Python expression This can be achieved with the call option „-k“
pytest -k
The call
pytest -k foobar
will only execute the test files having the substring 'foobar' in their name. In this case, they are test_foobar.py and test_foobar_plus.py:
$ pytest -k foobar
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex2, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 3 items / 1 deselected                                               

test_foobar.py .                                                         [ 50%]
test_foobar_plus.py .                                                    [100%]

==================== 2 passed, 1 deselected in 0.01 seconds ====================

We will select now only the files containing 'plus' and 'fibo'
pytest -k 'plus or fibo'
$ pytest -k 'plus or fibo'
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex2, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 3 items / 1 deselected                                               

test_fibonacci.py .                                                      [ 50%]
test_foobar_plus.py .                                                    [100%]

==================== 2 passed, 1 deselected in 0.01 seconds ====================


Markers in Pytest

Test functions can be marked or tagged by decorating them with 'pytest.mark.'.

Such a marker can be used to select or deselect test functions. You can see the markers which exist for your test suite by typing

$ pytest --markers
$ pytest --markers
@pytest.mark.openfiles_ignore: Indicate that open files should be ignored for this test

@pytest.mark.remote_data: Apply to tests that require data from remote servers

@pytest.mark.internet_off: Apply to tests that should only run when network access is deactivated

@pytest.mark.filterwarnings(warning): add a warning filter to the given test. see https://docs.pytest.org/en/latest/warnings.html#pytest-mark-filterwarnings 

@pytest.mark.skip(reason=None): skip the given test function with an optional reason. Example: skip(reason="no way of currently testing this") skips the test.

@pytest.mark.skipif(condition): skip the given test function if eval(condition) results in a True value.  Evaluation happens within the module global context. Example: skipif('sys.platform == "win32"') skips the test if we are on the win32 platform. see https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.xfail(condition, reason=None, run=True, raises=None, strict=False): mark the test function as an expected failure if eval(condition) has a True value. Optionally specify a reason for better reporting and run=False if you don't even want to execute the test function. If only specific exception(s) are expected, you can list them in raises, and if the test fails in other ways, it will be reported as a true failure. See https://docs.pytest.org/en/latest/skipping.html

@pytest.mark.parametrize(argnames, argvalues): call a test function multiple times passing in different arguments in turn. argvalues generally needs to be a list of values if argnames specifies only one name or a list of tuples of values if argnames specifies multiple names. Example: @parametrize('arg1', [1,2]) would lead to two calls of the decorated test function, one with arg1=1 and another with arg1=2.see https://docs.pytest.org/en/latest/parametrize.html for more info and examples.

@pytest.mark.usefixtures(fixturename1, fixturename2, ...): mark tests as needing all of the specified fixtures. see https://docs.pytest.org/en/latest/fixture.html#usefixtures 

@pytest.mark.tryfirst: mark a hook implementation function such that the plugin machinery will try to call it first/as early as possible.

@pytest.mark.trylast: mark a hook implementation function such that the plugin machinery will try to call it last/as late as possible.


This contains also custom defined markers!



Registering Markers

Since pytest version 4.5 markers have to be registered.
They can be registered in the init file pytest.ini, placed in the test directory.
We register the markers 'slow' and 'crazy', which we will use in the following example:
[pytest]
markers =
    slow: mark a test as a 'slow' (slowly) running test
    crazy: stupid function to test :-)
We add a recursive and inefficient version rfib to our fibonacci module and mark the corresponding test routine with slow, besides this rfib is marked with crazy as well:
# content of fibonacci.py

def fib(n):
    old, new = 0, 1
    for i in range(n):
        old, new = new, old + new
    return old 


def rfib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return rfib(n-1) + rfib(n-2)
The corresponding test file:
#content of test_fibonacci.py

import pytest
from fibonacci import fib, rfib

def test_fib():
    assert fib(0) == 0
    assert fib(1) == 1
    assert fib(34) == 5702887

@pytest.mark.crazy
@pytest.mark.slow
def test_rfib():
    assert fib(0) == 0
    assert fib(1) == 1
    assert rfib(34) == 5702887
Besides this we will add the files foobar.py and test_foobar.py as well. We mark the test functions in test_foobar.py as crazy.
# content of foobar.py

def foo():
    return "foo"

def bar():
    return "bar"
This is the correponding test file:
# content of test_foobar.py
import pytest
from foobar import foo, bar

@pytest.mark.crazy
def test_foo():
    assert foo() == "foo"


@pytest.mark.crazy
def test_bar():
    assert bar() == "bar"
We will start tests now depending on the markers.
Let's start all tests, which are not marked as slow:
$ pytest -svv -k "slow"
===================================== test session starts ======================================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0 -- /home/bernd/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_tagging, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 4 items / 3 deselected                                                               

test_fibonacci.py::test_rfib PASSED

============================ 1 passed, 3 deselected in 7.05 seconds ============================
We will run now only the tests which are not marked as slow or crazy:
$ pytest -svv -k "not slow and not crazy"
======================= test session starts =======================
platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0 -- /home/bernd/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_tagging, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.1, doctestplus-0.2.0, arraydiff-0.3
collected 4 items / 3 deselected                                                               

test_fibonacci.py::test_fib PASSED

===================== 1 passed, 3 deselected in 0.01 seconds ====================


skipif Marker

If you wish to skip a function conditionally then you can use skipif. In the following example the function test_foo is marked with a skipif. The function will not be executed, if the Python version is 3.6.x:
import pytest
import sys
from foobar import foo, bar


@pytest.mark.skipif(
    sys.version_info[0] == 3 and sys.version_info[1] == 6,
    reason="Python version has to be higher than 3.5!")
def test_foo():
    assert foo() == "foo"

@pytest.mark.crazy
def test_bar():
    assert bar() == "bar"

Instead of a conditional skip we can also use an uncoditional skip. This way we can always skip. We can add a reason. The following example shows how this can be accomplished by marking the function test_bar with a skip marker. The reason we give is that it is "even fooer than foo":
import pytest
import sys
from foobar import foo, bar

@pytest.mark.skipif(
    sys.version_info[0] == 3 and sys.version_info[1] == 6,
    reason="Python version has to be higher than 3.5!")
def test_foo():
    assert foo() == "foo"


@pytest.mark.skip(reason="Even fooer than foo, so we skip!")
def test_bar():
    assert bar() == "bar"
If we call pytest on this code, we get the following output:
$ pytest -v
================ test session starts ===============
platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/bernd/anaconda3/envs/unittest/bin/python
cachedir: .pytest_cache
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_tagging2, inifile: pytest.ini
collected 4 items                                                                                                  

test_fibonacci.py::test_fib PASSED                                                                           [ 25%]
test_fibonacci.py::test_rfib PASSED                                                                          [ 50%]
test_foobar.py::test_foo SKIPPED                                                                             [ 75%]
test_foobar.py::test_bar PASSED                                                                              [100%]

============= 3 passed, 1 skipped in 0.01 seconds =============


Parametrization with Markers

We will demonstrate parametrization with markers with our Fibonacci function.
# content of fibonacci.py

def fib(n):
    old, new = 0, 1
    for _ in range(n):
        old, new = new, old + new
    return old 
We write a pytest test function which will test against this fibonacci function with various values:
# content of the file test_fibonacci.py

import pytest

from fibonacci import fib

@pytest.mark.parametrize(
    'n, res', [(0, 0), 
               (1, 1), 
               (2, 1),
               (3, 2), 
               (4, 3),
               (5, 5),
               (6, 8)])
def test_fib(n, res):
    assert fib(n) == res
When we call pytest, we get the following results:
$ pytest -v
============================ test session starts ============================
platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/bernd/anaconda3/envs/unittest/bin/python
cachedir: .pytest_cache
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_parametrization1
collected 7 items                                                                              

test_fibonacci.py::test_fib[0-0] PASSED                                                  [ 14%]
test_fibonacci.py::test_fib[1-1] PASSED                                                  [ 28%]
test_fibonacci.py::test_fib[2-1] PASSED                                                  [ 42%]
test_fibonacci.py::test_fib[3-2] PASSED                                                  [ 57%]
test_fibonacci.py::test_fib[4-3] PASSED                                                  [ 71%]
test_fibonacci.py::test_fib[5-5] PASSED                                                  [ 85%]
test_fibonacci.py::test_fib[6-8] PASSED                                                  [100%]

========================== 7 passed in 0.01 seconds =========================
The numbers inside of the square brackets on front of the word "PASSED" are the values of 'n' and 'res'.

Prints in Functions

If there are prints in the functions which we test, we will not see this output in our pytests, unless we call pytest with the option "-s".
To demonstrate this we will add a print line to our fibonacci function:
def fib(n):
    old, new = 0, 1
    for _ in range(n):
        old, new = new, old + new
    print("result: ", old)
    return old
Calling "pytest -s -v" will deliver the following output:

=============== test session starts ==============
platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/bernd/anaconda3/envs/unittest/bin/python
cachedir: .pytest_cache
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_parametrization1
collected 7 items                                                                              

test_fibonacci.py::test_fib[0-0] result:  0
PASSED
test_fibonacci.py::test_fib[1-1] result:  1
PASSED
test_fibonacci.py::test_fib[2-1] result:  1
PASSED
test_fibonacci.py::test_fib[3-2] result:  2
PASSED
test_fibonacci.py::test_fib[4-3] result:  3
PASSED
test_fibonacci.py::test_fib[5-5] result:  5
PASSED
test_fibonacci.py::test_fib[6-8] result:  8
PASSED

============= 7 passed in 0.01 seconds =================


Command Line Options / Fixtures

We will write a test version for our fibonacci function which depends on command line arguments. We can add custom command line options to pytest with the pytest_addoption hook that allows us to manage the command line parser and the setup for each test.
At first, we have to write a file conftest.py with the functions cmdopt and pytest_addoption:
import pytest

def pytest_addoption(parser):
    parser.addoption("--cmdopt", 
                     action="store", 
                     default="full", 
                     help="'num' of tests or full")

@pytest.fixture
def cmdopt(request):
    return request.config.getoption("--cmdopt")
The code for our fibonacci test module looks like. The test_fif function has a parameter 'cmdopt' which gets the parameter option:
from fibonacci import fib

results = [0, 1, 1, 2, 3, 5, 8, 13, 21, 
           34, 55, 89, 144, 233, 377]


def test_fib(cmdopt):
    if cmdopt == "full":
        num = len(results)
    else:
        num = len(results)
        if int(cmdopt) < len(results):
            num = int(cmdopt)
    for i in range(num):
        assert fib(i) == results[i]
We can call it now with various options, as we can see in the following:
$ pytest -q --cmdopt=full -v -s
============ test session starts ================
platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_cmd_line
collected 1 item                                                                               

test_fibonacci.py running 15 tests!
.

============= 1 passed in 0.01 seconds ============



$ pytest -q --cmdopt=6 -v -s
============= test session starts ==============
platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_cmd_line
collected 1 item                                                                               

test_fibonacci.py running  6 tests!
.

=========================== 1 passed in 0.01 seconds ================================
Let's put an error in our test results:
results = [0, 1, 1, 2, 3, 1001, 8,…]
Calling pytest with 'pytest -q --cmdopt=10 -v -s' gives us the following output:
================== test session starts ================== platform linux -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/bernd/Dropbox (Bodenseo)/kurse/python_en/examples/pytest/ex_cmd_line collected 1 item test_fibonacci.py running 10 tests! F =============== FAILURES =================== _______________ test_fib ___________________ cmdopt = '10' def test_fib(cmdopt): if cmdopt == "full": num = len(results) else: num = len(results) if int(cmdopt) < len(results): num = int(cmdopt) print(f"running {num:2d} tests!") for i in range(num): > assert fib(i) == results[i] E assert 5 == 1001 E + where 5 = fib(5) test_fibonacci.py:16: AssertionError ================ 1 failed in 0.03 seconds =================