Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review the message for skipping dependent tests #12

Open
RKrahl opened this issue Oct 8, 2017 · 0 comments
Open

Review the message for skipping dependent tests #12

RKrahl opened this issue Oct 8, 2017 · 0 comments
Labels
enhancement New feature or request

Comments

@RKrahl
Copy link
Owner

RKrahl commented Oct 8, 2017

PR #11 changed the message provided when a test is skipped due to missing dependencies in that the name of the skipped test was added. There are however still inconsistencies and issues with these messages that the PR did not address.

In the most basic case, the skip messages look like:

$ py.test -rs basic.py
============================= test session starts ==============================
[...]
=========================== short test summary info ============================
SKIP [1] /usr/lib/python3.4/site-packages/pytest_dependency.py:65: test_e depends on test_c
SKIP [1] /usr/lib/python3.4/site-packages/pytest_dependency.py:65: test_c depends on test_a

================ 2 passed, 2 skipped, 1 xfailed in 0.02 seconds ================

The most obvious issue is the trace line /usr/.../pytest_dependency.py:65 that appears in each skip message. It is due to the fact that skipping a test in pytest is an exception and this is the trace with the line of code that calls pytest.skip(). It is always the same in each skip message and clutters the message without containing any useful bit of information. It should be removed. Unfortunately, adding this trace is hard coded deep in the internals of pytest, so it won't be easy to get rid of it.

The name of the dependent test appearing in the message is actually the name of the function or method of the test:

$ py.test -rs named.py 
============================= test session starts ==============================
[...]
=========================== short test summary info ============================
SKIP [1] /usr/lib/python3.4/site-packages/pytest_dependency.py:65: test_c depends on a
SKIP [1] /usr/lib/python3.4/site-packages/pytest_dependency.py:65: test_e depends on c

================ 2 passed, 2 skipped, 1 xfailed in 0.01 seconds ================

For the name of the dependent test, the message ignores the name set with the name argument to the pytest.mark.dependency() marker, while for the dependency, this setting is taken into account. This is somewhat inconsistent.

This inconsistency would be acceptable, if it would lead to unambiguous names. Unfortunately, this is not the case. Consider the following example:

import pytest

def test_b():
    pass

class TestClass(object):

    @pytest.mark.dependency()
    @pytest.mark.xfail(reason="deliberate fail")
    def test_a(self):
        assert False

    @pytest.mark.dependency(depends=["TestClass::test_a"])
    def test_b(self):
        pass

The output looks like:

$ py.test -rs testclass.py 
============================= test session starts ==============================
[...]
=========================== short test summary info ============================
SKIP [1] /usr/lib/python3.4/site-packages/pytest_dependency.py:65: test_b depends on TestClass::test_a

================ 1 passed, 1 skipped, 1 xfailed in 0.01 seconds ================

This is misleading, one may believe that the test function test_b() had been skipped, but it was the method TestClass::test_b() in fact.

@RKrahl RKrahl added the enhancement New feature or request label Oct 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant