Example of test failure output with pytest

Wed May 13 18:39:15 EDT 2015

Hi

Some pytest feedback:
I'm sorry for being lazy and not approaching upstream, but it is so 
appealing to talk directly to the friendly ambassadors ;-)

One use case, with 
https://bitbucket.org/conservancy/kallithea/commits/9b8ba0f1c87b4fe89481b2c2c8723f6636d147cd 
, I got (valid) test failures:

...
../kallithea-venv/lib/python2.7/site-packages/WebHelpers-1.3-py2.7.egg/webhelpers/pylonslib/flash.py:349: 
UnicodeEncodeError
---------------------------------------------------------------------------------------------- 
Captured stdout call 
-----------------------------------------------------------------------------------------------
7 literal(u'Created repository <a 
href="/vcs_test_hg_new%C4%85%C4%99%C5%82">vcs_test_hg_new\u0105\u0119\u0142</a>')
__________________________________________________________________________ 
TestHomeController.test_index_with_anonymous_access_disabled 
___________________________________________________________________________

self = <kallithea.tests.functional.test_home.TestHomeController 
testMethod=test_index_with_anonymous_access_disabled>

     def test_index_with_anonymous_access_disabled(self):
         with fixture.anon_access(False):
             response = self.app.get(url(controller='home', action='index'),
 >                                   status=302)

kallithea/tests/functional/test_home.py:42:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
build/bdist.linux-x86_64/egg/webob/request.py:1049: in get_response
     ???
build/bdist.linux-x86_64/egg/webob/request.py:1022: in call_application
     ???
../kallithea-venv/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/lint.py:179: 
in lint_app
     iterator = application(environ, start_response_wrapper)
.eggs/Paste-2.0.1-py2.7.egg/paste/gzipper.py:31: in __call__
     return self.application(environ, start_response)
.eggs/Paste-2.0.1-py2.7.egg/paste/cascade.py:130: in __call__
     return self.apps[-1](environ, start_response)
.eggs/Paste-2.0.1-py2.7.egg/paste/registry.py:379: in __call__
     app_iter = self.application(environ, start_response)
kallithea/lib/middleware/wrapper.py:43: in __call__
     return self.application(environ, start_response)
kallithea/lib/base.py:277: in __call__
     return self._handle_request(environ, start_response)
kallithea/lib/middleware/simplegit.py:68: in _handle_request
     return self.application(environ, start_response)
kallithea/lib/base.py:277: in __call__
     return self._handle_request(environ, start_response)
kallithea/lib/middleware/simplehg.py:73: in _handle_request
     return self.application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/middleware.py:150: 
in __call__
     self.app, environ, catch_exc_info=True)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/util.py:48: 
in call_wsgi_application
     app_iter = application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/WebError-0.10.3-py2.7.egg/weberror/errormiddleware.py:156: 
in __call__
     return self.application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Beaker-1.6.4-py2.7.egg/beaker/middleware.py:155: 
in __call__
     return self.wrap_app(environ, session_start_response)
../kallithea-venv/lib/python2.7/site-packages/Routes-1.13-py2.7.egg/routes/middleware.py:131: 
in __call__
     response = self.app(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/wsgiapp.py:107: 
in __call__
     response = self.dispatch(controller, environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/wsgiapp.py:312: 
in dispatch
     return controller(environ, start_response)
kallithea/lib/base.py:383: in __call__
     return WSGIController.__call__(self, environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:211: 
in __call__
     response = self._dispatch_call()
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:162: 
in _dispatch_call
     response = self._inspect_call(func)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:105: 
in _inspect_call
     result = self._perform_call(func, args)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:57: 
in _perform_call
     return func(**args)
<string>:2: in index
     ???
kallithea/lib/auth.py:785: in __wrapper
     return redirect_to_login()
kallithea/lib/auth.py:726: in redirect_to_login
     h.flash(h.literal(message), category='warning')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <kallithea.lib.helpers.Flash object at 0x7fe013463410>, message = 
literal(u'None'), category = 'warning', ignore_duplicate = False

     def __call__(self, message, category=None, ignore_duplicate=False):
         """Add a message to the session.

             ``message`` is the message text.

             ``category`` is the message's category. If not specified, 
the default
             category will be used.  Raise ``ValueError`` if the 
category is not
             in the list of allowed categories.

             If ``ignore_duplicate`` is true, don't add the message if 
another
             message with identical text has already been added. If the new
             message has a different category than the original message, 
change the
             original message to the new category.

             """
         print 7, repr(message)
 >       assert str(message) != 'None', message
E       AssertionError: None

../kallithea-venv/lib/python2.7/site-packages/WebHelpers-1.3-py2.7.egg/webhelpers/pylonslib/flash.py:349: 
AssertionError
---------------------------------------------------------------------------------------------- 
Captured stdout call 
-----------------------------------------------------------------------------------------------
7 literal(u'None')
_______________________________________________________________________ 
TestHomeController.test_repo_summary_with_anonymous_access_disabled 
_______________________________________________________________________

self = <kallithea.tests.functional.test_home.TestHomeController 
testMethod=test_repo_summary_with_anonymous_access_disabled>

     def test_repo_summary_with_anonymous_access_disabled(self):
         with fixture.anon_access(False):
             response = self.app.get(url(controller='summary',
                                         action='index', repo_name=HG_REPO),
 >                                       status=302)

kallithea/tests/functional/test_home.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
build/bdist.linux-x86_64/egg/webob/request.py:1049: in get_response
     ???
build/bdist.linux-x86_64/egg/webob/request.py:1022: in call_application
     ???
../kallithea-venv/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/lint.py:179: 
in lint_app
     iterator = application(environ, start_response_wrapper)
.eggs/Paste-2.0.1-py2.7.egg/paste/gzipper.py:31: in __call__
     return self.application(environ, start_response)
.eggs/Paste-2.0.1-py2.7.egg/paste/cascade.py:130: in __call__
     return self.apps[-1](environ, start_response)
.eggs/Paste-2.0.1-py2.7.egg/paste/registry.py:379: in __call__
     app_iter = self.application(environ, start_response)
kallithea/lib/middleware/wrapper.py:43: in __call__
     return self.application(environ, start_response)
kallithea/lib/base.py:277: in __call__
     return self._handle_request(environ, start_response)
kallithea/lib/middleware/simplegit.py:68: in _handle_request
     return self.application(environ, start_response)
kallithea/lib/base.py:277: in __call__
     return self._handle_request(environ, start_response)
kallithea/lib/middleware/simplehg.py:73: in _handle_request
     return self.application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/middleware.py:150: 
in __call__
     self.app, environ, catch_exc_info=True)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/util.py:48: 
in call_wsgi_application
     app_iter = application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/WebError-0.10.3-py2.7.egg/weberror/errormiddleware.py:156: 
in __call__
     return self.application(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Beaker-1.6.4-py2.7.egg/beaker/middleware.py:155: 
in __call__
     return self.wrap_app(environ, session_start_response)
../kallithea-venv/lib/python2.7/site-packages/Routes-1.13-py2.7.egg/routes/middleware.py:131: 
in __call__
     response = self.app(environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/wsgiapp.py:107: 
in __call__
     response = self.dispatch(controller, environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/wsgiapp.py:312: 
in dispatch
     return controller(environ, start_response)
kallithea/lib/base.py:383: in __call__
     return WSGIController.__call__(self, environ, start_response)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:211: 
in __call__
     response = self._dispatch_call()
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:162: 
in _dispatch_call
     response = self._inspect_call(func)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:105: 
in _inspect_call
     result = self._perform_call(func, args)
../kallithea-venv/lib/python2.7/site-packages/Pylons-1.0-py2.7.egg/pylons/controllers/core.py:57: 
in _perform_call
     return func(**args)
<string>:2: in index
     ???
kallithea/lib/auth.py:785: in __wrapper
     return redirect_to_login()
kallithea/lib/auth.py:726: in redirect_to_login
     h.flash(h.literal(message), category='warning')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <kallithea.lib.helpers.Flash object at 0x7fe013463410>, message = 
literal(u'None'), category = 'warning', ignore_duplicate = False

     def __call__(self, message, category=None, ignore_duplicate=False):
         """Add a message to the session.

             ``message`` is the message text.

             ``category`` is the message's category. If not specified, 
the default
             category will be used.  Raise ``ValueError`` if the 
category is not
             in the list of allowed categories.

             If ``ignore_duplicate`` is true, don't add the message if 
another
             message with identical text has already been added. If the new
             message has a different category than the original message, 
change the
             original message to the new category.

             """
         print 7, repr(message)
 >       assert str(message) != 'None', message
E       AssertionError: None

../kallithea-venv/lib/python2.7/site-packages/WebHelpers-1.3-py2.7.egg/webhelpers/pylonslib/flash.py:349: 
AssertionError
---------------------------------------------------------------------------------------------- 
Captured stdout call 
-----------------------------------------------------------------------------------------------
7 literal(u'None')
=============================================================================== 
6 failed, 1588 passed, 2 skipped in 191.92 seconds 
================================================================================

So ... I guess the full horizontal lines have the highest priorities and 
separates different test failures.

1. An extra empty line before would have made it more obvious (but it 
would perhaps look weird in cases where the output following the lines 
is less verbose).

2. I would expect to be able to run 'py.test 
TestHomeController.test_index_with_anonymous_access_disabled' to rerun 
the failure. But I can't. A quick look at py.test -h tells me that it 
wants a filename but doesn't give any hint I can use directly.

3. It is not clear to me what the '_ _ _ _' lines separate - having some 
title text inside these lines too could perhaps give me a clue.

4. An extra colon like '---- Captured stdout call: -----' could also 
make it more clear that it is a description of what is following.

The test might be bad and can be blamed for most of the problems, but 
py.test (or our configuration of it) could perhaps explain more 
carefully what is going on and what my options are.

/Mads