pytest adoption: current status (beginning of May)

[Mads Kiilerich]

On 05/06/2015 07:34 AM, Jan Heylen wrote:
> On Wed, May 6, 2015 at 3:46 AM, Mads Kiilerich <mads at kiilerich.com> wrote:
>> On 05/04/2015 09:01 PM, Thomas De Schampheleire wrote:
>>> Hi all,
>>>
>>> So the official part of 'pytest adoption month' is over, so it's high
>>> time to look back.
>>
>> Yes, thanks a lot for the contributions. Just making it possible to run the
>> tests with pytest was a significant amount of work. Awesome!
>>
>> pytest is now a fine alternative to nosetests ... but with some caveats:
>>
>> * it is running a different number of tests - I am not entirely sure the
>> pytest coverage is a superset of nosetest
>>
>> * we have some documentation of how to run tests with nosetests but no
>> documentation for pytest
> I must say without any prior knowledge of pytest before I started
> using it (yesterday), it is as simple as excecuting 'py.test' in the
> root of your kallithea repo.

So just as as simple as nosetests ;-)

> The only line to add to the docs is: "You can run the test with
> pytest: just execute 'py.test' in the root of kallithea repo" and the
> other documentation is the documentation of pytest, creating a subset
> of this documentation is not useful in my opinion, better a reference
> to https://pytest.org/latest/usage.html ?

There is also references to nosetests in other places outside 
documentation. If we want to replace nose with pytest, these should be 
addressed too.

I think it would be nice to have a couple of examples to get started. 
People who just want to run the tests before contributing should not 
have to go elsewhere and learn pytest; they should be able to find the 
commands they most likely will and should be able to just cut'n'paste them.

>
>> * it seems like nosetests makes it more clear what is failing and makes it
>> easy to rerun specific tests after failing (even though it often doesn't
>> work) but it doesn't seem as easy with pytest ... but that might be a
>> documentation issue
> py.test kallithea/tests/functional/test_changeset_comments.py
> or another e.g. (as seen in https://pytest.org/latest/usage.html)
> py.test -k "not api
>
> What makes you say pytest has less clear info on what is failing? As
> it has no immediate output when a test fails? (maybe checkout the -x
> option)

So after spending N minutes running the tests, I have to run them all 
again to figure out which one failed?

I do not find it easy to figure out what to do after seeing the 
following - I don't even know where one error stops and the next starts:

...
                 print_stderr("Actual response (no %r):" % s)
                 print_stderr(str(self))
                 raise IndexError(
 >                   "Body does not contain string %r" % s)
E               IndexError: Body does not contain string '<span 
class="repotag">hg'

../kallithea-venv/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py:445: 
IndexError
---------------------------------------------------------------------------------------------- 
Captured stderr call 
-----------------------------------------------------------------------------------------------
Actual response (no '<span class="repotag">hg'):Response: 302 Found
Cache-Control: no-cache
Content-Type: text/plain; charset=UTF-8
Location: http://localhost/_admin/login?came_from=%2Fvcs_test_hg
Pragma: no-cache
302 Found
The resource was found at 
http://localhost/_admin/login?came_from=%2Fvcs_test_hg; you should be 
redirected automatically.
___________________________________________________________________________________ 
TestSummaryController.test_index_by_id_git 
____________________________________________________________________________________

self = <kallithea.tests.functional.test_summary.TestSummaryController 
testMethod=test_index_by_id_git>

     def test_index_by_id_git(self):
         self.log_user()
         ID = Repository.get_by_repo_name(GIT_REPO).repo_id
         response = self.app.get(url(controller='summary',
                                     action='index',
                                     repo_name='_%s' % ID))

         #repo type
         response.mustcontain(
 >           """<span class="repotag">git"""
         )

kallithea/tests/functional/test_summary.py:105:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <302 Found text/plain location: 
http://localhost/_admin/login?came_from=%2Fvcs_test_git body='302 
Found...y.  '/135>, strings = ('<span class="repotag">git',), kw = {}, 
no = []
s = '<span class="repotag">git'

     def mustcontain(self, *strings, **kw):
         """
             Assert that the response contains all of the strings passed
             in as arguments.
...


>
>> * we haven't gotten much further than just being able to run the existing
>> tests with pytest - we haven't seen any significant benefits from the new
>> tooling yet
> for me it looks better, it is one command to do the tests, and gives
> very decent output when failing.

IMO, it is currently not as good as with nosetests.

But no doubt: Pytest is nice and has potential and is close to being 
better than nosetests, even though I haven't seen the "killer" feature yet.

/Mads