Making pytest the standard test suite (instead of nose)

[Thomas De Schampheleire]

Hi Brianna, all,

On Tue, Jun 2, 2015 at 6:34 PM, Thomas De Schampheleire
<patrickdepinguin at gmail.com> wrote:
> On June 2, 2015 4:10:19 PM CEST, Brianna Laugher <brianna.laugher at gmail.com> wrote:
>>On 31 May 2015 at 12:57, Thomas De Schampheleire
>><patrickdepinguin at gmail.com
>>> wrote:
>>
>>> Hi,
>>>
>>> I've been using pytest for a while now, and think we should consider
>>> making it the default test suite instead of nose.
>>>
>>> As I see it, there are following points to do:
>>>
>>> 1. silence the database setup code by default. Currently, the initial
>>> step of the tests sets up the database and all commands are passing
>>by
>>> on the terminal. This should be silent, just like with nose.
>>> However, I don't know how to fix this correctly, and I hope Marc,
>>> Ronny or Brianna could help here.
>>
>>
>>So I downloaded kallithea and had a bit of a poke around. My impression
>>is
>>that the database setup is done in kallithea/tests/conftest.py
>>pytest_configure, specifically loadapp. While under nosetests, more or
>>less
>>the same is done at pylons.tests PylonsPlugin.begin. So I think the
>>Pylons
>>nose plugin is suppressing all output (and in fact I couldn't make it
>>deliberately show me the output).
>
> IIRC, 'nosetests -s' shows that output.
>
>>
>>Whereas with pytest, I think it is not expecting test setup to be done
>>in
>>pytest_configure, and so the options for controlling output have no
>>effect
>>on what happens in this function.
>>
>>So 1) Maybe pytest should be changed to allow suppression of output
>>from
>>pytest_configure
>>or 2) Maybe this functionality belongs under a different hook function.
>
> Any hook is fine for us, as long as it works and we can hide the output of the setup code. :)
>
>>
>>With pytest style fixtures I don't think it would be a problem. However
>>maybe there still would need to be something at this level, I don't
>>fully
>>understand the implications of what is written here re nosetest, maybe
>>the
>>same problems would occur with pytest.
>>http://pylons-webframework.readthedocs.org/en/latest/testing.html
>>
>
> I have no idea about this, I guess it needs to be tried to know for sure.
>
> Thanks for looking at this!
>

Another thing that is not yet working properly with pytest is the
feature that runs the setup code once, and then directly starts
testing, as mentioned here:
http://kallithea.readthedocs.org/en/latest/contributing.html#running-tests

The result output of:
KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 py.test
kallithea/tests/functional/
is:

Results (239.58s):
     383 passed
      18 failed
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:84:
AssertionError: no repo vcs_test_git_new in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:175:
AssertionError: no repo ingroup in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:324:
AssertionError: no repo ingroup_inherited_git in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:267:
AssertionError: no repo ingroup in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:124:
AssertionError: no repo vcs_test_git_newąęł in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/__init__.py:221:
AssertionError: msg `Error creating repository vcs_test_git_new` not
found - session has no flash
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:399:
AssertionError: no repo vcs_test_new_to_delete_git in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:451:
AssertionError: no repo vcs_test_git_newąęł in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:84:
AssertionError: no repo vcs_test_hg_new in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:175:
AssertionError: no repo ingroup in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:324:
AssertionError: no repo ingroup_inherited_hg in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:267:
AssertionError: no repo ingroup in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:124:
AssertionError: no repo vcs_test_hg_newąęł in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/__init__.py:221:
AssertionError: msg `Error creating repository vcs_test_hg_new` not
found - session has no flash
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:399:
AssertionError: no repo vcs_test_new_to_delete_hg in filesystem
         - /home/tdescham/repo/contrib/kallithea/kallithea/tests/functional/test_admin_repos.py:451:
AssertionError: no repo vcs_test_hg_newąęł in filesystem
         - /home/tdescham/repo/contrib/kallithea-venv/local/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py:445:
IndexError: Body does not contain string 'var data = {"totalRecords":
16'
         - /home/tdescham/repo/contrib/kallithea-venv/local/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py:445:
IndexError: Body does not contain string '{"totalRecords": 0, "sort":
null, "startIndex": 0, "dir": "asc", "records": []};'


Fixing this sounds like a prerequisite for pytest-as-default-test-runner too.
I have no idea whether this is difficult...

Best regards,
Thomas