Turbogears2 migration: paging failures in changelog

[Thomas De Schampheleire]

Hi Mads,

On Mon, Jul 18, 2016 at 12:52 AM, Mads Kiilerich <mads at kiilerich.com> wrote:
> On 07/15/2016 09:32 PM, Thomas De Schampheleire wrote:
>>
>> On Tue, Jul 12, 2016 at 12:55 PM, Mads Kiilerich <mads at kiilerich.com>
>> wrote:
>>>
>>> On 06/30/2016 08:11 PM, Thomas De Schampheleire wrote:
>>>
>>> I'm now looking at the other tests. One specific failure is easy to
>>> reproduce, either:
>>>
>>> $ py.test kallithea/tests/functional/test_changelog.py
>>>
>>> or interactively by launching kallithea with the test.ini:
>>>
>>> $ paster serve kallithea/tests/test.ini
>>> then browsing to one of either repos vce_test_hg/git and then to
>>> 'Changelog'.
>>>
>>> This renders a 500 Server Error without traceback or further information.
>>> I was able to zoom in on the problem with manual traces, and found it
>>> is related to pagination. If I remove the pagination block from the
>>> template (see patch below) then the Changelog page works fine (except
>>> for pagination, of course).
>>>
>>> diff --git a/kallithea/templates/changelog/changelog.html
>>> b/kallithea/templates/changelog/changelog.html
>>> --- a/kallithea/templates/changelog/changelog.html
>>> +++ b/kallithea/templates/changelog/changelog.html
>>> @@ -160,7 +160,6 @@
>>>                    </div>
>>>
>>>                    <div class="pagination-wh pagination-left">
>>> -                    ${c.pagination.pager('$link_previous ~2~
>>> $link_next')}
>>>                    </div>
>>>                </div>
>>>            </div>
>>>
>>>
>>> However, it is unclear to me how to proceed. What is the problem with
>>> this pagination, why is it a problem under Turbogears2 and not under
>>> Pylons? How to see more details about the failures?
>>>
>>>
>
>>> I think that I am missing some debugging skills in python - Kallithea
>>> - mako - Turbogears2. I would be grateful if others with more
>>> experience could help me on that. In this problem, it looks like there
>>> is some kind of template rendering problem, but without any hint in
>>> the logs.
>>>
>>> Thanks,
>>> Thomas
>
>
>
> I tried to test it but encountered some problems trying to follow
> https://bitbucket.org/conservancy/kallithea/wiki/Turbogears2Migration.md .
>
> It seems like it really wants "setup.py develop"? "pip install -e ." doesn't
> work? Do you know ... what is the story around that?

I just created a new virtualenv, then used 'pip install --upgrade pip
setuptools; pip install -e .' without issue.
Which error did you encounter?

>
> Also, it took me a while to figure out that even though the config file and
> database must be created with another installation, the ini file must be
> changed to use sqlalchemy.url without db1 . Any other changes to watch out
> for?
>
> Please consider clarifying the wiki page.

I made some small updates now.
Currently only development.ini and test.ini have been fixed, and the
.ini generation scripts have not yet been adapted.
The instructions did say to use development.ini but did not
explicitized this so it was easy for experienced users to assume any
.ini file would be fine.

>
>
> Anyway, I do get a nice traceback on stdout when encountering the 500 from
> the pager:
>
>   File "/home/mk/kallithea-tg/kallithea/controllers/changelog.py", line 176,
> in index
>     return render('changelog/changelog.html')
>   File "/home/mk/kallithea-tg/kallithea/lib/base.py", line 65, in render
>     return render_template({'url': url}, 'mako', template_path)
>   File "/home/mk/kallithea-venv/lib/python2.7/site-packages/tg/render.py",
> line 212, in render
>     kwargs['result'] = render_function(template_name, tg_vars, **kwargs)
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/tg/renderers/mako.py",
> line 126, in __call__
>     cache_type=cache_type, cache_expire=cache_expire)
>   File "/home/mk/kallithea-venv/lib/python2.7/site-packages/tg/render.py",
> line 278, in cached_template
>     return render_func()
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/tg/renderers/mako.py",
> line 123, in render_template
>     return Markup(template.render_unicode(**template_vars))
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/Mako-1.0.0-py2.7.egg/mako/template.py",
> line 452, in render_unicode
>     as_unicode=True)
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/Mako-1.0.0-py2.7.egg/mako/runtime.py",
> line 803, in _render
>     **_kwargs_for_callable(callable_, data))
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/Mako-1.0.0-py2.7.egg/mako/runtime.py",
> line 835, in _render_context
>     _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/Mako-1.0.0-py2.7.egg/mako/runtime.py",
> line 860, in _exec_template
>     callable_(context, *args, **kwargs)
>   File "_base_root_html", line 203, in render_body
>
>   File "_base_base_html", line 41, in render_body
>
>   File "changelog_changelog_html", line 218, in render_main
>
>   File "/home/mk/kallithea-tg/kallithea/lib/helpers.py", line 1020, in pager
>     result = re.sub(r'~(\d+)~', self._range, format)
>   File "/usr/lib64/python2.7/re.py", line 155, in sub
>     return _compile(pattern, flags).sub(repl, string, count)
>   File "/home/mk/kallithea-tg/kallithea/lib/helpers.py", line 977, in _range
>     nav_items.append(self._pagerlink(thispage, text_))
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/WebHelpers-1.3-py2.7.egg/webhelpers/paginate.py",
> line 841, in _pagerlink
>     link_url = url_generator(**link_params)
>   File
> "/home/mk/kallithea-venv/lib/python2.7/site-packages/Routes-1.13-py2.7.egg/routes/util.py",
> line 269, in url_for
>     (args, kargs))
> GenerationException: url_for could not generate URL. Called with args: ()
> {'page': 2}
>
> One reason I get a nice traceback might be that I use pip for installing as
> much as possible and thus don't get the annoying eggs but readable python.

Can you clarify in more detail how you set up your virtualenv?
I used 'pip install -e .' now, when I look in the virtualenv I do see
.py files for e.g. routes, webhelpers, ... Still I am not seeing such
a traceback. That missing piece of information is crucial in analyzing
this and other problems, so I really want to understand what is
different between your setup and mine.

>
> It seems like the problem is that this helpers.py pager function ends up in
> webhelpers/paginate.py which has some pylons specific code and uses
> url_generator = pylons.url.current when using Pylons.
>
> It can apparently be worked around with
>
> --- a/kallithea/lib/helpers.py
> +++ b/kallithea/lib/helpers.py
> @@ -911,6 +911,10 @@ class Page(_Page):
>      Custom pager to match rendering style with YUI paginator
>      """
>
> +    def __init__(self, *args, **kwargs):
> +        kwargs.setdefault('url', url.current)
> +        _Page.__init__(self, *args, **kwargs)
> +
>      def _get_pos(self, cur_page, max_page, items):
>          edge = (items / 2) + 1
>          if (cur_page <= edge):
> @@ -1050,11 +1054,12 @@ class Page(_Page):
>  class RepoPage(Page):
>
>      def __init__(self, collection, page=1, items_per_page=20,
> -                 item_count=None, url=None, **kwargs):
> +                 item_count=None, url=url.current, **kwargs):
>
>          """Create a "RepoPage" instance. special pager for paging
>          repository
>          """
> +        # FIXME: Why not calling baseclass __init__?
>          self._url_generator = url
>
>          # Safe the kwargs class-wide so they can be used in the pager()
> method
>
> This could probably just be upstreamed now, already while still using
> Pylons.

Thanks for this patch, I confirm it fixes this problem.
About the FIXME, I do not know why. Have you tried calling the
baseclass __init__?

>
> Closing comment: This made me look at the Page code. Now I feel sad. But the
> TG stuff generally seems to work. That makes me happy.

:-)

Thanks a lot for looking at this already,

Thomas