[PATCH 1 of 6] setup.py: allow Paste 3.0.x

[Thomas De Schampheleire]

El sáb., 29 dic. 2018 a las 18:35, Mads Kiilerich
(<mads at kiilerich.com>) escribió:
>
> On 12/29/18 3:51 PM, Thomas De Schampheleire wrote:
> >> (Also, "celery" is just an implementation detail. And a quite complex
> >> and fragile one - especially if using rabbitmq. But at our scale, it
> >> would probably be better to just use a sql database. And most
> >> immportant: Our commands and documentation should perhaps just talk
> >> about "worker process".)
> > Can you elaborate on the following, please?
> >
> > - "especially if using rabbitmq" -- in which way is rabbitmq a bigger
> > problem than other brokers?
>
>
> It is my experience that rabbitmq can be tricky to configure. It is a
> separate and very different server process and different environment
> that can be hard to manage. No doubt it is great for what it is, but it
> might also be overkill.

Ok. In our case we have started it with the default settings and it
seems to do enough for Kallithea usage.

>
>
> > - what do you mean with 'use a sql database'? To replace what,
> > rabbitmq? And how exactly? If you meant to replace celery, then I'm
> > totally lost.
> >
> > Celery is a 'distributed task queue', while in Kallithea we probably
> > just need something to asynchronously handle tasks. Unless Kallithea
> > would be deployed at Github scale, maybe?
> > Do you suggest a different approach?
>
>
> Since all tasks operate on the database anyway, there is no point in
> making these async tasks scalable beyond what the database can handle.
> Celery *can* apparently use a plain database as backend ... with the
> caveat that it needs inefficient polling. That could perhaps be an
> alternative to rabbitmq.

Are you referring to this:
http://docs.celeryproject.org/en/3.1/getting-started/brokers/sqlalchemy.html
This page mentions:
"Experimental Status
The SQLAlchemy transport is unstable in many areas and there are
several issues open. Unfortunately we don’t have the resources or
funds required to improve the situation, so we’re looking for
contributors and partners willing to help."

In Celery 4 it is no longer listed:
http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html

>
> (But also, in our usecase we try to abstract and embed celery. I'm not
> sure how much value the celery features add. It would perhaps be easy
> and simple to just implement the few things we need: a database table
> with pending commands, and a long-running cli command that pop commands
> and execute them. But now we have a working celery intergration, so as
> long as it is low maintenance, it is probably better to keep it.
> Depending on how tricky it is to upgrade.)

Ok, we'll see later.

>
>
> >
> >> Yes. If we assess that an area has good test coverage, then a safe
> >> upgrade is easy.
> > The problem is to make a statement about test coverage about external
> > components. What is our coverage of 'Paste' ?
>
>
> The Paste coverage is certainly better than our coverage of celery ;-)
>
> Perhaps take a look at where we use paste. Try to hack the integration
> points and introduce some errors in the source and see if they are hit
> when running the test suite. Points without coverage need some manual
> testing. And it is also nice to do a bit of manual "does it really work"
> testing for areas with some test coverage. Especially if it is close to
> the front-end where there are different "right" ways to do it ... or
> where something might be "wrong" in unknown ways.

I looked what exactly we are using from Paste, at least directly (not
indirectly through other packages).
And then I identified some uses that could be removed, I'll send
patches shortly.
What is left is:

kallithea/bin/kallithea_cli_iis.py:    from paste.deploy import loadapp
kallithea/bin/kallithea_cli_base.py:import paste.deploy
kallithea/bin/kallithea_cli_base.py:                kallithea.CONFIG =
paste.deploy.appconfig('config:' + path_to_ini_file)
kallithea/lib/base.py:import paste.httpexceptions
kallithea/lib/base.py:import paste.auth.basic
kallithea/lib/base.py:import paste.httpheaders
kallithea/lib/base.py:class BasicAuth(paste.auth.basic.AuthBasicAuthenticator):
kallithea/lib/base.py:        head =
paste.httpheaders.WWW_AUTHENTICATE.tuples('Basic realm="%s"' %
self.realm)
kallithea/lib/base.py:            return
paste.httpexceptions.HTTPForbidden(headers=head)
kallithea/lib/base.py:        return
paste.httpexceptions.HTTPUnauthorized(headers=head)
kallithea/lib/base.py:        authorization =
paste.httpheaders.AUTHORIZATION(environ)
kallithea/lib/base.py:
paste.httpheaders.AUTH_TYPE.update(environ, 'basic')
kallithea/lib/base.py:
paste.httpheaders.REMOTE_USER.update(environ, result)
kallithea/lib/hooks.py:    from paste.deploy import appconfig
kallithea/tests/conftest.py:from paste.deploy import loadwsgi
kallithea/lib/middleware/sessionmiddleware.py:        if
environ.get('paste.registry'):
kallithea/lib/middleware/sessionmiddleware.py:            if
environ['paste.registry'].reglist:
kallithea/lib/middleware/sessionmiddleware.py:
environ['paste.registry'].register(self.session, session)
kallithea/lib/middleware/sessionmiddleware.py:        if
'paste.testing_variables' in environ and 'webtest_varname' in options:
kallithea/lib/middleware/sessionmiddleware.py:
environ['paste.testing_variables'][options['webtest_varname']] =
session
kallithea/tests/scripts/manual_test_concurrency.py:from paste.deploy
import appconfig

So mostly paste.deploy and some paste.httpheaders,
paste.httpexceptions, paste.auth.

Via pytest-cov on Kallithea itself, I could see that the stuff in
kallithea/lib/base.py is mostly covered by the tests, not all but
looks good to me.

/Thomas