[PATCH 1 of 6] setup.py: allow Paste 3.0.x

[Mads Kiilerich]

On 12/29/18 3:51 PM, Thomas De Schampheleire wrote:
>> (Also, "celery" is just an implementation detail. And a quite complex
>> and fragile one - especially if using rabbitmq. But at our scale, it
>> would probably be better to just use a sql database. And most
>> immportant: Our commands and documentation should perhaps just talk
>> about "worker process".)
> Can you elaborate on the following, please?
>
> - "especially if using rabbitmq" -- in which way is rabbitmq a bigger
> problem than other brokers?


It is my experience that rabbitmq can be tricky to configure. It is a 
separate and very different server process and different environment 
that can be hard to manage. No doubt it is great for what it is, but it 
might also be overkill.


> - what do you mean with 'use a sql database'? To replace what,
> rabbitmq? And how exactly? If you meant to replace celery, then I'm
> totally lost.
>
> Celery is a 'distributed task queue', while in Kallithea we probably
> just need something to asynchronously handle tasks. Unless Kallithea
> would be deployed at Github scale, maybe?
> Do you suggest a different approach?


Since all tasks operate on the database anyway, there is no point in 
making these async tasks scalable beyond what the database can handle. 
Celery *can* apparently use a plain database as backend ... with the 
caveat that it needs inefficient polling. That could perhaps be an 
alternative to rabbitmq.

(But also, in our usecase we try to abstract and embed celery. I'm not 
sure how much value the celery features add. It would perhaps be easy 
and simple to just implement the few things we need: a database table 
with pending commands, and a long-running cli command that pop commands 
and execute them. But now we have a working celery intergration, so as 
long as it is low maintenance, it is probably better to keep it. 
Depending on how tricky it is to upgrade.)


>
>> Yes. If we assess that an area has good test coverage, then a safe
>> upgrade is easy.
> The problem is to make a statement about test coverage about external
> components. What is our coverage of 'Paste' ?


The Paste coverage is certainly better than our coverage of celery ;-)

Perhaps take a look at where we use paste. Try to hack the integration 
points and introduce some errors in the source and see if they are hit 
when running the test suite. Points without coverage need some manual 
testing. And it is also nice to do a bit of manual "does it really work" 
testing for areas with some test coverage. Especially if it is close to 
the front-end where there are different "right" ways to do it ... or 
where something might be "wrong" in unknown ways.


/Mads