Bug: ssl error with postgres and celery

[Mads Kiilerich]

On 11/5/20 1:11 AM, Mads Kiilerich wrote:
> On 11/3/20 5:08 PM, Valentin Kleibel wrote:
>> Hi,
>>
>> We are using kallithea with a postgres database on another server 
>> connected via ssl.
>> It seems that celery in version 4 uses prefork in its worker pool 
>> model now, which leads to issues with such a database connection: 
>
> I think you are right something is wrong.
>
> First, the celery-run will not only take a config file as parameter 
> and read it. It will also initialize the WSGI app and database 
> connection, before calling into celery. Effectively making it 
> pre-fork. I think you are right that we only should initialize things 
> after celery has forked and started worker processes.


The first changes on 
https://kallithea-scm.org/repos/kallithea/pull-request/292/_/celery_and_stuff 
do this. After some refactorings (that may or may not be essential for 
the change), I think the proper fix is in 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/e32c37b84c53bc6342e7b4698cb06168a7bcff15 
.

I don't think we should have to dispose the engine. We just shouldn't 
initialize it pre-fork.

Does this work for you?

Disposing the engine for each task seems inefficient and shouldn't be 
necessary. As an alternative to your patch for disposing in dbsession, 
it could perhaps just dispose in 
kallithea.CELERY_APP.loader.on_worker_process_init .


> But also, it might be a bit odd that we have the celery-run entry 
> point. It would perhaps be better to somehow use "celery" as entry 
> point and let it call into Kallithea in each worker. But I haven't 
> investigated how feasible that would be.


This is still TODO.

There is still a lot of room for improvement, making sure that we use 
Celery in a simple and recommended way, avoiding the need for 
initializing Celery at *import* time, and upgrading to Celery 5. 
Contributions in this area would be welcome ;-)

/Mads