Bug: ssl error with postgres and celery

[Valentin Kleibel]

Thanks for your work.
Unfortunately i can not test this change in a timely manner as it 
depends on too many changes compared to stable 6.2 which we run.

/Valentin

On 09/11/2020 17.56, Mads Kiilerich wrote:
> On 11/5/20 1:11 AM, Mads Kiilerich wrote:
>> On 11/3/20 5:08 PM, Valentin Kleibel wrote:
>>> Hi,
>>>
>>> We are using kallithea with a postgres database on another server 
>>> connected via ssl.
>>> It seems that celery in version 4 uses prefork in its worker pool 
>>> model now, which leads to issues with such a database connection: 
>>
>> I think you are right something is wrong.
>>
>> First, the celery-run will not only take a config file as parameter 
>> and read it. It will also initialize the WSGI app and database 
>> connection, before calling into celery. Effectively making it 
>> pre-fork. I think you are right that we only should initialize things 
>> after celery has forked and started worker processes.
> 
> 
> The first changes on 
> https://kallithea-scm.org/repos/kallithea/pull-request/292/_/celery_and_stuff 
> do this. After some refactorings (that may or may not be essential for 
> the change), I think the proper fix is in 
> https://kallithea-scm.org/repos/kallithea-incoming/changeset/e32c37b84c53bc6342e7b4698cb06168a7bcff15 
> .
> 
> I don't think we should have to dispose the engine. We just shouldn't 
> initialize it pre-fork.
> 
> Does this work for you?
> 
> Disposing the engine for each task seems inefficient and shouldn't be 
> necessary. As an alternative to your patch for disposing in dbsession, 
> it could perhaps just dispose in 
> kallithea.CELERY_APP.loader.on_worker_process_init .
> 
> 
>> But also, it might be a bit odd that we have the celery-run entry 
>> point. It would perhaps be better to somehow use "celery" as entry 
>> point and let it call into Kallithea in each worker. But I haven't 
>> investigated how feasible that would be.
> 
> 
> This is still TODO.
> 
> There is still a lot of room for improvement, making sure that we use 
> Celery in a simple and recommended way, avoiding the need for 
> initializing Celery at *import* time, and upgrading to Celery 5. 
> Contributions in this area would be welcome ;-)
> 
> /Mads
> 
>