Bug: ssl error with postgres and celery

[Valentin Kleibel]

Hi,

On 05/11/2020 01.11, Mads Kiilerich wrote:
> On 11/3/20 5:08 PM, Valentin Kleibel wrote:
>> Hi,
>>
>> We are using kallithea with a postgres database on another server 
>> connected via ssl.
>> It seems that celery in version 4 uses prefork in its worker pool 
>> model now, which leads to issues with such a database connection: 
> 
> I think you are right something is wrong.
> 
> First, the celery-run will not only take a config file as parameter and 
> read it. It will also initialize the WSGI app and database connection, 
> before calling into celery. Effectively making it pre-fork. I think you 
> are right that we only should initialize things after celery has forked 
> and started worker processes.
> 
> But also, it might be a bit odd that we have the celery-run entry point. 
> It would perhaps be better to somehow use "celery" as entry point and 
> let it call into Kallithea in each worker. But I haven't investigated 
> how feasible that would be.

Thank you for the confirmation.
I'd be happy to help out and create a patch or do other work provided 
there is a clear understanding in which way we'd like to fix this.

> As a workaround, can you use celery.worker_concurrency = 1 but launch 
> celery-run several times?

This needs quite a bit of changes and I don't want to invest time in 
that. The patch I've sent works for us right now so we'll keep this 
workaround.
It might actually qualify as a proper fix as this is what the sqalchemy 
documentation recommends to do. [1]

Thanks,
Valentin

[1] 
https://docs.sqlalchemy.org/en/13/core/pooling.html#using-connection-pools-with-multiprocessing-or-os-fork