Bug: ssl error with postgres and celery

Valentin Kleibel valentin at vrvis.at
Tue Nov 3 16:08:21 UTC 2020


Hi,

We are using kallithea with a postgres database on another server 
connected via ssl.
It seems that celery in version 4 uses prefork in its worker pool model 
now, which leads to issues with such a database connection:

```
Task kallithea.lib.celerylib.whoosh_index[9c
4dbbcc-b9e9-48be-9455-0bbcd8d251a2] raised unexpected: 
OperationalError('(psycopg2.Operational
Error) SSL SYSCALL error: EOF detected\n')
Traceback (most recent call last):
   File 
"/mnt/webservices/virtualenv/kallithea-0.6.2/lib/python3.7/site-packages/sqlalchemy/eng
ine/base.py", line 1277, in _execute_context
     cursor, statement, parameters, context
   File 
"/mnt/webservices/virtualenv/kallithea-0.6.2/lib/python3.7/site-packages/sqlalchemy/eng
ine/default.py", line 593, in do_execute
     cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected

The above exception was the direct cause of the following exception:
(...)
```

We know that issues like this may arise from using prefork in uwsgi 
which we work around by setting lazy:true
The reason for this is that when a database connection with SSL (as we 
use) is already established with a valid SSL context before a process is 
forked, the connection will cease to work as such a connection is 
stateful on both sides and cannot cope well with a fork.

The only workaround to this we found is to setup celery to only use a 
single worker in production.ini:

```
celery.worker_concurrency = 1
celery.worker_max_tasks_per_child = 5
```

But this has the drawback that long running tasks like indexing will 
block short running tasks like the creation of a new repository. Thus, 
this setting is undesired and worker_concurrency should be at least 2 on 
a production system.

I tried to find an easy fix for this but it seems to need some changes 
to kallithea as long as celery and sqalchemy don't change to cope with 
this problem.

I hope someone can help with this.

Cheers,
Valentin


More information about the kallithea-general mailing list