horizontal scaling using uwsgi and celery

[Mads Kiilerich]

On 11/10/2016 11:46 AM, Jan Heylen wrote:
 > Hi,
 >
 > I've got a horizontal scaling question:
 >
 > The Kallithea docs state: "If celery is used each instance should run 
a separate Celery instance, but the message broker should be common to 
all of them"

That is on the stable branch.

We added 
https://kallithea-scm.org/repos/kallithea/changeset/d6942b2b421c#docsusageperformancerst_o43 
on the default branch. I think it applies to stable too. It was old 
confusing wording.

It should perhaps be clarified even more: It should be perfectly fine 
that multiple web server instances share the same celery worker (in 
addition to using the same queue).

(There could perhaps be concerns about single point of failure that 
could cause a desire for multiple celery workers and multiple queues. I 
doubt that will have any benefit unless you have special needs and 
*really* have everything under control.)

 > Is this true for uwsgi and how could I accomplish this with one 
shared kallithea.ini config file? (is it possible to have uwsgi start a 
separate celercy e.g.?)

All wsgi instances should (in my opinion) send to the same queue and 
should thus use the same .ini settings.

The celery worker has to be launched and managed somehow. It is possible 
that uwsgi can do that is good at it - I don't know. But the point with 
celery is that it is 100% decoupled from the web serving and only 
communicate through the queue (and the database and the file system). It 
is thus not obvious that it should be managed through uwsgi.

 > I've already set instance_id to '*' as @kiilerix indicated in some 
old bitbucket issue thread:

That is only relevant for the repository cache and expiration - not 
related to celery.

/Mads