horizontal scaling using uwsgi and celery

[Jan Heylen]

On Thu, Nov 10, 2016 at 1:19 PM, Mads Kiilerich <mads at kiilerich.com> wrote:

> On 11/10/2016 11:46 AM, Jan Heylen wrote:
> > Hi,
> >
> > I've got a horizontal scaling question:
> >
> > The Kallithea docs state: "If celery is used each instance should run a
> separate Celery instance, but the message broker should be common to all of
> them"
>
> That is on the stable branch.
>
> We added https://kallithea-scm.org/repos/kallithea/changeset/d6942b2b
> 421c#docsusageperformancerst_o43 on the default branch. I think it
> applies to stable too. It was old confusing wording.
>
Ok, thx, very useful to change this, although I do have the default branch
checked out, I didn't noticed this info. I was reading it on the
'readthedocs': (version 0.3.1, so stsble branch I assume)
http://kallithea.readthedocs.io/en/0.3.1/usage/performance.html


> It should perhaps be clarified even more: It should be perfectly fine that
> multiple web server instances share the same celery worker (in addition to
> using the same queue).
>
> (There could perhaps be concerns about single point of failure that could
> cause a desire for multiple celery workers and multiple queues. I doubt
> that will have any benefit unless you have special needs and *really* have
> everything under control.)
>
> > Is this true for uwsgi and how could I accomplish this with one shared
> kallithea.ini config file? (is it possible to have uwsgi start a separate
> celercy e.g.?)
>
> All wsgi instances should (in my opinion) send to the same queue and
> should thus use the same .ini settings.
>
> The celery worker has to be launched and managed somehow. It is possible
> that uwsgi can do that is good at it - I don't know. But the point with
> celery is that it is 100% decoupled from the web serving and only
> communicate through the queue (and the database and the file system). It is
> thus not obvious that it should be managed through uwsgi.

Ok, thx for the info, we will probably have paster still start celery, and
use uwsgi to scale Kallithea horizontally it's instances.


>
> > I've already set instance_id to '*' as @kiilerix indicated in some old
> bitbucket issue thread:
>
> That is only relevant for the repository cache and expiration - not
> related to celery.

indeed, just wanted to mention it as source of my information.


>
>
> /Mads
>

Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sfconservancy.org/pipermail/kallithea-general/attachments/20161110/3def149b/attachment.html>