Optimizing Kallithea for heavy traffic

[Jean-Francois Beaumont]

I've been running stress tests on my install and so far I've not been able
to make Kallithea responsive enough for the traffic I have (mostly cloning).

After doing some preliminary setup, I've configured postgresql and added
the 1.7k repositories to it. The browsing is not that bad given the 80GB
our repositories weight. However, this server is under heavy cloning
activities as some repositories are used for fetching configurations
(small) and some other for knowledge resources (big). CI servers are making
the life of this server difficult as when we hit midnight, they all start
testing jobs and must fetch a lot of data from the HG server.

The current server uses a simple hgweb and it works well. All queries are
successfully processed. However with Kallithea, I got all sort of proxy
errors. I've tried to increase the timeout a bit but it is clearly not
scaling. Since both services are installed on the same machine and I ran my
tests when hgweb was quiet, I was expecting to have the same kind of
performances as with hgweb but that's not what's happening.

So I've been looking around and it seems people are talking about Celery
but I'm not sure I need to pursue this. Moreover, I was wondering if it
would help if Kallithea would be running a WSGI script inside Apache's
configuration instead of 'paster serve' with processes/threads.

Thanks!

JF
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sfconservancy.org/pipermail/kallithea-general/attachments/20150420/da3c28cf/attachment.html>