Kallithea Performance Tuning

[Tim Downey]

Thanks Adi.  I'll try it out.

On Tue, Jan 27, 2015 at 4:25 AM, Adi Kriegisch <adi at cg.tuwien.ac.at> wrote:

> Hey!
>
> > >I'm running a Kallithea instance with about 100 repositories and
> > >healthy amount of CI activity across several branches on each
> > >repository.  I'm looking for some tips on tuning performance.  I'm
> > >running using waitress (at least I think I am) and see what look
> > >like several knobs in the production.ini file that can be turned,
> > >but I'm not really sure what I'm looking at.
> [...]
> > I am a relatively happy user of apache + mod_wsgi. It scales nicely
> > with multiple single-threaded worker processes but might require
> > more memory. (I might however investigate uwsgi.)
> We're using uwsgi. Performance and management is great; there are --
> however -- some caveates:
> * you'll need uwsgi >=1.9 (or your git pushes will fail due to some methods
>   not implemented in earlier versions[1])
> * you should do 'lazy' initialization; in case you're using a remote
>   database server you need 'lazy' initialization or you'll get SSL errors
>   when connecting to the database server:
>   'lazy' will first fork the required number of processes and then start
>   them. If you don't do that, the initialization of kallithea (including
>   establishing a database connection with SSL context) will take place and
>   then get forked. The error message you'll get is:
>   'sqlalchemy.exc.OperationalError: (OperationalError) SSL error:
>                                     decryption failed or bad record mac'
> * to allow larger mercurial commits (those may contain large to huge
> headers),
>   increase the buffer size default.
> * letting kallithea run as its own user is a good idea and can easily be
>   done in uwsgi by setting uid and gid; do not forget to allow the web
>   server user (www-data in most cases) to connect to the socket.
> * do not forget to 'calculate' your resource requirements
>   (reload-on-rss * processes); also leave some RAM for celery/rabbitmq, the
>   web server and the linux file system cache.
>
> The config itself is pretty simple --
> /etc/uwsgi/apps-enabled/kallithea.yaml:
>   | uwsgi:
>   |   plugins: python
>   |   chdir: /venvs/kallithea
>   |   virtualenv: /venvs/kallithea
>   |   pythonpath: = /venvs/kallithea
>   |   mount: /=/venvs/kallithea/dispatch.wsgi
>   |   uid: kallithea
>   |   gid: repos
>   |   umask: 007
>   |   memory-report: 1
>   |   processes: 4
>   |   reload-on-rss: 256
>   |   lazy: true
>   |   post-buffering: 4096
>   |   # set the buffer for headers
>   |   buffer-size: 16384
>   |   chown-socket: www-data
>   |   socket-timeout: 3600
>
> I personally prefer nginx as a frontend; that may be configured like this:
> /etc/nginx/sites-enabled/kallithea:
>   | upstream kallithea {
>   |     server unix:///run/uwsgi/app/kallithea/socket;
>   | }
>   | server {
>   | (...)
>   |     # maximum transfer size
>   |     client_max_body_size 256m;
>   |
>   |     # deliver static file from nginx
>   |     root /srv/virtualenv/kallithea/public;
>   |
>   |     location / {
>   |
>   |         try_files $uri @uwsgi;
>   |
>   |     }
>   |     location @uwsgi {
>   |         uwsgi_pass      kallithea;
>   |         uwsgi_param     HTTPS on;
>   |         uwsgi_param     SCRIPT_NAME "";
>   |         include         uwsgi_params;
>   |         uwsgi_read_timeout  3600;
>   |     }
>   | }
> but apache may also be able to connect to uwsgi either via socket
> (mod_uwsgi) or via http.
>
> Some random notes:
> * nginx currently has an issue with post buffering[2]: a post will be kept
>   in memory until it is complete; then it gets forwarded. This will be
>   fixed in 1.7/1.8 tree at some point; for nginx 1.2 and 1.4 patches exist.
> * keep in mind that Mercurial is based on python 2.7 that only has inferior
>   SSL support and is unable to do SNI[3]; your kallithea instance needs to
> be
>   the default virtual host for SSL. Finally, after 6 years of discussion,
>   python 2.7.9 might fix some of the more dangerous issues[4].
> * get some guide for securing your SSL config like bettercrypto[5] or
>   ssllabs[6].
> * for Debian, backporting uwsgi from unstable (2.0.7) is pretty simple.
> * running Apache is no option to me, because the current stable version of
>   Apache in Debian (2.2) does not support DH params > 1024bit (and that is
>   no reasonable key size). Debian maintainers backported EC-Support (the
>   NIST curves), so you may choose for yourself. Nginx from backports
>   has probably the best crypto support one could get out of OpenSSL atm.
>
> -- Adi
>
> [1] "IOError: seeking wsgi.input without post_buffering is IMPOSSIBLE !!!"
>     The changelog of v1.9 list a generic implementation of seek, read and
>     readlines:
> https://github.com/unbit/uwsgi-docs/blob/master/Changelog-1.9.rst
> [2] http://trac.nginx.org/nginx/ticket/251
> [3] http://bugs.python.org/issue5639
> [4] https://www.python.org/dev/peps/pep-0466/
> [5] https://bettercrypto.org
> [6] http://ssllabs.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sfconservancy.org/pipermail/kallithea-general/attachments/20150127/9b3a34dc/attachment.html>