Kallithea Performance Tuning

[Adi Kriegisch]

Hey!

> >I'm running a Kallithea instance with about 100 repositories and
> >healthy amount of CI activity across several branches on each
> >repository.  I'm looking for some tips on tuning performance.  I'm
> >running using waitress (at least I think I am) and see what look
> >like several knobs in the production.ini file that can be turned,
> >but I'm not really sure what I'm looking at.
[...]
> I am a relatively happy user of apache + mod_wsgi. It scales nicely
> with multiple single-threaded worker processes but might require
> more memory. (I might however investigate uwsgi.)
We're using uwsgi. Performance and management is great; there are --
however -- some caveates:
* you'll need uwsgi >=1.9 (or your git pushes will fail due to some methods
  not implemented in earlier versions[1])
* you should do 'lazy' initialization; in case you're using a remote
  database server you need 'lazy' initialization or you'll get SSL errors
  when connecting to the database server:
  'lazy' will first fork the required number of processes and then start
  them. If you don't do that, the initialization of kallithea (including
  establishing a database connection with SSL context) will take place and
  then get forked. The error message you'll get is:
  'sqlalchemy.exc.OperationalError: (OperationalError) SSL error:
                                    decryption failed or bad record mac'
* to allow larger mercurial commits (those may contain large to huge headers),
  increase the buffer size default.
* letting kallithea run as its own user is a good idea and can easily be
  done in uwsgi by setting uid and gid; do not forget to allow the web
  server user (www-data in most cases) to connect to the socket.
* do not forget to 'calculate' your resource requirements
  (reload-on-rss * processes); also leave some RAM for celery/rabbitmq, the
  web server and the linux file system cache.

The config itself is pretty simple --
/etc/uwsgi/apps-enabled/kallithea.yaml:
  | uwsgi:
  |   plugins: python
  |   chdir: /venvs/kallithea
  |   virtualenv: /venvs/kallithea
  |   pythonpath: = /venvs/kallithea
  |   mount: /=/venvs/kallithea/dispatch.wsgi
  |   uid: kallithea
  |   gid: repos
  |   umask: 007
  |   memory-report: 1
  |   processes: 4
  |   reload-on-rss: 256
  |   lazy: true
  |   post-buffering: 4096
  |   # set the buffer for headers
  |   buffer-size: 16384
  |   chown-socket: www-data
  |   socket-timeout: 3600

I personally prefer nginx as a frontend; that may be configured like this:
/etc/nginx/sites-enabled/kallithea:
  | upstream kallithea {
  |     server unix:///run/uwsgi/app/kallithea/socket;
  | }
  | server {
  | (...)
  |     # maximum transfer size
  |     client_max_body_size 256m;
  | 
  |     # deliver static file from nginx
  |     root /srv/virtualenv/kallithea/public;
  | 
  |     location / {
  | 
  |         try_files $uri @uwsgi;
  | 
  |     }
  |     location @uwsgi {
  |         uwsgi_pass      kallithea;
  |         uwsgi_param     HTTPS on;
  |         uwsgi_param     SCRIPT_NAME "";
  |         include         uwsgi_params;
  |         uwsgi_read_timeout  3600;
  |     }
  | }
but apache may also be able to connect to uwsgi either via socket
(mod_uwsgi) or via http.

Some random notes:
* nginx currently has an issue with post buffering[2]: a post will be kept
  in memory until it is complete; then it gets forwarded. This will be
  fixed in 1.7/1.8 tree at some point; for nginx 1.2 and 1.4 patches exist.
* keep in mind that Mercurial is based on python 2.7 that only has inferior
  SSL support and is unable to do SNI[3]; your kallithea instance needs to be
  the default virtual host for SSL. Finally, after 6 years of discussion,
  python 2.7.9 might fix some of the more dangerous issues[4].
* get some guide for securing your SSL config like bettercrypto[5] or
  ssllabs[6].
* for Debian, backporting uwsgi from unstable (2.0.7) is pretty simple.
* running Apache is no option to me, because the current stable version of
  Apache in Debian (2.2) does not support DH params > 1024bit (and that is
  no reasonable key size). Debian maintainers backported EC-Support (the
  NIST curves), so you may choose for yourself. Nginx from backports
  has probably the best crypto support one could get out of OpenSSL atm.

-- Adi

[1] "IOError: seeking wsgi.input without post_buffering is IMPOSSIBLE !!!"
    The changelog of v1.9 list a generic implementation of seek, read and
    readlines: https://github.com/unbit/uwsgi-docs/blob/master/Changelog-1.9.rst
[2] http://trac.nginx.org/nginx/ticket/251
[3] http://bugs.python.org/issue5639
[4] https://www.python.org/dev/peps/pep-0466/
[5] https://bettercrypto.org
[6] http://ssllabs.com