Investigating Kallithea performance issues

[Mads Kiilerich]

On 03/20/2017 08:08 PM, Thomas De Schampheleire wrote:
> Hi,
>
> Sometimes the performance of our Kallithea instance takes a hit, such 
> that page loading can take many many seconds, and it is unclear what 
> causes this.
>
> Could you suggest ways to investigate this?

I guess the only really good (and not good at all) suggestion is: Keep 
looking until you figure it out. It can be anything ... and probably not 
what it was last time.

I have often found it helpful to temporarily add extra log statements to 
narrow in on problems, and make sure simultaneous requests can be kept 
separate and correlated with their requests. That often shows some pattern.

Some of the problems I found in my setup has been addressed in the 
default branch but not in the stable branch. There are no migration 
steps for adding indices etc, so you will have to import your old data 
into a new empty database with the latest schema.

> Things that come to mind:
> - how to find database performance problems, how to determine if new 
> indexes are needed?

When using PostreSQL (which I highly recommend for production use), 
configure it to log all slow queries. For slow queries, put on your DBA 
hat and rerun them and get the query plan and see if anything pops up 
and suggests that an index is needed. Then try adding the index and see 
if it helps - if not remove it again. That can usually be done on live 
systems with minimal risk.

(Also, we found that when the database and the size of the result grows, 
it becomes more essential to run the database locally.)

> - how to detect cache (invalidation) issues? Which parameters to tweak?

I think the main thing there is that the default branch now is almost 
fully automatic, while the stable branch had some parameters to tweak.

If cache invalidation checks really shows up as a source of slowness, 
you must have a lot of entries in the database. They should be cleaned 
up. On the default branch you can do that with the paster command or 
from the UI ... or directly in the database.

Worst case is that the web ui might miss some new pushes - that can be 
fixed by restarting the Kallithea WSGIs.

> - how to detect I/O issues?

A good experienced sysadmin can perhaps help with that. There are many 
opinions and articles of that. But on linux, for example "iostat" (from 
"sysstat") or "iotop" (and plain "top") can be convenient to show what 
is going on right now.

/Mads