<img class="cloudmagic-smart-beacon"
src="https://tr.cloudmagic.com/h/v6/emailtag/tag/1490174573/23208ea609a3d53270dd5e8ba08bc371/404e184f97b08329e4f2296649e79a26/8afc15dff44fea5343fe21f3e5bdec5d/199d7891ac61235c9301ad4dcff1fe59/newton.gif"
style="border:0; width:10px; height:10px;" width="10" height="10"
align="right"><div dir="auto"><div><span>I think a product like
appenlight.com might come in handy. We use it together with RhodeCode to
monitor instance performance, and it works great. It gives you a
performance breakdown on each of application layer, db,
rendering-templates, etc. It also can notify you about slow requests,
exception or log.errors as they
occur.</span></div><div><span><br></span></div><div><span>It works
with any python application and works really good specially with
pylons and pyramid. So i believe it would work fine with Kallithea. Here
are instructions on how to connect it to pylons app: </span><a
href="https://getappenlight.com/page/python/pylons-exception-logging.html">https://getappenlight.com/page/python/pylons-exception-logging.html</a></div><div><span><br></span></div><div><span>You
can either connect for free to the SaaS version (appenlight.com) or
download VM image including the open-source free version to host it on
your
premises.</span></div><div><span><br></span></div><div><span>Best,</span></div><br><div
id="cm_footer" class="cm_footer"><div id="cm_signature"><div
style="font-family: arial, sans-serif; margin: 0px; padding: 0px; border:
0px; font-size: 10pt; line-height: inherit; vertical-align: baseline;
color: rgb(34, 34, 34);"><font color="#929292" style="font-family: inherit;
font-size: 11px;">--</font></div><div style="font-family: arial,
sans-serif; margin: 0px; padding: 0px; border: 0px; font-size: 10pt;
line-height: inherit; vertical-align: baseline; color: rgb(34, 34,
34);"><span style="font-family: Helvetica; margin: 0px; padding: 0px;
border: 0px; font-size: 11px; font-style: inherit; font-variant-caps:
inherit; line-height: inherit; vertical-align: baseline; color: rgb(146,
146, 146);">Marcin Kuzminski</span></div><div style="font-family: arial,
sans-serif; margin: 0px; padding: 0px; border: 0px; font-size: 10pt;
line-height: inherit; vertical-align: baseline; color: rgb(34, 34,
34);"><div style="font-family: inherit; margin: 0px; padding: 0px; border:
0px; font-size: 12.8px; font-style: inherit; font-variant-caps: inherit;
line-height: inherit; vertical-align: baseline;"><span style="font-family:
Helvetica; margin: 0px; padding: 0px; border: 0px; font-size: 11px;
font-style: inherit; font-variant-caps: inherit; line-height: inherit;
vertical-align: baseline; color: rgb(146, 146, 146);">RhodeCode,
Inc.</span></div></div></div></div><div id="cm_replymail_content_wrap"><div
class="cm_replymail_content_1490174212_wrapper">On Wed, Mar 22, 2017 at
8:01, Jan Heylen <heyleke@gmail.com> wrote:<br><div
id="cm_replymail_content_1490174212" style="overflow: visible;"><blockquote
style="margin:0;border-left: #D6D6D6 1px solid;padding-left: 10px;"><div
dir="ltr">Hi,<br><div class="gmail_extra"><br><div class="gmail_quote">On
Tue, Mar 21, 2017 at 11:11 AM, Adi Kriegisch <span dir="ltr"><<a
href="mailto:adi@cg.tuwien.ac.at">adi@cg.tuwien.ac.at</a>></span>
wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi!<br>
<span class="gmail-"><br>
> Sometimes the performance of our Kallithea instance takes a hit, such
that<br>
> page loading can take many many seconds, and it is unclear what causes
this.<br>
><br>
> Could you suggest ways to investigate this?<br>
</span>Step 1: get data. :-)<br>
<br>
Actually one of the reasons I use uwsgi for deployment is that I get
execution<br>
times 'for free': 'generated 309 bytes in 268 msecs' or 'generated 436
bytes<br>
in 56 msecs' including the uri and all kinds of useful information.<br>
<br></blockquote><div>That kind of monitoring is indeed what we do now,
with uwsgi:</div><div><div>{address space usage: 1769336832 bytes/1687MB}
{rss usage: 1298124800 bytes/1237MB} [pid: 52669|app: 0|req: 349/9564]
1</div><div>35.252.28.229 () {40 vars in 1686 bytes} [Wed Mar 22 07:39:38
2017] GET /review/ext/cvpsw-review/pull-request/15006/_/c</div><div>vp80
=> generated 76111 bytes in 924 msecs (HTTP/1.1 200) 5 headers in 149
bytes (1 switches on core 0)</div><div>{address space usage: 1687105536
bytes/1608MB} {rss usage: 1218387968 bytes/1161MB} [pid: 49555|app: 0|req:
209/9565] </div><div>35.252.28.229 () {42 vars in 1591 bytes} [Wed Mar 22
07:39:40 2017] GET /js/graph.js?ver=0.2.1 </div><div>=> generated 0
bytes in 156 msecs (HTTP/1.1 304) 3 headers in 98 bytes (1 switches on core
0) </div><div>2017-03-22 07:40:09.168 INFO [kallithea.RequestWrapper] IP:
135.224.206.40 Request to /review/ms/sw-review/branches-tags time: 66.745s
</div></div><div><br></div><div>But it are these kind of peeks we want to
understand on what Kallithea was exactly waiting upon, is it pure disk/io,
is it a qeury to the database that took long (postgresql with monitoring
with pg_activity)</div><div>Is there anything Kallithea debug info or
pythong profiler can provide us (without overloading kallithea itself with
all the debug) that we today are not aware of?</div><div><br></div><div>Our
database is 'only' 270MB big, so besides inefficient queries, I cannot
understand if it would be database access that is causing hickups of more
then a minute. Either we configured our database wrong, or I would think
postgresql can keep 270MB in memory (we might need to force it to keep it
in memory, even if other processes consume alot of memory), the server has
125GB of memory, with a 'free' count (even with caches) of 20GB (let me be
clear, the sever is used for other applications as
well).</div><div><br></div><div>What I do see as queries running long on
the database are these, (But I haven't correlated these directly with the
above peeks yet (but it could well be).)</div><div><div>47362 kallithea
kallithea 127.0.0.1 0.0 0.0 0.00B 0.00B 00:47.10
N N SELECT cache_invalidation.cache_id AS cache_invalidation_cache_id,
cache_invalidation.cache_key AS cache_invalidation_cache_key,
cache_invalidation.cache_args AS cache_invalidation_cache_args,
cache_invalidation.cache_active AS cache_invalidation_cache_active FROM
cache_invalidation WHERE cache_invalidation.cache_key =
'devws048-32241review/ms/sw-review'</div></div><div><br></div><div>As you
see, that query runs for 47 seconds at the moment I catch
it.</div><div><br></div><div> Anybody who can explain what this 'triggers'
in the database, as that table itself is quite small.</div><div><div><font
face="monospace, monospace"> public | cache_invalidation
| table | kallithea | 992 kB |
</font></div><div><font face="monospace, monospace"> public |
cache_invalidation_cache_id_seq | sequence
| kallithea | 8192 bytes | </font></div></div><div><br></div><div>I haven'
looked googled yet what this could mean, so forgive me if I'm asking basic
questions ;-)</div><div><br></div><div><br></div><blockquote
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
My setup is similar to this:<br>
<a
href="https://lists.sfconservancy.org/pipermail/kallithea-general/2015q1/000130.html"
rel="noreferrer">https://lists.sfconservancy.org/pipermail/kallithea-general/2015q1/000130.html</a><br>
<br>
You may even monitor execution times or graph them or do some statistics
work<br>
on them or -- simply put -- correlate higher latency and execution time
with<br>
system or I/O load (even simple tools like sysstat/sadc/sar are sufficient
to<br>
gain insights in this regard).<br></blockquote><div><br></div><div>I will
already have a look at these simple ones, I must admit I didn't use these
until now (I did use iowait/top/...)</div><div><a
href="http://www.thegeekstuff.com/2011/03/sar-examples/?utm_source=feedburner">http://www.thegeekstuff.com/2011/03/sar-examples/?utm_source=feedburner</a><br></div><div><br></div><div>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Hope, this helps...<br>
<br>
All the best,<br>
Adi<br>
_______________________________________________<br>
kallithea-general mailing list<br>
<a
href="mailto:kallithea-general@sfconservancy.org">kallithea-general@sfconservancy.org</a><br>
<a
href="https://lists.sfconservancy.org/mailman/listinfo/kallithea-general"
rel="noreferrer">https://lists.sfconservancy.org/mailman/listinfo/kallithea-general</a><br>
</blockquote></div><br></div><div class="gmail_extra">Thx!</div><div
class="gmail_extra"><br></div><div class="gmail_extra">Jan</div></div>
</blockquote></div></div></div></div>