Current caching

[Thomas De Schampheleire]

2018-02-04 13:28 GMT+01:00 Dominik Ruf <dominikruf at gmail.com>:
>
>
> Thomas De Schampheleire <patrickdepinguin at gmail.com> schrieb am So., 4. Feb.
> 2018 um 13:04 Uhr:
>>
>>
>>
>> On Feb 4, 2018 12:32, "Dominik Ruf" <dominikruf at gmail.com> wrote:
>>
>>
>>
>> On Sun, Feb 4, 2018, 12:03 Mads Kiilerich <mads at kiilerich.com> wrote:
>>>
>>> On 02/03/2018 10:12 PM, Dominik Ruf wrote:
>>>
>>>
>>> Mads Kiilerich <mads at kiilerich.com> schrieb am Sa., 3. Feb. 2018 um 19:32
>>> Uhr:
>>>>
>>>> On 02/01/2018 12:50 AM, Dominik Ruf wrote:
>>>> > Hi all,
>>>> >
>>>> > I'm currently looking at the caching Kallithea does. And I'm a
>>>> > bit...baffled.
>>>>
>>>> Yes, it is quite baffling and not very efficient.
>>>>
>>>> I think it very rarely gives any real benefit - it might even make
>>>> things slower. In real world setups with multiple repositories served by
>>>> each worker process, cache hits are quite rare.
>>>
>>> For large git repositories we certainly do need caching.
>>>
>>>
>>> Perhaps. But it is probably a very different kind of caching we need.
>>
>> Displaying a changelog page with 100 changesets currently takes 54 seconds
>> even with the current caching for the Linux Kernel.
>> I'm about to make some changes that will get this down to about 2.5
>> seconds. But of course this also depends on caching.
>>
>>
>> This is with a hot cache, ie not just invalidated?
>
> Yes with 'hot cache'. Without cache it is about 70 seconds.
>>
>>
>> Do you see the same for a similar mercurial repository, or is it specific
>> to git? What makes the difference?
>
> I don't know any hg repositories with 900000+ revisions.

Our own review repo is 350000+ revisions.
The public mozilla-central is 400000+ revisions. Perhaps this one
could be useful for you to compare with hg performance? If need be you
could strip the kernel repo to the same amount of revisions to get
meaningful comparisons.
https://hg.mozilla.org/mozilla-central/

>>
>>
>> Is this on an SSD disk or spinning disk?
>
> SSD
>>
>>
>> In our old installation we had performance problems with a large
>> (mercurial) repo with many heads (review repo where nothing ever gets
>> merged), particularly after invalidating of the cache. After upgrading
>> Kallithea we were able to bump mercurial (which was supposed to have
>> improvements to dealing with many heads), and additionally convert our repos
>> to the generaldelta storage format. Now performance is relatively fine.
>> Loading a changelog page definitely does not take 54 seconds, with both cold
>> and hot cache.
>
> Here is what I did so far.
> https://kallithea.dominikruf.com/kallithea/kallithea-domruf/changeset/89934e26b1fe4e4c5ef36416e1afbb22299233e7
>

Is this working without other changes? I see you added max_revisions
as argument to get_changesets, but that method does not accept that
parameter in latest tip.