Current caching

[Thomas De Schampheleire]

2018-02-04 14:25 GMT+01:00 Thomas De Schampheleire <patrickdepinguin at gmail.com>:
> 2018-02-04 13:28 GMT+01:00 Dominik Ruf <dominikruf at gmail.com>:
>>
>>
>> Thomas De Schampheleire <patrickdepinguin at gmail.com> schrieb am So., 4. Feb.
>> 2018 um 13:04 Uhr:
>>>
>>>
>>>
>>> On Feb 4, 2018 12:32, "Dominik Ruf" <dominikruf at gmail.com> wrote:
>>>
>>>
>>>
>>> On Sun, Feb 4, 2018, 12:03 Mads Kiilerich <mads at kiilerich.com> wrote:
>>>>
>>>> On 02/03/2018 10:12 PM, Dominik Ruf wrote:
>>>>
>>>>
>>>> Mads Kiilerich <mads at kiilerich.com> schrieb am Sa., 3. Feb. 2018 um 19:32
>>>> Uhr:
>>>>>
>>>>> On 02/01/2018 12:50 AM, Dominik Ruf wrote:
>>>>> > Hi all,
>>>>> >
>>>>> > I'm currently looking at the caching Kallithea does. And I'm a
>>>>> > bit...baffled.
>>>>>
>>>>> Yes, it is quite baffling and not very efficient.
>>>>>
>>>>> I think it very rarely gives any real benefit - it might even make
>>>>> things slower. In real world setups with multiple repositories served by
>>>>> each worker process, cache hits are quite rare.
>>>>
>>>> For large git repositories we certainly do need caching.
>>>>
>>>>
>>>> Perhaps. But it is probably a very different kind of caching we need.
>>>
>>> Displaying a changelog page with 100 changesets currently takes 54 seconds
>>> even with the current caching for the Linux Kernel.
>>> I'm about to make some changes that will get this down to about 2.5
>>> seconds. But of course this also depends on caching.
>>>
>>>
>>> This is with a hot cache, ie not just invalidated?
>>
>> Yes with 'hot cache'. Without cache it is about 70 seconds.
>>>
>>>
>>> Do you see the same for a similar mercurial repository, or is it specific
>>> to git? What makes the difference?
>>
>> I don't know any hg repositories with 900000+ revisions.
>
> Our own review repo is 350000+ revisions.
> The public mozilla-central is 400000+ revisions. Perhaps this one
> could be useful for you to compare with hg performance? If need be you
> could strip the kernel repo to the same amount of revisions to get
> meaningful comparisons.
> https://hg.mozilla.org/mozilla-central/
>
>>>
>>>
>>> Is this on an SSD disk or spinning disk?
>>
>> SSD
>>>
>>>
>>> In our old installation we had performance problems with a large
>>> (mercurial) repo with many heads (review repo where nothing ever gets
>>> merged), particularly after invalidating of the cache. After upgrading
>>> Kallithea we were able to bump mercurial (which was supposed to have
>>> improvements to dealing with many heads), and additionally convert our repos
>>> to the generaldelta storage format. Now performance is relatively fine.
>>> Loading a changelog page definitely does not take 54 seconds, with both cold
>>> and hot cache.
>>
>> Here is what I did so far.
>> https://kallithea.dominikruf.com/kallithea/kallithea-domruf/changeset/89934e26b1fe4e4c5ef36416e1afbb22299233e7
>>
>
> Is this working without other changes? I see you added max_revisions
> as argument to get_changesets, but that method does not accept that
> parameter in latest tip.

Ok, I see that now in your other PR...