Timeout with git clone

[Mads Kiilerich]

Hi

The code in this area has worked surprisingly well since Kallithea 
inherited it, even though it has popped up regularly needing tricky 
maintenance. I agree it would be nice to refactor / reimplement this 
area. It is just that nobody invested time or sponsorship in doing it. I 
guess it hasn't caused enough pain for anybody to justify it ;-)

Yes, I agree that it probably would be much better to go back to use 
dulwich both for protocol serving and for providing data for the web 
frontend, instead of forking out to git. Disclaimer: I don't know has 
fast dulwich is these days. It could perhaps also be relevant to 
research what other python git hosting solutions do.

If interested in contributing in this area, a first step could be to 
create a proof of concept of switching back to Dulwich and doing some 
benchmarks - both for local cloning with infinite network bandwidth 
(where I doubt dulwich can match pure git) and for more realistic remote 
internet bandwidth (where I guess it doesn't matter much).

But also note that subprocessio no longer only is used by pygrack. It is 
also used for run_git_command in 
kallithea/lib/vcs/backends/git/repository.py (introduced in 
1f4d4b8d72f5), mainly for cloning and listing changesets. A full 
solution would require somehow replacing run_git_command with dulwich. 
But that can be done one at a time.

/Mads


On 18/04/2023 16:55, Quentin Wenger wrote:
> Digging a bit deeper:
>
> - The changeset that you linked (https://kallithea-scm.org/repos/kallithea/changeset/034e4fe1ebb2#rhodecodelibsubprocessiopy_n127) actually shows that historically it went the other way round, that is at first dulwich's server was used but then considered "buggy", therefore the implementation was replaced by some custom code.
>
> - That custom code looks like coming from https://github.com/dvdotsenko/git_http_backend.py. That repo hasn't been updated since 2012, neither do its forks show any sign of recent activity.
>
> - In contrast, dulwich, while officially still in beta, is actively developed.
>
> IMhO the proper move would be to go back to dulwich. Chances are that those buggy things have been fixed in the last ten years. And if they haven't, better report them upstream than reinvent the wheel. By the way, do we have any more precise idea of what was considered buggy at the time?
>
> What do you think?