From branko at majic.rs Tue Jul 23 11:17:30 2024 From: branko at majic.rs (Branko Majic) Date: Tue, 23 Jul 2024 13:17:30 +0200 Subject: Authorization required for cloning Kallithea with Mercurial >= 6.5.0 Message-ID: <20240723131730.4b616fc0@majic.rs> Hello, Attempting to clone the official Kallithea repository (at https://kallithea-scm.org/repos/kallithea) anonymously with newer versions of Mercurial seems to fail, and prompts user for credentials: ----%---- (hgtest) branko at che:~/projects/hgtest$ hg version Mercurial Distributed SCM (version 6.5.3) (see https://mercurial-scm.org for more information) Copyright (C) 2005-2023 Olivia Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Enabled extensions: histedit internal strip internal (hgtest) branko at che:~/projects/hgtest$ hg clone https://kallithea-scm.org/repos/kallithea destination directory: kallithea http authorization required for https://kallithea-scm.org/repos/kallithea realm: Kallithea authentication user: ----%---- Based on the realm part - I assume this could be something related to Kerberos? From what I can tell, this happens only with "newer" versions of Mercurial - starting with version 6.5.0. I've tested with 6.4.5 and did not get prompted for credentials. This testing was all done from within dedicated Python virtual environment. Since I am not experiencing the same issue with my test installation of Kallithea, I assume this is related to how the official Kallithea instance has been configured. Maybe if it detects that client can do Kerberos-based authentication it insists on using it? Best regards, Branko P.S. Of course, could be it's not Kerberos at all, just me jumping to conclusions. :) -- Branko Majic XMPP: branko at majic.rs Please use only Free formats when sending attachments to me. Бранко Мајић XMPP: branko at majic.rs Молим вас да додатке шаљете искључиво у слободним форматима. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From mads at kiilerich.com Tue Jul 23 14:36:03 2024 From: mads at kiilerich.com (Mads Kiilerich) Date: Tue, 23 Jul 2024 16:36:03 +0200 Subject: Authorization required for cloning Kallithea with Mercurial >= 6.5.0 In-Reply-To: <20240723131730.4b616fc0@majic.rs> References: <20240723131730.4b616fc0@majic.rs> Message-ID: Hi Branko Thanks for reporting. I have indeed recently caught and pushed support for latest Mercurial version and updated the server. But it was apparently not complete. This should be fixed by https://kallithea-scm.org/repos/kallithea-incoming/changeset/0245e0ebddd09c6bcc7b1cd74708cf19a49ee57e - can you confirm that? Regards, Mads On 23/07/2024 13:17, Branko Majic wrote: > Hello, > > Attempting to clone the official Kallithea repository (at > https://kallithea-scm.org/repos/kallithea) anonymously with newer > versions of Mercurial seems to fail, and prompts user for credentials: > > ----%---- > (hgtest) branko at che:~/projects/hgtest$ hg version > Mercurial Distributed SCM (version 6.5.3) > (see https://mercurial-scm.org for more information) > > Copyright (C) 2005-2023 Olivia Mackall and others > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > Enabled extensions: > > histedit internal > strip internal > (hgtest) branko at che:~/projects/hgtest$ hg clone https://kallithea-scm.org/repos/kallithea > destination directory: kallithea > http authorization required for https://kallithea-scm.org/repos/kallithea > realm: Kallithea authentication > user: > ----%---- > > Based on the realm part - I assume this could be something related to > Kerberos? > > From what I can tell, this happens only with "newer" versions of > Mercurial - starting with version 6.5.0. I've tested with 6.4.5 and did not get > prompted for credentials. This testing was all done from within dedicated > Python virtual environment. > > Since I am not experiencing the same issue with my test installation of > Kallithea, I assume this is related to how the official Kallithea > instance has been configured. Maybe if it detects that client can do > Kerberos-based authentication it insists on using it? > > Best regards, > Branko > > P.S. > Of course, could be it's not Kerberos at all, just me jumping to > conclusions. :) > > > _______________________________________________ > kallithea-general mailing list > kallithea-general at sfconservancy.org > https://lists.sfconservancy.org/mailman/listinfo/kallithea-general From branko at majic.rs Tue Jul 23 14:50:04 2024 From: branko at majic.rs (Branko Majic) Date: Tue, 23 Jul 2024 16:50:04 +0200 Subject: Authorization required for cloning Kallithea with Mercurial >= 6.5.0 In-Reply-To: References: <20240723131730.4b616fc0@majic.rs> Message-ID: <20240723164930.08b041de@majic.rs> On Tue, 23 Jul 2024 16:36:03 +0200 Mads Kiilerich wrote: > Hi Branko > > Thanks for reporting. > > I have indeed recently caught and pushed support for latest Mercurial > version and updated the server. But it was apparently not complete. > > This should be fixed by > https://kallithea-scm.org/repos/kallithea-incoming/changeset/0245e0ebddd09c6bcc7b1cd74708cf19a49ee57e > - can you confirm that? > > Regards, > Mads Hello Mads, That did the trick - works fine now with newer Mercurial CLI. Thank you for a quick fix. :) Best regards, Branko -- Branko Majic XMPP: branko at majic.rs Please use only Free formats when sending attachments to me. Бранко Мајић XMPP: branko at majic.rs Молим вас да додатке шаљете искључиво у слободним форматима. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From valentin at vrvis.at Mon Aug 26 15:06:15 2024 From: valentin at vrvis.at (Valentin Kleibel) Date: Mon, 26 Aug 2024 17:06:15 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad Message-ID: Hi, we have recently noticed a lot of errors in Kallithea from probing for a php vulnerability [1] looking like: "WebApp Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 0: invalid start byte" This can be reproduced with curl: curl https://example.com/?%AD The error stems from webob naively trying to utf-8 decode all %-encoded bytes in URL-parameters. In my opinion this exception should be handled and a error 400 should be returned. Attached you can find a small patch i created to check for this in kallithea/controllers/base.py:_basic_security_checks(). Best Regards, Valentin [1] https://devco.re/blog/2024/06/06/security-alert-cve-2024-4577-php-cgi-argument-injection-vulnerability-en/ -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-UnicodeDecodeError.diff Type: text/x-patch Size: 820 bytes Desc: not available URL: From mads at kiilerich.com Mon Aug 26 20:42:30 2024 From: mads at kiilerich.com (Mads Kiilerich) Date: Mon, 26 Aug 2024 22:42:30 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad In-Reply-To: References: Message-ID: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> Hi Thanks for the report and the patch. We could also catch this exception in the big try-except clause in __call__, and we could catch the more generic UnicodeError. But that would perhaps catch too much - also things that really are programming errors and shouldn't give a 400 reply. I think I would prefer to just catch this Unicode error if it happens, rather than trying to trigger it early. Perhaps by wrapping the call of _basic_security_checks. Do  you think that would catch too much or too little? /Mads On 26/08/2024 17:06, Valentin Kleibel wrote: > Hi, > > we have recently noticed a lot of errors in Kallithea from probing for > a php vulnerability [1] looking like: > "WebApp Error: UnicodeDecodeError: 'utf-8' codec can't decode byte > 0xad in position 0: invalid start byte" > > This can be reproduced with curl: > curl https://example.com/?%AD > > The error stems from webob naively trying to utf-8 decode all > %-encoded bytes in URL-parameters. > > In my opinion this exception should be handled and a error 400 should > be returned. > > Attached you can find a small patch i created to check for this in > kallithea/controllers/base.py:_basic_security_checks(). > > Best Regards, > Valentin > > > [1] > https://devco.re/blog/2024/06/06/security-alert-cve-2024-4577-php-cgi-argument-injection-vulnerability-en/ > > _______________________________________________ > kallithea-general mailing list > kallithea-general at sfconservancy.org > https://lists.sfconservancy.org/mailman/listinfo/kallithea-general From valentin at vrvis.at Tue Aug 27 09:52:35 2024 From: valentin at vrvis.at (Valentin Kleibel) Date: Tue, 27 Aug 2024 11:52:35 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad In-Reply-To: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> References: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> Message-ID: <8a7046bc-7f1a-4b43-97c7-660e0e7781e2@vrvis.at> Hi, > We could also catch this exception in the big try-except clause in > __call__, and we could catch the more generic UnicodeError. But that > would perhaps catch too much - also things that really are programming > errors and shouldn't give a 400 reply. I agree that this would catch too much. > I think I would prefer to just catch this Unicode error if it happens, > rather than trying to trigger it early. Perhaps by wrapping the call of > _basic_security_checks. Do  you think that would catch too much or too > little? I would prefer to catch this in webob where user provided data is decoded and raise a more specific error there... I just checked the code and noticed that request.GET will contain the parsed URL parameters for all request methods, so no reason to access request.POST. I think good options are: * wrap the _basic_security_checks call * wrap the check if webutils.session_csrf_secret_name in request.GET: in _basic_security_checks * trigger it early in a separate check on request.GET As webob will only do the processing once and store the processed GET for further access i still think triggering early in a separate check would be the cleaner option. I'm open to provide or test a patch using any of the options above. Thanks for your help, Valentin > On 26/08/2024 17:06, Valentin Kleibel wrote: >> Hi, >> >> we have recently noticed a lot of errors in Kallithea from probing for >> a php vulnerability [1] looking like: >> "WebApp Error: UnicodeDecodeError: 'utf-8' codec can't decode byte >> 0xad in position 0: invalid start byte" >> >> This can be reproduced with curl: >> curl https://example.com/?%AD >> >> The error stems from webob naively trying to utf-8 decode all >> %-encoded bytes in URL-parameters. >> >> In my opinion this exception should be handled and a error 400 should >> be returned. >> >> Attached you can find a small patch i created to check for this in >> kallithea/controllers/base.py:_basic_security_checks(). >> >> Best Regards, >> Valentin >> >> >> [1] >> https://devco.re/blog/2024/06/06/security-alert-cve-2024-4577-php-cgi-argument-injection-vulnerability-en/ >> >> _______________________________________________ >> kallithea-general mailing list >> kallithea-general at sfconservancy.org >> https://lists.sfconservancy.org/mailman/listinfo/kallithea-general > > From mads at kiilerich.com Sat Aug 31 15:47:35 2024 From: mads at kiilerich.com (Mads Kiilerich) Date: Sat, 31 Aug 2024 17:47:35 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad In-Reply-To: <8a7046bc-7f1a-4b43-97c7-660e0e7781e2@vrvis.at> References: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> <8a7046bc-7f1a-4b43-97c7-660e0e7781e2@vrvis.at> Message-ID: Hi I propose handling it like https://kallithea-scm.org/repos/kallithea-incoming/changeset/8b7c72355750fcf791cd4abfc332e26e556aee90 . We don't really *know* if the error came from the URL encoding, so I prefer to just catch the error we actually saw, and state the fact that we got a decode exception while processing request parameters. Do you agree patch that would be fine - and that the attribution is fine? I'm not positively sure webob can't choke on decoding anything else. That could perhaps be nice to test/review too. But that is apparently not a real problem (yet). /Mads From valentin at vrvis.at Tue Sep 3 09:29:01 2024 From: valentin at vrvis.at (Valentin Kleibel) Date: Tue, 3 Sep 2024 11:29:01 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad In-Reply-To: References: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> <8a7046bc-7f1a-4b43-97c7-660e0e7781e2@vrvis.at> Message-ID: <303306b5-40c5-465c-ae22-255ce3813f05@vrvis.at> Hi, > I propose handling it like > https://kallithea-scm.org/repos/kallithea-incoming/changeset/8b7c72355750fcf791cd4abfc332e26e556aee90 . > > We don't really *know* if the error came from the URL encoding, so I > prefer to just catch the error we actually saw, and state the fact that > we got a decode exception while processing request parameters. > > Do you agree patch that would be fine - and that the attribution is fine? Thanks for your work, that patch looks/works good and the attribution is fine with me. > I'm not positively sure webob can't choke on decoding anything else. > That could perhaps be nice to test/review too. But that is apparently > not a real problem (yet). I'm pretty sure it can, but one might need to send a wrong encoding in the Content-type header. For the URL-parameters, in contrast, every byte sequence encoded will always be interpreted as UTF-8, even before csrf or auth checks. Valentin From mads at kiilerich.com Sat Sep 21 19:26:02 2024 From: mads at kiilerich.com (Mads Kiilerich) Date: Sat, 21 Sep 2024 21:26:02 +0200 Subject: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad In-Reply-To: <303306b5-40c5-465c-ae22-255ce3813f05@vrvis.at> References: <010ad32f-9eb1-40ba-83a4-8aad9e09dc5a@kiilerich.com> <8a7046bc-7f1a-4b43-97c7-660e0e7781e2@vrvis.at> <303306b5-40c5-465c-ae22-255ce3813f05@vrvis.at> Message-ID: <2f04ef27-e15d-4b37-a073-41fdb91f867b@kiilerich.com> Thank you. I pushed that to the stable branch. /Mads On 03/09/2024 11:29, Valentin Kleibel wrote: > Hi, > >> I propose handling it like >> https://kallithea-scm.org/repos/kallithea-incoming/changeset/8b7c72355750fcf791cd4abfc332e26e556aee90 >> . >> >> We don't really *know* if the error came from the URL encoding, so I >> prefer to just catch the error we actually saw, and state the fact >> that we got a decode exception while processing request parameters. >> >> Do you agree patch that would be fine - and that the attribution is >> fine? > > Thanks for your work, that patch looks/works good and the attribution > is fine with me. > >> I'm not positively sure webob can't choke on decoding anything else. >> That could perhaps be nice to test/review too. But that is apparently >> not a real problem (yet). > > I'm pretty sure it can, but one might need to send a wrong encoding in > the Content-type header. > For the URL-parameters, in contrast, every byte sequence encoded will > always be interpreted as UTF-8, even before csrf or auth checks. > > Valentin