<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Ok, finally got it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I built Kallithea from the source repo. I changed waitress for uwsgi and I also changed the sqlite db for a postgres one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Had to readjust my git client http post buffer:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">git config http.postBuffer 524288000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">After that, got a 11Mb pdf file into the git repo with no problems.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Mat<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Matey Chopov
<br>
<b>Sent:</b> Wednesday, June 13, 2018 6:16 PM<br>
<b>To:</b> Thomas De Schampheleire <patrickdepinguin@gmail.com><br>
<b>Cc:</b> kallithea-general@sfconservancy.org<br>
<b>Subject:</b> RE: Potential issue with pdf files and git repos<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Thomas,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Here is a similar version to the original .pdf I was talking about here:
<a href="http://www.gimpel.com/html/manual.pdf">http://www.gimpel.com/html/manual.pdf</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I think you are right about the issue being with file size instead of file content.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">When I converted the original pdf file to text and back to pdf it drastically changed size. The original is about 4.5MB while the converted one was around 1MB.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I proceeded to push a bigger .pdf file (around 11.1 MB size with no encryption) and I got the same trace error, so apparently it isn’t about the file being encrypted or not.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I am getting a similar trace when using uwsgi, I played around with the buffer size, but to no avail.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Here is another sample log file when I try to push a big .pdf file to a git repo on Kallithea 0.3.5 using uwsgi as a web server.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><a href="https://pastebin.com/Y1piX0vX">https://pastebin.com/Y1piX0vX</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Tomorrow, I will try to test with the default branch instead of 0.3.5.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Mat<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Thomas De Schampheleire [<a href="mailto:patrickdepinguin@gmail.com">mailto:patrickdepinguin@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, June 13, 2018 4:52 PM<br>
<b>To:</b> Matey Chopov <<a href="mailto:matey.chopov@ca.abb.com">matey.chopov@ca.abb.com</a>><br>
<b>Cc:</b> <a href="mailto:kallithea-general@sfconservancy.org">kallithea-general@sfconservancy.org</a><br>
<b>Subject:</b> Re: Potential issue with pdf files and git repos<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jun 13, 2018, 22:29 Thomas De Schampheleire <<a href="mailto:patrickdepinguin@gmail.com">patrickdepinguin@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">2018-06-13 20:39 GMT+02:00 Matey Chopov <<a href="mailto:matey.chopov@ca.abb.com" target="_blank">matey.chopov@ca.abb.com</a>>:<br>
> Hi,<br>
><br>
> It looks like it happens with a specific .pdf manual, I tested it with another .pdf file and the exception didn't occur, the file got pushed correctly.<br>
><br>
> Here's the line in the trace I think is the most interesting:<br>
><br>
> DatabaseError: (DatabaseError) file is encrypted or is not a database u'SELECT ui.ui_id AS ui_ui_id, ui.ui_section AS ui_ui_section, ui.ui_key AS ui_ui_key, ui.ui_value AS ui_ui_value, ui.ui_active AS ui_ui_active \nFROM ui \nWHERE ui.ui_key = ?' ('push_ssl',)<br>
> 2018-06-13 11:07:04.946 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A12756&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174375499&sdata=W6criUpKycuZzM7RhwN%2FHAFWlvo3ORA%2BYjGzFo%2BVbDQ%3D&reserved=0" target="_blank">
0.0.0.0:12756</a> at 0x7f06c5923e90><br>
><br>
> I have uploaded the surrounding log on pastebin: <a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2Fn9fY4xae&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174385503&sdata=mg7SjCBrBxdwo%2FuD9Bj5O%2FVUgXb4J7A%2BiHT5N3R9X38%3D&reserved=0" target="_blank">
https://pastebin.com/n9fY4xae</a><br>
><br>
> So the problematic pdf that I though wasn't encrypted, was actually encrypted with RC4, the weird thing is that in Git Extensions you can still see the file contents in the "diff" section.<br>
><br>
> Apparently, pdf readers automatically decrypt such files if there is no password (which is the current case).<br>
><br>
> I used qpdf to decrypt the file (with no password) which gave another valid .pdf file with no encryption (at least that's what I get when I analyze the file with pdfinfo).<br>
><br>
> Tried pushing that file too, but it still failed.<br>
><br>
> I played with the pdf headers, changed the Creator and Producer values to the ones of a .pdf file I know could be uploaded. Same error.<br>
><br>
> I tried converting the file from pdf1.3 to pdf1.4, same issue.<br>
><br>
> So, what finally worked for me was converting from pdf to ps, then to text, then from the text file, to ps, and then to pdf. The indexing table got screwed, but that doesn't really bother me. Finally, pushed the new pdf file to the git repo with success.<br>
><br>
> Commands:<br>
><br>
> pdftops test.pdf <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftest.ps&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174385503&sdata=TGgIAOfHl3F23zTRt%2BBFffRTM9%2FOdJxPIERyGF%2Fq82o%3D&reserved=0" target="_blank">
test.ps</a><br>
> ps2txt <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftest.ps&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174395507&sdata=zeS75LIlalR4ER%2BJuA%2FQJ4QeYBj1zBaLRS4TMjIC%2B0E%3D&reserved=0" target="_blank">
test.ps</a> test.txt<br>
><br>
> enscript -B --margins=10:10 -o <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftest.ps&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174406836&sdata=s6sM2Khn5hyKssqEOxzjpxGO3Tt7N7aGnDhJmX8iCq4%3D&reserved=0" target="_blank">
test.ps</a> -f <a href="mailto:Courier@7.3/1">Courier@7.3/1</a> test.txt<br>
> ps2pdf <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftest.ps&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174406836&sdata=s6sM2Khn5hyKssqEOxzjpxGO3Tt7N7aGnDhJmX8iCq4%3D&reserved=0" target="_blank">
test.ps</a> test_last.pdf<br>
><br>
<br>
Your log also shows:<br>
<br>
Traceback (most recent call last):<br>
File "/opt/Kallithea/local/lib/python2.7/site-packages/waitress/task.py",<br>
line 74, in handler_thread<br>
task.service()<br>
File "/opt/Kallithea/local/lib/python2.7/site-packages/waitress/channel.py",<br>
line 368, in service<br>
request._close()<br>
File "/opt/Kallithea/local/lib/python2.7/site-packages/waitress/parser.py",<br>
line 249, in _close<br>
body_rcv.getbuf()._close()<br>
File "/opt/Kallithea/local/lib/python2.7/site-packages/waitress/buffers.py",<br>
line 303, in _close<br>
buf._close()<br>
File "/opt/Kallithea/local/lib/python2.7/site-packages/waitress/buffers.py",<br>
line 110, in _close<br>
self.file.close()<br>
IOError: [Errno 9] Bad file descriptor<br>
<br>
<br>
which reminds me of following two open issues:<br>
<br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.org%2Fconservancy%2Fkallithea%2Fissues%2F219%2Fwaitress-exception-when-serving-file&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174415529&sdata=exLgLZUqvSCv2fdNzcX1RwwbcqYT%2BO6KZUFySQhS0NU%3D&reserved=0" target="_blank">https://bitbucket.org/conservancy/kallithea/issues/219/waitress-exception-when-serving-file</a><br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.org%2Fconservancy%2Fkallithea%2Fissues%2F229%2Fbad-file-descriptor&data=02%7C01%7Cmatey.chopov%40ca.abb.com%7C2430c095be924131e48d08d5d16f7f99%7C372ee9e09ce04033a64ac07073a91ecd%7C0%7C0%7C636645199174425545&sdata=Yl31jgLaplWLQL6IAczbkc8jO4x%2Fk2ngaC%2Bh4i%2BqURg%3D&reserved=0" target="_blank">https://bitbucket.org/conservancy/kallithea/issues/229/bad-file-descriptor</a><br>
<br>
<br>
<br>
Is the PDF on which you see the issue something you could share?<br>
Or could you create another PDF with dummy data that also exhibits the issue?<br>
<br>
If at all it would be possible, could you test with the default branch<br>
of Kallithea, instead of 0.3.5 ?<o:p></o:p></p>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Note that it doesn't make sense to me that the contents of the file would matter. I think it is more likely about the file size. Could you check the file sizes of the different files you tested with?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Also, the reporter of issue #219 reported back that his issue was gone when switching away from waitress to another web server, in his case uwsgi. Could you try that too ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thomas<o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>