<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:3.0cm 2.0cm 3.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:862090070;
mso-list-type:hybrid;
mso-list-template-ids:741759472 232822146 67502083 67502085 67502081 67502083 67502085 67502081 67502083 67502085;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="DA" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Mads,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-US" style="font-family:Wingdings;color:#1F497D"><span style="mso-list:Ignore">Ø<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:DA">When debugging the problem, it might be simpler to let Kallithea create a new database and focus on whether it can store and
read unicode.</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">That’s what I did before reporting the bug: Kallithea does _<i>not</i>_ store data in Unicode format – and if you add new data to the database in Unicode, Kallithea breaks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">If I open the MySQL client and do, say, “update users set lastname = ‘Skjærlund’ where firstname = ‘Lars’”, then Kallithea breaks because the MySQL client adds data in Unicode.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">If the above command is to work, I have to type “update users set lastname = ‘Skjærlund’ where firstname = ‘Lars’”. In that case Kallithea survives – and it displays my name correctly in the UI.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><a href="http://www.dbc.dk/"><span style="color:#0563C1;mso-fareast-language:DA;text-decoration:none"><img border="0" width="80" height="19" id="Billede_x0020_1" src="cid:image002.png@01D121E6.DC951ED0" alt="mail_logo"></span></a><span style="color:#1F497D;mso-fareast-language:DA"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D;mso-fareast-language:DA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA">Med venlig hilsen<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA">Lars Skjærlund<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA">DevOps<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA">Tlf.: 44 86 77 77<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA">DBC as<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;mso-fareast-language:DA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#F26722;mso-fareast-language:DA"><a href="http://www.dbc.dk/"><span lang="EN-US" style="color:#F26722;text-decoration:none">www.dbc.dk</span></a></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#F26722;mso-fareast-language:DA"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#F26722;mso-fareast-language:DA"><a href="mailto:las@dbc.dk"><span style="color:#F26722;text-decoration:none">las@dbc.dk</span></a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:windowtext;mso-fareast-language:DA">Fra:</span></b><span style="color:windowtext;mso-fareast-language:DA"> Mads Kiilerich [mailto:mads@kiilerich.com]
<br>
<b>Sendt:</b> 17. november 2015 18:13<br>
<b>Til:</b> Lars Skjærlund <las@dbc.dk>; kallithea-general@sfconservancy.org<br>
<b>Emne:</b> Re: Bug in MySQL code?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 11/17/2015 03:56 PM, Lars Skjærlund wrote:<span style="font-size:12.0pt;mso-fareast-language:DA"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I’m afraid I’ve hit a bug:</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I want to migrate our Kallithea database from SQLite to MySQL. In order to do that, I dumped the SQLite database to an SQL script, modified the SQL commands to MySQL dialect, and ran the script against the MySQL database.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">It worked like a charm – except that Kallithea kept crashing with Unicode errors.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">But everything _<i>was</i>_ Unicode: The dump from SQLite was Unicode, my edits where fully Unicode compatible, and the database as well as the tables where created in MySQL as UTF8 compatible. After fighting this for
a long time, I tried letting Kallithea populate a new MySQL database – and discovered that Kallithea doesn’t store data in UTF8 format. It appears that the data is encoded for UTF8 twice, so my record looks like</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console",serif">+-----------+--------------+</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console",serif">| firstname | lastname |</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console",serif">+-----------+--------------+</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console",serif">| Lars | Skjærlund |</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Lucida Console",serif">+-----------+--------------+</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">If update my name to be true UTF8, Kallithea crashes. I haven’t tried other databases, but the encoding in SQLite is correct.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I solved my problem by running the SQL scriptfile through iconv before submitting it to MySQL, claiming the input was Latin1 and asking for UTF8 as output: In that way I got the same double-encoding that Kallithea appears
to require…</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:DA"><br>
Generally Kallithea works fine with unicode. It can however be tricky when it is interfacing with VCS or database. It is my impression that mysql also just works, but I use postgresql and haven't tried mysql myself.<br>
<br>
If the database really is in utf8, I guess some other layer in the stack (sqlalchemy or the database driver) messes it up.<br>
<br>
These two issue reports might give hints of what to check<br>
<a href="https://bitbucket.org/conservancy/kallithea/issues/9/doc-unicode-utf-8-issues-in-the-changelog">https://bitbucket.org/conservancy/kallithea/issues/9/doc-unicode-utf-8-issues-in-the-changelog</a><br>
<a href="https://bitbucket.org/conservancy/kallithea/issues/147/unicodeencodeerror-ascii-codec-cant-encode">https://bitbucket.org/conservancy/kallithea/issues/147/unicodeencodeerror-ascii-codec-cant-encode</a><br>
<br>
When debugging the problem, it might be simpler to let Kallithea create a new database and focus on whether it can store and read unicode. Next, you can make sure your converted database use the same encoding.<br>
<br>
/Mads<o:p></o:p></span></p>
</div>
</body>
</html>