Issue #378: UnicodeDecodeError after upgrade (conservancy/kallithea)

[akoscomp2]

New issue 378: UnicodeDecodeError after upgrade
https://bitbucket.org/conservancy/kallithea/issues/378/unicodedecodeerror-after-upgrade

Akos:

I am upgradeing kallithea 0.3.2 to 0.6.1, and after upgrade when I try to access users and repos than contain unicode names, I got this error in logs:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>

‌

Full logfile when I try to access Users page \(I have LDAP users with unicode charactes\):

Jun 15 16:23:56 hg gearbox\[17152\]: Traceback \(most recent call last\):  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/session.py", line 71, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     response = self.next\_handler\(controller, environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/i18n.py", line 71, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.next\_handler\(controller, environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/wsgiapp.py", line 243, in \_dispatch  
Jun 15 16:23:56 hg gearbox\[17152\]:     return controller\(environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/lib/base.py", line 536, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return super\(BaseController, self\).**call**\(environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 118, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     response = self.\_perform\_call\(context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 107, in \_perform\_call  
Jun 15 16:23:56 hg gearbox\[17152\]:     r = self.\_call\(action, params, remainder=remainder, context=context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoratedcontroller.py", line 129, in \_call  
Jun 15 16:23:56 hg gearbox\[17152\]:     output = controller\_caller\(context\_config, bound\_controller\_callable, remainder, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoration.py", line 21, in \_decorated\_controller\_caller  
Jun 15 16:23:56 hg gearbox\[17152\]:     return application\_controller\_caller\(tg\_config, controller, remainder, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/configurator/components/dispatch.py", line 114, in \_call\_controller  
Jun 15 16:23:56 hg gearbox\[17152\]:     return controller\(\*remainder, \*\*params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/controllers/admin/users.py", line 70, in index  
Jun 15 16:23:56 hg gearbox\[17152\]:     .order\_by\(func.lower\(User.username\)\)   
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all  
Jun 15 16:23:56 hg gearbox\[17152\]:     return list\(self\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in **iter**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.\_execute\_and\_instances\(context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in \_execute\_and\_instances  
Jun 15 16:23:56 hg gearbox\[17152\]:     result = conn.execute\(querycontext.statement, self.\_params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     return meth\(self, multiparams, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in \_execute\_on\_connection  
Jun 15 16:23:56 hg gearbox\[17152\]:     return connection.\_execute\_clauseelement\(self, multiparams, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in \_execute\_clauseelement  
Jun 15 16:23:56 hg gearbox\[17152\]:     distilled\_params,  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute\_context_  
_Jun 15 16:23:56 hg gearbox\[17152\]:     e, statement, parameters, cursor, context_  
_Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1521, in handle\_dbapi\_exception_  
_Jun 15 16:23:56 hg gearbox\[17152\]:     util.raise_\(exc\_info\[1\], with\_traceback=exc\_info\[2\]\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise  
Jun 15 16:23:56 hg gearbox\[17152\]:     raise exception  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in \_execute\_context  
Jun 15 16:23:56 hg gearbox\[17152\]:     cursor, statement, parameters, context  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do\_execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     cursor.execute\(statement, parameters\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     res = self.\_query\(query\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 317, in \_query  
Jun 15 16:23:56 hg gearbox\[17152\]:     self.\_post\_get\_result\(\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 352, in \_post\_get\_result  
Jun 15 16:23:56 hg gearbox\[17152\]:     self.\_rows = self.\_fetch\_row\(0\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 325, in \_fetch\_row  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.\_result.fetch\_row\(size, self.\_fetch\_type\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/usr/lib/python3.6/encodings/cp1252.py", line 15, in decode  
Jun 15 16:23:56 hg gearbox\[17152\]:     return codecs.charmap\_decode\(input,errors,decoding\_table\)  
Jun 15 16:23:56 hg gearbox\[17152\]: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>  
Jun 15 16:23:56 hg gearbox\[17152\]: Traceback \(most recent call last\):  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/session.py", line 71, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     response = self.next\_handler\(controller, environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/i18n.py", line 71, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.next\_handler\(controller, environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/wsgiapp.py", line 243, in \_dispatch  
Jun 15 16:23:56 hg gearbox\[17152\]:     return controller\(environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/lib/base.py", line 536, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return super\(BaseController, self\).**call**\(environ, context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 118, in **call**  
Jun 15 16:23:56 hg gearbox\[17152\]:     response = self.\_perform\_call\(context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 107, in \_perform\_call  
Jun 15 16:23:56 hg gearbox\[17152\]:     r = self.\_call\(action, params, remainder=remainder, context=context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoratedcontroller.py", line 129, in \_call  
Jun 15 16:23:56 hg gearbox\[17152\]:     output = controller\_caller\(context\_config, bound\_controller\_callable, remainder, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoration.py", line 21, in \_decorated\_controller\_caller  
Jun 15 16:23:56 hg gearbox\[17152\]:     return application\_controller\_caller\(tg\_config, controller, remainder, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/configurator/components/dispatch.py", line 114, in \_call\_controller  
Jun 15 16:23:56 hg gearbox\[17152\]:     return controller\(\*remainder, \*\*params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/controllers/admin/users.py", line 70, in index  
Jun 15 16:23:56 hg gearbox\[17152\]:     .order\_by\(func.lower\(User.username\)\)   
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all  
Jun 15 16:23:56 hg gearbox\[17152\]:     return list\(self\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in **iter**  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.\_execute\_and\_instances\(context\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in \_execute\_and\_instances  
Jun 15 16:23:56 hg gearbox\[17152\]:     result = conn.execute\(querycontext.statement, self.\_params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     return meth\(self, multiparams, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in \_execute\_on\_connection  
Jun 15 16:23:56 hg gearbox\[17152\]:     return connection.\_execute\_clauseelement\(self, multiparams, params\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in \_execute\_clauseelement  
Jun 15 16:23:56 hg gearbox\[17152\]:     distilled\_params,  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute\_context_  
_Jun 15 16:23:56 hg gearbox\[17152\]:     e, statement, parameters, cursor, context_  
_Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1521, in handle\_dbapi\_exception_  
_Jun 15 16:23:56 hg gearbox\[17152\]:     util.raise_\(exc\_info\[1\], with\_traceback=exc\_info\[2\]\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise  
Jun 15 16:23:56 hg gearbox\[17152\]:     raise exception  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in \_execute\_context  
Jun 15 16:23:56 hg gearbox\[17152\]:     cursor, statement, parameters, context  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do\_execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     cursor.execute\(statement, parameters\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute  
Jun 15 16:23:56 hg gearbox\[17152\]:     res = self.\_query\(query\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 317, in \_query  
Jun 15 16:23:56 hg gearbox\[17152\]:     self.\_post\_get\_result\(\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 352, in \_post\_get\_result  
Jun 15 16:23:56 hg gearbox\[17152\]:     self.\_rows = self.\_fetch\_row\(0\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 325, in \_fetch\_row  
Jun 15 16:23:56 hg gearbox\[17152\]:     return self.\_result.fetch\_row\(size, self.\_fetch\_type\)  
Jun 15 16:23:56 hg gearbox\[17152\]:   File "/usr/lib/python3.6/encodings/cp1252.py", line 15, in decode  
Jun 15 16:23:56 hg gearbox\[17152\]:     return codecs.charmap\_decode\(input,errors,decoding\_table\)  
Jun 15 16:23:56 hg gearbox\[17152\]: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>