f



UnicodeDecodeError: 'ascii' codec can't decode byte

Hello

	It seems like I have Unicode data in a CSV file but Python is using
a different code page, so isn't happy when I'm trying to read and put
this data into an SQLite database with APSW:

========
sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"
cursor.executemany(sql, records("test.tsv"))
"""
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position
18: ordinal not in range(128)
"""
========

What should I do so Python doesn't raise this error? Should I convert
data in the CVS file, or is there some function that I should call
before APSW's executemany()?

Thank you.
0
nospam21 (19047)
6/17/2008 12:02:45 AM
comp.lang.python 77058 articles. 5 followers. Post Follow

2 Replies
1517 Views

Similar Articles

[PageSpeed] 6

Gilles Ganault wrote:

> It seems like I have Unicode data in a CSV file but Python is using
> a different code page, so isn't happy when I'm trying to read and put
> this data into an SQLite database with APSW:

My guess is that you have non-ascii characters in a bytestring.
 
> What should I do so Python doesn't raise this error? Should I convert
> data in the CVS file, or is there some function that I should call
> before APSW's executemany()?

You cannot have unicode data in a file, only unicode converted to
bytestrings using some encoding. Assuming that encoding is UTF-8 and that
apsw can cope with unicode, try to convert your data to unicode before
feeding it to the database api:

> sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"

  rows = ([col.decode("utf-8") for col in row] for row in
records("test.tsv")) 
  cursor.executemany(sql, rows)

Peter
0
__peter__ (4031)
6/17/2008 7:23:28 AM
On Tue, 17 Jun 2008 09:23:28 +0200, Peter Otten <__peter__@web.de>
wrote:
> Assuming that encoding is UTF-8 and that apsw can cope
> with unicode, try to convert your data to unicode before
> feeding it to the database api:
>
>> sql = "INSERT INTO mytable (col1,col2) VALUES (?,?)"
>
>  rows = ([col.decode("utf-8") for col in row] for row in
>records("test.tsv")) 
>  cursor.executemany(sql, rows)

Thanks again.
0
nospam21 (19047)
6/17/2008 9:09:30 PM
Reply: