sqlite utf8 encoding error

I have an application that uses sqlite3 to store job/error data.  When
I log in as a German user the error codes generated are translated into
German.  The error code text is then stored in the db.  When I use the
fetchall() to retrieve the data to generate a report I get the
following error:

Traceback (most recent call last):
  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 199, in
OnGenerateButtonNow
    self.OnGenerateButton(event)
  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 243, in
OnGenerateButton
    warningresult =3D messagecursor1.fetchall()
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong?  The string
that I store in the database table is:

'Keinen Text f=FCr =DCbereinstimmungsfehler gefunden'

I thought that all strings were stored in unicode in sqlite.

Greg Miller

0
11/17/2005 11:47:00 AM
comp.lang.python 74934 articles. 0 followers. Post Follow

8 Replies
190 Views

Similar Articles

[PageSpeed] 41
Greg Miller wrote:

> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
> unsupported Unicode code range
>
> does anyone have any idea on what could be going wrong?  The string
> that I store in the database table is:
>
> 'Keinen Text f�r �bereinstimmungsfehler gefunden'

$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text f�r �bereinstimmungsfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
  File "test.py", line 4, in ?
    u = s.decode("utf-8") # <-- this gives an error
  File "lib/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

> I thought that all strings were stored in unicode in sqlite.

did you pass in a Unicode string or an 8-bit string when you stored the text ?

</F> 



0
fredrik2101 (5275)
11/17/2005 12:04:25 PM
Greg Miller enlightened us with:
> 'Keinen Text f�r �bereinstimmungsfehler gefunden'

You posted it as "Keinen Text f<FC>r ...", which is Latin-1, not
UTF-8.

> I thought that all strings were stored in unicode in sqlite.

Only if you put them into the DB as such. Make sure you're inserting
UTF-8 text, since the DB won't do character conversion for you.

Sybren
-- 
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself? 
                                             Frank Zappa
0
11/17/2005 12:39:02 PM
Fredrik Lundh napisa�(a):

>>UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
>>unsupported Unicode code range
>>
>>does anyone have any idea on what could be going wrong?  The string
>>that I store in the database table is:
>>
>>'Keinen Text f�r �bereinstimmungsfehler gefunden'
> 
> $ more test.py
> # -*- coding: iso-8859-1 -*-
> u = u'Keinen Text f�r �bereinstimmungsfehler gefunden'
> s = u.encode("iso-8859-1")
> u = s.decode("utf-8") # <-- this gives an error
> 
> $ python test.py
> Traceback (most recent call last):
>   File "test.py", line 4, in ?
>     u = s.decode("utf-8") # <-- this gives an error
>   File "lib/encodings/utf_8.py", line 16, in decode
>     return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
> unsupported Unicode code range

I cann't wait for the moment when encoded strings go away from Python.
The more I program in this language, the more confusion this difference
is causing. Now most of functions and various object's methods accept
strings and unicode, making it hard to find sources of Unicode*Errors.

-- 
Jarek Zgoda
http://jpa.berlios.de/
0
jzgoda (227)
11/17/2005 8:08:45 PM
Jarek Zgoda wrote:
> Fredrik Lundh napisa=B3(a):
>
> >>UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
> >>unsupported Unicode code range
> >>
> >>does anyone have any idea on what could be going wrong?  The string
> >>that I store in the database table is:
> >>
> >>'Keinen Text f=FCr =DCbereinstimmungsfehler gefunden'
> >
> > $ more test.py
> > # -*- coding: iso-8859-1 -*-
> > u =3D u'Keinen Text f=FCr =DCbereinstimmungsfehler gefunden'
> > s =3D u.encode("iso-8859-1")
> > u =3D s.decode("utf-8") # <-- this gives an error
> >
> > $ python test.py
> > Traceback (most recent call last):
> >   File "test.py", line 4, in ?
> >     u =3D s.decode("utf-8") # <-- this gives an error
> >   File "lib/encodings/utf_8.py", line 16, in decode
> >     return codecs.utf_8_decode(input, errors, True)
> > UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
> > unsupported Unicode code range
>
> I cann't wait for the moment when encoded strings go away from Python.
> The more I program in this language, the more confusion this difference
> is causing. Now most of functions and various object's methods accept
> strings and unicode, making it hard to find sources of Unicode*Errors.

Library writers can speed up the transition by hiding 8bit interface,
for example:

import sqlite
sqlite.I_promise_to_pass_8bit_string_only_in_utf8_encoding(my_signature=3D"=
sig.gif")

if you don't call this function 8bit strings will not be accepted :)
IMHO if libraries keep on excepting both str and unicode till python
3=2E0, it will just prolong the confusion of unicode newbies instead of
guiding them in the right direction _right now_.

0
Serge.Orlov (257)
11/17/2005 9:35:42 PM
On 17 Nov 2005 03:47:00 -0800, "Greg Miller" <et1ssgmiller@gmail.com>
wrote:

>I have an application that uses sqlite3 to store job/error data.  When
>I log in as a German user the error codes generated are translated into
>German.  The error code text is then stored in the db.  When I use the
>fetchall() to retrieve the data to generate a report I get the
>following error:
>
>Traceback (most recent call last):
>  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 199, in
>OnGenerateButtonNow
>    self.OnGenerateButton(event)
>  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 243, in
>OnGenerateButton
>    warningresult = messagecursor1.fetchall()
>UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
>unsupported Unicode code range
>
>does anyone have any idea on what could be going wrong?  The string
>that I store in the database table is:
>
>'Keinen Text f�r �bereinstimmungsfehler gefunden'
>
>I thought that all strings were stored in unicode in sqlite.
>


No, they are stored as UTF-8 in sqlite and pysqlite has no way to make
sure the string you insert into the database is really encoded in
UTF-8 (the only secure way is to use Unicode strings).

How did you insert that string?

As a partial solution, try to disable automatic conversion of text
fields in Unicode strings:


def convert_text(s):
    # XXX do not use Unicode
    return s


# Register the converter with SQLite
sqlite.register_converter("TEXT", convert_text)


....connect("...",
detect_types=sqlite.PARSE_DECLTYPES|sqlite.PARSE_COLNAMES
)




Regards  Manlio Perillo
0
11/18/2005 9:44:11 AM
Thank you for all your suggestions.  I ended up casting the string to
unicode prior to inserting into the database.  

Greg Miller

0
11/18/2005 5:09:24 PM
On 18 Nov 2005 09:09:24 -0800, "Greg Miller" <et1ssgmiller@gmail.com>
wrote:

>Thank you for all your suggestions.  I ended up casting the string to
>unicode prior to inserting into the database.  
>

Don't do it by hand if it can be done by an automated system.

Try with:

from pysqlite2 import dbapi2 as sqlite

def adapt_str(s):
    # if you have declared this encoding at begin of the module
    return s.decode("iso-8859-1") 

sqlite.register_adapter(str, adapt_str)


Read pysqlite documentation for more informations:
http://initd.org/pub/software/pysqlite/doc/usage-guide.html



Regards  Manlio Perillo
0
11/19/2005 8:55:50 AM
Thanks again, I'll look into this method.

Greg Miller

0
11/21/2005 12:13:09 PM
Reply:
Similar Artilces:

Error 500
Hi, I am trying to pass the following and it keeps giving the same error... http://www.megamotza.com/cst_hsql.php?firstlogin=Y&abc=sysman&sql=select%20*%20from%20sysuser%20where%20companies%20LIKE'%0002%'%20AND%20usrflag%20='U'&tblname=curSysuser ....the problem is the LIKE '%0002%'. If I remove the %'s from each side of the value, no error. Anyone got any ideas Regards Doug Johnston "Doug Johnston" wrote: > Hi, > > I am trying to pass the following and it keeps giving the same error... > > http://www.megamotza.com/cst...

Re: installing SAS
another thing: if you are still not successful, try to find actual releases of that MS tools on the MS webpages and install them. Reboot your PC after that and try to install SAS after that. If there is still something missing, the install routine should tell you that. Tell us about your success. Did you contact SI's tech support? Maybe they had already heared about that problem. Regards, Gerhard On Fri, 22 Dec 2006 07:48:18 -0500, Gerhard Hellriegel <gerhard.hellriegel@T-ONLINE.DE> wrote: >Try to stop everything which might be running. Detatch the computer from the >networ...

ComboBox error (Windows compile)
Major bug with comboboxes, Windows build only: 1) If you have a comboBox, the value of which you want to fill in via code, for example: comboBox.Text = "TEST" 2) It may appear that the comboBox took the value (the word "TEST" will appear in the comboBox), but if you try reading it... MsgBox comboBox.Text You will find the value displayed is the value that was in the box before you executed step one. This does not appear in Mac OS X, but, as far as I have seen, in Windows applications built using the Mac OS X IDE. I'm using RB 5.5.4 on Mac OS...

Error in ==> dio = digitalio('parallel', 'LPT1'); Could not copy the WINIO.SYS device driver to the System32 directory.
I'm trying to synchronize my PC with another PC using the parallel port: dio = digitalio('parallel', 'LPT1'); hwlines = addline(dio, 0:7, 'out'); hwlines = addline(dio, 8:16, 'in'); and it's giving me this error message: ??? Error using ==> digitalio.digitalio at 93 Error using ==> digitalio.digitalio>localCreateDigitalIOObject at 187 PARALLEL: Could not copy the WINIO.SYS device driver to the System32 directory. Error in ==> dio = digitalio('parallel', 'LPT1'); I was wondering if anyone knows why I'...

SOAP error
I'm running Tomcat 3.2.4 with SOAP 2.2. I've deployed a small java class. The problem is that after invoking it's method i get a response with a fault which says: Unable to resolve target object: Class org.apache.soap.server.http.ServerHTTPUtils can not access a member of class samples.airport.DatabaseConnector with modifiers "" I'm clueless as what to do, the samples that came with soap are working, at least I've tested a few simple, like addressBook etc. Any hint? ...

regex and utf8 characters (german umlauts)
Hi, the following little perl snippet perl -e '($string = "AAA ÄÄÄ BBB CCC DDD") =~ s/(\p{IsUpper}+)/\L\u\1\E/g; print $string . "\n"' gives this result: Aaa ÄÄÄ Bbb Ccc Ddd How do I turn those umlauts into "Äää" also? I tried adding "use utf8;", but that didn't help. Thanks... Dirk -- Dirk Heinrichs | Tel: +49 (0)162 234 3408 Configuration Manager | Fax: +49 (0)211 47068 111 Capgemini Deutschland | Mail: dirk.heinrichs@capgemini.com Hambornerstraße 55 | Web...

Re: Error codes and SIG(0)
> However, there seems to be a problem. The TSIG extended rcode values are > stored in the TSIG, while the "extended RCODE" value is in the OPT RR. > According to the IANA database, both have reserved the code 16 for BADVERS > and BADSIG. There is no space for an extended rcode value in the SIG(0) RR, > unless the spec is changed to give remap a field to that purpose. > > Or else the two competing rcode assignments be reconciled somehow. the original edns proposal had a major and a minor version number, with the intent that minor version numbers coul...

&quot;LLB of that name already exists&quot; Error
Hi - &nbsp; I just tried to move a LabVIEW VI from a desktop to a&nbsp;laptop PC.&nbsp; 6-year-old desktop wasn't powerful enough to run the program. &nbsp; In order to group the main VI with the sub VIs, I used the "create folder" option.&nbsp; I then copied this over to the laptop.&nbsp; The&nbsp;laptop has operated LabVIEW for a while.&nbsp; I also recently loaded the drivers for Labjack U12 onto it (my program reads voltages from a Labjack). &nbsp; I was able to run the program perfectly fine until I restarted the computer.&nbsp; When ...

Help : Error in scrubyt
hi i have one problem when i runing the my code will come this erros : ================================ C:/Ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_re quire': no such file to load -- firewatir (LoadError) from C:/Ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `re quire' from C:/Ruby/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navi gation/agents/firewatir.rb:2 from C:/Ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `ge m_original_require' from C:/Ruby/lib/ruby/site_ruby/1.8/rub...

CRC errors with WF2 download?
I have tried twice to download the WF2 windows CDs from PTC site, but both times when extracting with winzip I get CRC errors several times. If I just carry on and install WF2 it installs ok but upon trying to run WF2 nothing happens (well proe.exe appears for about a second in task manager). The license server installs and runs fine though. Anyone else managed ok? Any suggestions? Cheers Scott "scott" <scott@spam.com> wrote in message news:<c7fpsq$miu$1@news.f.de.plusline.net>... > I have tried twice to download the WF2 windows CDs from PTC site,...

ORacle 8 or 9 Trigger Compile error help please
Here is my trigger below!!! please help with those compile errors!! SQL> CREATE OR REPLACE TRIGGER trg_insertreg19toreg20 2 INSTEAD OF INSERT ON BD_REGISTRATIONS1_9_VIEW 3 FOR EACH ROW 4 BEGIN 5 INSERT INTO BD_REGISTRATIONS (REG_USER_ID, REG_BLDRV_ID, REG_DONATESCHTIME, 6 REG_DONORWALKIN, REG_DONATEACTTIME, REG_DONOREXCUSED, REG_REIMBURSED, REG_THANKYOUSENT) 7 VALUES (:NEW.REG_USER_ID, 8 :NEW.REG_BLDRV_ID, 9 :NEW.REG_DONATESCHTIME, 10 :NEW.REG_DONORWALKIN, 11 :NEW.REG_DONATEACTTIME, 12 :NEW.REG_DONOREXCUSED, 13 ...

TAO Encoding
ACE VERSION: 5.5a_p4 HOST MACHINE and OPERATING SYSTEM: Windows XP Professional Version 2002 SP2 TARGET MACHINE and OPERATING SYSTEM, if different from HOST:NO COMPILER NAME AND VERSION (AND PATCHLEVEL): Visual studio 2005 THE $ACE_ROOT/ace/config.h FILE #define ACE_DISABLE_WIN32_ERROR_WINDOWS #define ACE_DISABLE_WIN32_INCREASE_PRIORITY #define ACE_HAS_XML_SVC_CONF #include "config-win32.h" DOES THE PROBLEM AFFECT: COMPILATION? NO LINKING? NO EXECUTION? NO OTHER YES Good day. I interest as TAO select codes...

ERROR: Connect: Class not registered
Hi all, Can some one help me to sort out the problem which I am facing while exporting the data from sas to Excel. PROC EXPORT DATA= tera.Physician_level_summary OUTFILE= "C:\Weekly.xls" DBMS=EXCEL; SHEET="Phy_level_summary"; RUN; NOTE: New file "C:\...\Weekly.xls" will be created if the export process succeeds. ERROR: Connect: Class not registered NOTE: The SAS System stopped processing this step because of errors. NOTE: PROCEDURE EXPORT used (Total process time): real time 0.46 seconds cpu time ...

File Error 206004 again
Recently I've experienced "File Error 206004" CW alert when trying to open an .mcp that has been taken from the SourceSafe database. Somebody else checked it in on another computer. I couldn't open the project till I haven't removed the OS X Volume entry from my CW preferences. I've tried to define OS X Volume again - I've got the same error each time I wanted to save the project preferences. After long and exhausting In article <f0040453c5ec48e1e569174a39370265@localhost.talkaboutmac.com>, "nava" <nava.carmon@creo.com> wr...

Extrange error Why?
Hi: I'm testing with fireforx 1.0 on MdkLinux10.0 I can not figure out why am I getting this error: Error: [Exception... "'Permiso denegado para obtener propiedad XULElement.selectedIndex' when calling method: [nsIAutoCompletePopup::selectedIndex]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "JS frame :: file:///home/mcanedo/Desktop/forma.html :: validar :: line 151" data: no] Archivo Fuente: file:///home/mcanedo/Desktop/forma.html L�nea: 151 This is the code I'm working on: <html> <head><TITLE></T...

trade off by using Base64 encoding
Hi, I ma thinking whether to use Base64 encoding to encode the binary content in the XML file. I have done some simple calculations, it seems to me that the size for encoded content increases by ~30%, is this the drawback for using the encoding scheme? Thanks. By my math, you will use 32 bits for every 24 encoded (4 base64 characters (bytes) for 3 octets (bytes) worth of bits, or exactly 1-1/3 times the space required for the raw bits. Since there's no commonly-available, safe alternative for carrying binary data in XML elements, I suppose you should be asking "drawback relative to...

TCP checksum error
...

SQLite, SQLite++, wxSQLite or LiteSQL? OODB?
Looking for a file-based SQL RDBMS for my application, I discovered that there are at least 5 different flavours of it that can be used with wxWidgets: wxSQLite: http://sourceforge.net/projects/wxsqlite http://wiki.wxwidgets.org/wiki.pl?WxSQLite SQLite++: http://www.eztools-software.com/tools/SqlitePlus/sqliteplusplus.asp SQLitepp: http://amol.detach.it/wiki/SQLitepp LiteSQL: http://litesql.sourceforge.net/ And, of course, the original SQLIte: http://www.sqlite.org/ Now, I wonder which one is the best bet. Do you have any experience with any of them? Any suggestio...

Prime SQLite Database?
Hello, Curious if any consideration may have been given to bundling SQLite with the current or future programming environments within the HP Prime? I think there are many potential applications for a database engine, including possibly for OS purposes even. Planning on getting the Prime soon to start mucking around with the programming end of things, and this came to mind right away with the expanded memory, etc. -- Jacob Wall http://sqlite.org/about.html On 10/19/2013 2:26 PM, Jacob Wall wrote: > Hello, > > Curious if any consideration may have been gi...

Error using konstruct for kde 3.3.1
Hi: This is the first time I use konstruct, I have Mdk10 with Kde 3.2Branch. I have downloaded konstruct. I go to my localizad folder as indicated in Readme. I type make install, it starts compiling, but then it stops, this is the tail of the messages. Any Idea what I need to do? thanks! make[5]: No se hace nada para `install-data-am'. make[5]: Leaving directory `/home/mcanedo/Software/konstruct/kde/kdelibs/work/kdelibs-3.3.1/dcop/client' make[4]: Leaving directory `/home/mcanedo/Software/konstruct/kde/kdelibs/work/kdelibs-3.3.1/dcop/client' make[3]: Leaving directory `/home/mcane...

encoder
Hi!I have a really easy questiion but I'm new to Labview and have difficulties solving it. I hope sb can help me.The problem:The information I read from the encoders gives me an unsigned byte(8 bit integer) . depending on if I want to read MSB or LSB I get different bytes. I have 2 buttons which indicates if I should get a MSB or LSB information out from encoders.I want to use a case to be able to do some opperations on each one of the bytes depending on if it is a MSB or LSB.How can I specify the cases so the program understands when MSB is read it should do the opporations I want it to d...

Error in Autosar codegen using RTW
Hello, I got the error below while generating Autosar code from my model, i have variant subsystems too in my model. Can anyone plz guide on the source of this error and how to resolve this! Unexpected or internal error encountered in "in CGIR: .\sl_engin\rtwcg_context_impl.cpp line 5870: SlBlockEngineHelper::getInternalBlockTypeEnum(srcBlock) == SL_SIGNAL_CONVERSION_BLOCK && (slbGetReasonForInsertion(srcBlock) == SIGCONVERT_REASON_AUTOSAR) ". Please report this to MathWorks if you can cause it to recur Thanks in advance! "Archana " <adwivedi1910@g...

1.4.2 translate errors
I programmed a simple GUI using Swing with build 1.3.1 on a Windows machine. Now someone is interested in using this code under RH Linu! with a Blackdown 1.4.2 build. It has been some years since I looked at Java but I will dig into it if I must. I just want to do this as fast as possible. I get the following errors when I try to translate. There may not be enough information here but I'd rather not post code and use more bandwidth. If someone sees something obvious please let me know. The JDC doesn't offer support so before I try that route I will try here. Thank you. - Andrew...

Error: Couldn't find unicodeMap file for the ____ encoding
Hello, I'm using kubuntu feisty. When I do the following command pdftohtml -enc windows-1252 -i -p file.pdf I get the following error Error: Couldn't find unicodeMap file for the 'windows-1252' encoding Regardless of what encoding I try. The reason I need to do this is that when I convert PDFs to HTML files I get some errors in the encoding. It's all English, but the quotes for example get replaced by a series of characters and so on. Any ideas what I can do to fix this? Thanks. On 29 Apr, 00:53, fli...@gmail.com wrote: > Hello, I'm using kubuntu feisty. Whe...

Character Encodings etc
If I run clisp in a terminal where the default character coding is UTF8, how can I get lisp to output the sequence 0xc3 0x98. If I do it like this: Break 2 [4]> (defun test () (princ (character #xc3))(princ (character #x98)) nil) TEST Break 2 [4]> (test) Ø NIL It has actually output 0xc3 0x83 0xc2 0x98 I have managed out output characters to a file. But if I replace the stream with *standard-output* it causes an error. (I don't have the code I used to hand). --=-=-= Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable ...