Develop Reference

portugese characters in DBsqlite and python parsing not recognised

I got a database in DBsqlite.in this DBsqlite database I have have a records containing portugese text like "Hiper-radiação simétrica periocular bem delimitada, homogênea."
and the characters like ç ã é ê don't parse right in my python script.
While normal english text is doing it perfectly.
In my terminal window (I use a mac) the
I know it has something do to with the encoding. but the code still doesn't recognise portugese.
my sample code:
# -*- coding: UTF-8 -*-
import xml.etree.ElementTree as ET
import sqlite3
#open a database connection to the database translateDB.sqlite
conn = sqlite3.connect('translateDB.sqlite')
#prepare a cursor object using cursus() method
cursor = conn.cursor()
#test input
# this doesn't work
text = ('Hiper-radiação simétrica periocular bem delimitada, homogênea')
# this does work in english
#text = ('Well delimited, homogeneous symmetric periocular hyper- radiation.')
# Execute SQL query using execute() method.
cursor.execute('SELECT * FROM translate WHERE L2_portugese=?', (text,))
# Fetch a single row using fetchone() method and display it.
print cursor.fetchone()
# Disconnect from server
conn.close()
any tips & tricks are greatly appreciated. Ron

Python to Pull Oracle Data in Unicode (Arabic) format

I am using cx_Oracle to fetch some data stored in Arabic characters from an Oracle database. Below is how I try to connect to the database. When I try to print the results, specially those columns stored in Arabic, I get something like "?????" which seems to me that the data was not coded properly.
I tried to print random Arabic string in Python it went alright, which indicates the problem is in the manner in which I am pulling data from the database.
connection = cx_Oracle.connect(username, password, instanceName)
wells = getWells(connection)
def getWells(conn):
cursor = conn.cursor()
wells = []
cursor.execute(sql)
clmns = len(cursor.description)
for row in cursor.fetchall():
print row
well = {}
for i in range(0, clmns):
if type(row[i]) is not datetime.datetime:
well[cursor.description[i][0]] = row[i]
else:
well[cursor.description[i][0]] = row[i].isoformat()
wells.append(well)
cursor.close()
connection.close()
return wells

In order to force a reset of the default encoding from the environment, you can call the setdefaultencoding method in the sys module.
As this is not recommended, it is not visible by default and a reload is required.
It is recommended that you attempt to fix the encoding set in the shell for the user on the host system rather than modifying in a script.
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

Hebrew appears as gibberish, DB importing with PyPyODBC

I'm trying to work with a Hebrew database, unfortunately the output is gibberish. What am I doing wrong?
# -*- coding: utf-8 -*-
import pypyodbc
conn = pypyodbc.connect('Driver={Microsoft Access Driver (*.mdb)};DBQ=C:\\client.mdb')
cur = conn.cursor()
cur.execute('''SELECT * FROM Client''')
d = cur.fetchone()
for field in d:
print field
If I look at cur.fetchone():
'\xf0\xf1\xe0\xf8', '\xe0\xe9\xe0\xe3'
Output:
αΘαπ
2001
εδßΘ
αΘ°σ

If either of נסאר or איאד is meaningful, then try:
field.decode('cp1255')
Google Translate suggests this might correspond to a person named Iyad Nassar.

try use:
field.encode('utf-8')

Python stores " ó " as " ó "

I use scrapy to take information from a website that according to w3 validator is utf-8..
My python project has
# -*- coding: utf-8 -*-
I receive some names like López J and when I print it, it shows fine...
But when I want to store it into the mysql I get some error about ascii not being able to encode blah blah blah...
If I use .encode ('ascii', 'ignore') i get: Lpez J
If I use .encode ('ascii', 'replace') i get: López J
if I use .encode ('utf-8') i get: López J
What should I do?
I'm in a big trouble here :'(

When you connect to the database use charset='utf-8', use_unicode=True with other keywords to connect() method. This should make the dababase accept and return unicode values, so you don't have to (and shouldn't) encode them manually.
Example:
>>> import MySQLdb
>>> conn = MySQLdb.connect(... , use_unicode=True, charset='utf8')
>>> cur = conn.cursor()
>>> cur.execute('CREATE TABLE testing(x VARCHAR(20))')
0L
>>> cur.execute('INSERT INTO testing values(%s)', ('López J',))
1L
>>> cur.execute('SELECT * FROM testing')
1L
>>> print cur.fetchall()[0][0]
López J

Check your server, database, table, column and connection character sets.
As a quick test, try executing
SET NAMES 'utf8';
immediately after connecting.

python to mssql encoding problem

Greetings
By using pymssql library, I want to write data to a MSSQL database however I encounter encoding issues. Here is my sample code to write to the DB:
# -*- coding: utf-8 -*-
import _mssql
....
Connection info data here
....
def mssql_connect():
return _mssql.connect(server=HOST, user=USERNAME, password=PASS, database=DB, charset="utf-8")
con = mssql_connect()
INSERT_EX_SQL = "INSERT INTO myDatabsae (Id, ProgramName, ProgramDetail) VALUES (1, 'Test Characters ÜŞiçÇÖö', 'löşüIIğĞü');"
con.execute_non_query(INSERT_EX_SQL)
con.close()
Sadly the data that was written to DB is corrupted:
The Collacation of my mssql db is: Turkish_CI_AS
How can this be solved?

Here is a possible solution:
The key is INSERT_EX_SQ.encode('your language encoder').
Try this instead:
con.execute_non_query(INSERT_EX_SQ.encode('your language encoder'))