Develop Reference

MySQL to JSON using Python

I'm trying to develop a (really) simple server who an iOS app will interrogate. The Python script has to connect to the MySQL database and return data in JSON format. I can not achieve that it works also with special characters like è or é. This is a short and simplified version of my code with a lot of debugging printing inside...
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import MySQLdb
import json
print ("Content-Type: application/json; charset=utf-8\n\n")
db = MySQLdb.connect("localhost","root","******","*******" )
cursor = db.cursor()
sql = "SELECT * FROM places WHERE name IN (\"Palazzina Majani\")"
try:
cursor.execute(sql)
num_fields = len(cursor.description)
field_names = [i[0] for i in cursor.description]
results = cursor.fetchall()
print ("------------------results:")
print (results)
output_json = []
for row in results:
output_json.append(dict(zip(field_names,row)))
print ("------------------output_json:")
print (output_json)
output = json.dumps(output_json, ensure_ascii=False)
print ("------------------output:")
print (output)
except:
print ("Error")
db.close()
And this is what I get with terminal and also with browser:
Content-Type: application/json; charset=utf-8
------------------results:
(('Palazzina Majani', 'kasj \xe8.\xe9', 'palazzina_majani'),)
------------------output_json:
[{'imageName': 'palazzina_majani', 'name': 'Palazzina Majani', 'description': 'kasj \xe8.\xe9'}]
------------------output:
[{"imageName": "palazzina_majani", "name": "Palazzina Majani", "description": "kasj ?.?"}]
How can I manage those special characters (and the mainly used from latin-1)?
What if I simply replace single quotes with double quotes from "output_json" insted of using json.dumps?
Thank you!
SOLUTION
As Parfait said in the comments, passing charset='utf8' inside connect() solved the problem!

pyodbc: How to store Chinese characters in a SQL database?

I'm trying to store chinese text in a Microsoft SQL database.
My column type is nvarchar. And if I run this query directly in SSMS the results are saved correctly:
insert into xtemp (ind, desp)
values (1, N'人大花来民北村分社訃実真属葉')
But if I try to do it from python code, database just stores it as "????????????"
Can anyone point me in the right direction?
This is my python code:
connectionString1 = 'Driver={SQL Server};Server=x.database.windows.net,1433;Database=x;Uid=x;Pwd=x;Connection Timeout=30;Encrypt=yes;CHARSET=UTF8;'
connection1 = pyodbc.connect(connectionString1)
cursor1 = connection1.cursor()
q1 = '''
insert into xtemp (ind, desp)
values (2, N'人大花来民北村分社訃実真属葉')
'''
cursor1.execute(q1)
connection1.commit()
print("done")
cursor1.close()
connection1.close()

Try adding # -*- coding: utf-8 -*- at the beginning of your file, and also change
q1 = '''
to
q1 = u'''
This should enable Unicode support in your script and properly encode Chinese characters when sending then to the database engine.

portugese characters in DBsqlite and python parsing not recognised

I got a database in DBsqlite.in this DBsqlite database I have have a records containing portugese text like "Hiper-radiação simétrica periocular bem delimitada, homogênea."
and the characters like ç ã é ê don't parse right in my python script.
While normal english text is doing it perfectly.
In my terminal window (I use a mac) the
I know it has something do to with the encoding. but the code still doesn't recognise portugese.
my sample code:
# -*- coding: UTF-8 -*-
import xml.etree.ElementTree as ET
import sqlite3
#open a database connection to the database translateDB.sqlite
conn = sqlite3.connect('translateDB.sqlite')
#prepare a cursor object using cursus() method
cursor = conn.cursor()
#test input
# this doesn't work
text = ('Hiper-radiação simétrica periocular bem delimitada, homogênea')
# this does work in english
#text = ('Well delimited, homogeneous symmetric periocular hyper- radiation.')
# Execute SQL query using execute() method.
cursor.execute('SELECT * FROM translate WHERE L2_portugese=?', (text,))
# Fetch a single row using fetchone() method and display it.
print cursor.fetchone()
# Disconnect from server
conn.close()
any tips & tricks are greatly appreciated. Ron

Python stores " ó " as " ó "

I use scrapy to take information from a website that according to w3 validator is utf-8..
My python project has
# -*- coding: utf-8 -*-
I receive some names like López J and when I print it, it shows fine...
But when I want to store it into the mysql I get some error about ascii not being able to encode blah blah blah...
If I use .encode ('ascii', 'ignore') i get: Lpez J
If I use .encode ('ascii', 'replace') i get: López J
if I use .encode ('utf-8') i get: López J
What should I do?
I'm in a big trouble here :'(

When you connect to the database use charset='utf-8', use_unicode=True with other keywords to connect() method. This should make the dababase accept and return unicode values, so you don't have to (and shouldn't) encode them manually.
Example:
>>> import MySQLdb
>>> conn = MySQLdb.connect(... , use_unicode=True, charset='utf8')
>>> cur = conn.cursor()
>>> cur.execute('CREATE TABLE testing(x VARCHAR(20))')
0L
>>> cur.execute('INSERT INTO testing values(%s)', ('López J',))
1L
>>> cur.execute('SELECT * FROM testing')
1L
>>> print cur.fetchall()[0][0]
López J

Check your server, database, table, column and connection character sets.
As a quick test, try executing
SET NAMES 'utf8';
immediately after connecting.

python to mssql encoding problem

Greetings
By using pymssql library, I want to write data to a MSSQL database however I encounter encoding issues. Here is my sample code to write to the DB:
# -*- coding: utf-8 -*-
import _mssql
....
Connection info data here
....
def mssql_connect():
return _mssql.connect(server=HOST, user=USERNAME, password=PASS, database=DB, charset="utf-8")
con = mssql_connect()
INSERT_EX_SQL = "INSERT INTO myDatabsae (Id, ProgramName, ProgramDetail) VALUES (1, 'Test Characters ÜŞiçÇÖö', 'löşüIIğĞü');"
con.execute_non_query(INSERT_EX_SQL)
con.close()
Sadly the data that was written to DB is corrupted:
The Collacation of my mssql db is: Turkish_CI_AS
How can this be solved?

Here is a possible solution:
The key is INSERT_EX_SQ.encode('your language encoder').
Try this instead:
con.execute_non_query(INSERT_EX_SQ.encode('your language encoder'))