I'm using the mysql-connector-python library to connect and write to a MySQL 5.7 db. I've set the encoding to utf8mb4 with cursor.execute('SET CHARACTERS SET utf8mb4'), and even included it in my connect settings:
import mysql.connector
from mysql.connector import Error
sg_titles_db_settings = {
'user': <user>,
'password': <password>,
'host': <host>,
'port': <port>,
'database': <db>,
'charset': 'utf8'
}
def get_mysql_connection():
try:
db_connection = mysql.connector.connect(**sg_titles_db_settings)
return db_connection
except Error as e:
print("Error: ", e)
return False
But any non-latin (non-english letters), such as any eastern european alpha character or special symbols (♬ for example), is inserted as ?.
Here's the error I receive if I don't change the encoding:
1366 (HY000): Incorrect string value: '\xD0\x9E\xD1\x82\xD0\xB2...' for column...
I don't understand what I need to do in order to resolve this issue. Every article I stumble upon doesn't seem to help.
Thanks in advance!
That's strange that it's not working for you. I have a project where cyrillic worked for me and I had something roughly like:
import MySQLdb
db = MySQLdb.connect(host='', user='', passwd='', db='', use_unicode=True)
cur = db.cursor()
cur.execute('SET NAMES utf8mb4')
cur.execute("SET CHARACTER SET utf8mb4")
cur.execute("SET character_set_connection=utf8mb4")
# Actual database stuff goes here
db.commit()
cur.close()
db.close()
I notice you are using a different module than the one I typically use. Have you tried other mysql modules like MySQLdb?
In the MySQL server run this command SET character_set_results=utf8;, that should fix it. However, closing the server may not persist that change.
I'm working with docker and it does not persist. The only way to persist the encoding change is to include it in the docker-compose.yml file:
services:
<db name>:
environment:
LANG: C.UTF-8
Related
I tried to write a function to drop database :
def deleteDb(self, dbName: str):
conn = psycopg2.connect(dbname="postgres", user="postgres")
conn.autocommit = True
curs = conn.cursor()
curs.execute("DROP DATABASE {};".format(dbName))
curs.close()
conn.close()
When I try to test it with an existing db :
def test_deleteDb(self):
self.deleteDb("dbTest")
I get this error :
"psycopg2.errors.InvalidCatalogName: database "dbtest" does not exist"
I tried to play with the isolation level, to drop all the connections to the database and to connect directly to the database but it did not work
Remember to put quotes around identifiers that contain upper-case letters:
curs.execute('DROP DATABASE "{}";'.format(dbName))
Note that string substitution into SQL-statements is generally a bad idea beause it is vulnerable to SQL-injection.
after connecting the database server I want to loop certain databases.
The names of the databases are in the list dbList. These names were taken from different customer files with lines like
name_database='database_1' or
name_database='database-2'.
The shortened dbList looks like ["'database_1'", "'database-2'"].
If I for-loop the database list (removing the single quotes from the database name, do some action) all is fine except in the name of the database is a hyphen. Than the program stops with the error
1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '-2' at line 1
That is exactly where I call
dbConn.database=dbName
I tried various "dbConn.database=..." ways; nothing worked.
The only way it works is to not use the loop-variable; just write dbConn.database='database-2'
But this is not my intention.
coding extract is:
import mysql.connector
...
dbConn = connect2dbServer ( loginParameterFile, fileSection )
authentication for DB server access in general (no DB selected)
connect2dbServer fetches user name, password, server IP, and port number
connect2dbServer returns the DB server connector.
...
for dbName in dbList:
dbName = removeChar ("'", dbName)
dbConn.database=dbName # connecting with (next) DB
# ...
select round (sum(size)/1024/1024/1024, 2) as 'Used HDD space (GB)' from oc_filecache where path in ('cache', 'documents', 'files')
... # and some more queries
import re
def removeChar (char, string):
stripped=''
stripped = re.sub(char, "", string)
return stripped
OS macOS 10.12
mysql client 5.7
mysql-connector 2.1.5
python 3.6
If you need more information please ask for.
Thank you very much for your help in advance.
I have set up a CloudSQL instance that I am attempting to use with my Django app on AppEngine. I've confirmed that the server is set to use utf8mb4 character set via the CloudSQL console for my database:
utf8mb4 utf8mb4_unicode_ci
If I connect directly with the mysql cli, I can successfully insert and read emojis. However, if I insert the same emoji characters through the Django admin it's just inserted as "? ? ? ?".
I attempted to ensure the MySQLdb-python client is using utf8mb4 with:
'ENGINE': 'django.db.backends.mysql',
...
'OPTIONS': {
'charset': "utf8mb4",
}
But this causes me to receive the following error on AppEngine:
(2019, "Can't initialize character set utf8mb4 (path: /usr/local/mysql/share/charsets/)")
My app.yaml is using the "latest" MySQLdb library:
libraries:
- name: MySQLdb
version: "latest"
I just chatted with google and got everything working for our instance!
The standard way to get utf8mb4 working in Django is to specify it as DATABASES['default']['OPTIONS'] in settings.py, like this:
'OPTIONS': {'charset': 'utf8mb4'},
This causes an OperationalError in App Engine, on MySQLdb 1.2.4b4 / 1.2.4 / 1.2.5; which apparently means that the MySQL C client google is compiling against is missing the utf8mb4 character set.
Remove this OPTIONS setting.
The workaround is to manually call SET NAMES; edit lib/django/db/backends/mysql/base.py and add a conn.query("SET NAMES utf8mb4") line into DatabaseWrapper.get_new_connection, so it looks like this:
def get_new_connection(self, conn_params):
conn = Database.connect(**conn_params)
conn.encoders[SafeText] = conn.encoders[six.text_type]
conn.encoders[SafeBytes] = conn.encoders[bytes]
conn.query("SET NAMES utf8mb4")
return conn
Make sure that you also have utf8mb4 enabled on the backend. The migration commands in the App Engine Django tutorial result in a Cloud SQL instance configured for utf8. I needed to run these commands to enable utf8mb4 on the two tables:
ALTER TABLE polls_question CONVERT TO CHARACTER SET utf8mb4;
ALTER TABLE polls_choice CONVERT TO CHARACTER SET utf8mb4;
I have case :
import pymysql
conn = pymysql.connect(host='127.0.0.1', unix_socket='/opt/lampp/var/mysql/mysql.sock', user='root', passwd=None, db='test')
cur = conn.cursor()
cur.execute("test < /mypath/test.sql")
cur.close()
conn.close()
I always get error :
1064 , "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'test < /mypath/test.sql' at line 1"
I tried to use source and it still failed. Did you know why?
Thank you.
Your error message says that the MySQL server can't understand
test < /mypath/test.sql' at line 1
If you're a long time *nix user, it seems intuitive that you should be able to use commands like this to pass various sorts of data streams to various programs. But that's not the way the Python sql API (or most language-specific) sql APIs works.
You need to pass a valid SQL query to the execute() method in the API, so the API can pass it to the database server. A vaild query will be something like INSERT or CREATE TABLE.
Look, the server might be on a different host machine, so telling the server to read from /mypath/test.sql is very likely a meaningless instruction to that server. Even if it did understand it, it might say File test.sql not found.
The mysql(1) command line client software package can read commands from files. Is that what you want?
>>> import MySQLdb
>>> db = MySQLdb.connect(host = 'demodb', user = 'root', passwd = 'root', db = 'mydb')
>>> cur = db.cursor()
>>> cur.execute('select * from mytable')
>>> rows = cur.fetchall()
Install MySQL-Python package to use MySQLdb.
I've just spent a week on the problems recorded in this question: https://stackoverflow.com/questions/21315427/why-does-the-ca-information-need-to-be-in-a-tuple-for-mysqldb
Have now boiled it down to one problem. Here's a script that connects to the MySQL server I have on Amazon RDS:
#! /usr/bin/env python
import MySQLdb
ssl = ({'ca': '/home/James/Downloads/mysql-ssl-ca-cert-copy.pem'},)
conn = MySQLdb.connect(host='db.doopdoop.eu-west-n.rds.amazonaws.com', user='user', passwd='pass', ssl=ssl)
cursor = conn.cursor()
cursor.execute("SHOW STATUS LIKE 'Ssl_Cipher'")
print cursor.fetchone()
This gives me back ('Ssl_cipher', ''), which I gather means the connection is not encrypted. Have also tried this using Django's manage.py shell function, and got the same result. Why, then, am I getting no exceptions? Data is flowing, certainly, but apparently the security is just being ignored. Any help on where I'm going wrong here would be appreciated.
I have tried updating MySQL-Python to 1.2.5 with no success.
Possible workaround for your issue: Use a default file. It will look something like this:
[mysql]
host = db.doopdoop.eu-west-n.rds.amazonaws.com
user = user
password = pass
ssl-ca = /home/James/Downloads/mysql-ssl-ca-cert-copy.pem
Then change your connect() call to:
conn = MySQLdb.connect(read_default_file=options_path)
where options_path is the path to the file above. This also keeps authentication data out of your code.
Django settings will look like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
Ref: https://docs.djangoproject.com/en/dev/ref/databases/#connecting-to-the-database