I have set up a CloudSQL instance that I am attempting to use with my Django app on AppEngine. I've confirmed that the server is set to use utf8mb4 character set via the CloudSQL console for my database:
utf8mb4 utf8mb4_unicode_ci
If I connect directly with the mysql cli, I can successfully insert and read emojis. However, if I insert the same emoji characters through the Django admin it's just inserted as "? ? ? ?".
I attempted to ensure the MySQLdb-python client is using utf8mb4 with:
'ENGINE': 'django.db.backends.mysql',
...
'OPTIONS': {
'charset': "utf8mb4",
}
But this causes me to receive the following error on AppEngine:
(2019, "Can't initialize character set utf8mb4 (path: /usr/local/mysql/share/charsets/)")
My app.yaml is using the "latest" MySQLdb library:
libraries:
- name: MySQLdb
version: "latest"
I just chatted with google and got everything working for our instance!
The standard way to get utf8mb4 working in Django is to specify it as DATABASES['default']['OPTIONS'] in settings.py, like this:
'OPTIONS': {'charset': 'utf8mb4'},
This causes an OperationalError in App Engine, on MySQLdb 1.2.4b4 / 1.2.4 / 1.2.5; which apparently means that the MySQL C client google is compiling against is missing the utf8mb4 character set.
Remove this OPTIONS setting.
The workaround is to manually call SET NAMES; edit lib/django/db/backends/mysql/base.py and add a conn.query("SET NAMES utf8mb4") line into DatabaseWrapper.get_new_connection, so it looks like this:
def get_new_connection(self, conn_params):
conn = Database.connect(**conn_params)
conn.encoders[SafeText] = conn.encoders[six.text_type]
conn.encoders[SafeBytes] = conn.encoders[bytes]
conn.query("SET NAMES utf8mb4")
return conn
Make sure that you also have utf8mb4 enabled on the backend. The migration commands in the App Engine Django tutorial result in a Cloud SQL instance configured for utf8. I needed to run these commands to enable utf8mb4 on the two tables:
ALTER TABLE polls_question CONVERT TO CHARACTER SET utf8mb4;
ALTER TABLE polls_choice CONVERT TO CHARACTER SET utf8mb4;
Related
I'm using the mysql-connector-python library to connect and write to a MySQL 5.7 db. I've set the encoding to utf8mb4 with cursor.execute('SET CHARACTERS SET utf8mb4'), and even included it in my connect settings:
import mysql.connector
from mysql.connector import Error
sg_titles_db_settings = {
'user': <user>,
'password': <password>,
'host': <host>,
'port': <port>,
'database': <db>,
'charset': 'utf8'
}
def get_mysql_connection():
try:
db_connection = mysql.connector.connect(**sg_titles_db_settings)
return db_connection
except Error as e:
print("Error: ", e)
return False
But any non-latin (non-english letters), such as any eastern european alpha character or special symbols (♬ for example), is inserted as ?.
Here's the error I receive if I don't change the encoding:
1366 (HY000): Incorrect string value: '\xD0\x9E\xD1\x82\xD0\xB2...' for column...
I don't understand what I need to do in order to resolve this issue. Every article I stumble upon doesn't seem to help.
Thanks in advance!
That's strange that it's not working for you. I have a project where cyrillic worked for me and I had something roughly like:
import MySQLdb
db = MySQLdb.connect(host='', user='', passwd='', db='', use_unicode=True)
cur = db.cursor()
cur.execute('SET NAMES utf8mb4')
cur.execute("SET CHARACTER SET utf8mb4")
cur.execute("SET character_set_connection=utf8mb4")
# Actual database stuff goes here
db.commit()
cur.close()
db.close()
I notice you are using a different module than the one I typically use. Have you tried other mysql modules like MySQLdb?
In the MySQL server run this command SET character_set_results=utf8;, that should fix it. However, closing the server may not persist that change.
I'm working with docker and it does not persist. The only way to persist the encoding change is to include it in the docker-compose.yml file:
services:
<db name>:
environment:
LANG: C.UTF-8
I tried the exact same code as found here...
https://github.com/TomMalkin/SimQLe
I am not sure how to connect to mysql database.
from simqle import ConnectionManager
cm = ConnectionManager("connections.yaml")
sql = "SELECT name, age FROM people WHERE category = :category"
params = {"category": 5}
result = cm.recordset(con_name="my-database", sql=sql, params=params)
Getting an error:
UnknownConnectionError: Unknown connection my-database
This is how I can connect to mysql database from command prompt.
mysql -h 172.31.84.39 -udba -pXXXX -P 3392
How do I write the connection string?
I usually use sqlalchemy to connect mysql database. I have readed the document of SimQLe which you are using. In SimQLe document,
cm = ConnectionManager("connections.yaml")
is the way to connect to database and you should put your login param in this yaml file called connections.yaml.
Here is the offical document simple:
https://github.com/TomMalkin/SimQLe#the-connectionsyaml-file
connections:
# The name of the connection - this is what will be used in your project
# to reference this connection.
- name: my-sql-server-database
driver: mssql+pyodbc:///?odbc_connect=
connection: DRIVER={SQL Server};UID=<username>;PWD=<password>;SERVER=<my-server>
# some odbc connections require urls to be escaped, this is managed by
# setting url_escaped = true:
url_escape: true
# File based databases like sqlite are slightly different - the driver
# is very simple.
- name: my-sqlite-database
driver: sqlite:///
# put a leading '/' before the connection for an absolute path, or omit
# if it's relative to the project path
connection: databases/my-database.db
# This connection will be used if no name is given if the default
# parameter is used:
default: true
Maybe you should change some params in here:
driver: mssql+pyodbc:///?odbc_connect=
connection: DRIVER={SQL Server};UID=<username>;PWD=<password>;SERVER=<my-server>
And from the document, it says that SimQle is built on SQLAlchemy,
SimQLe is built on top of the fantastic SQLAlchemy library.
maybe you can use the SQLAlchemy's login params to connect the database in SimQLe. Such like this:
mysql+pymysql://<username>:<password>#<host>/<dbname>[?<options>]
Changed to:
driver: mysql+pymysql://
connection: <username>:<password>#<host>/<dbname>[?<options>]
Offical documents:
https://simqle.readthedocs.io/en/latest/
https://docs.sqlalchemy.org/en/14/dialects/mysql.html#module-sqlalchemy.dialects.mysql.pymysql
https://docs.sqlalchemy.org/en/14/dialects/mssql.html#module-sqlalchemy.dialects.mssql.pyodbc
While attempting to connect to a PosgresQL database I got an authentication error. The issue turned out to be that Peewee's approach to parse the connection URL breaks when URL-special sensitive characters are used in the password e.g.
The following will not work with an authentication error:
from peewee import *
from playhouse.db_url import connect
url = "postgresql://host:port/database_name?user=username&password=123#456"
# or even
url = "postgresql://host:port/database_name?user=username&password=123%456"
db = connect(url)
However this will work:
from psycopg2 import connect
db = connect(url)
I am in a corporate setting so the passwords have strong requirements including the mandatory use of special characters. Is there a way to encode these characters so that they will work with playhouse.db_url's connect implementation?
I have a Flask app with using flask_sqlalchemy:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config.from_pyfile(filename='settings.py', silent=True)
db = SQLAlchemy(app=app)
I want connect to same database from daemon. In daemon I just import db and use db.engine.execute for SQLAlchemy queries.
But when daemon starts main app can't connect to database.
In log I see that:
fdb.fbcore.DatabaseError: ('Error while connecting to database:\n- SQLCODE:
-902\n- I/O error during "lock" operation for file "main.fdb"\n- Database
already opened with engine instance, incompatible with current', -902,
335544344)
I trying use isolation level:
from fdb.fbcore import ISOLATION_LEVEL_READ_COMMITED_LEGACY
class TPBAlchemy(SQLAlchemy):
def apply_driver_hacks(self, app_, info, options):
if 'isolation_level' not in options:
options['isolation_level'] = ISOLATION_LEVEL_READ_COMMITED_LEGACY
return super(TPBAlchemy, self).apply_driver_hacks(app_, info, options)
And replace this:
db = SQLAlchemy()
To:
db = TPBAlchemy()
But this only make another error:
TypeError: Invalid argument(s) 'isolation_level' sent to create_engine(),
using configuration FBDialect_fdb/QueuePool/Engine. Please check that the
keyword arguments are appropriate for this combination of components.
I would appreciate the full example to address my issue.
Your connection string is for an embedded database. You're only allowed to have one 'connection' to an embedded database at a time.
If you have the Loopback provider enabled you can change your connection string to something like:
localhost:/var/www/main.fdb
or if you have the Remote provider enabled you will have to access your database from another remote node, and assuming your Firebird server lives on 192.168.1.100 change your connection string to
192.168.1.100:/var/www/main.fdb
If you're intending to use the Engine12 provider (the embedded provider), then you have to stop whatever is already connected to that database because you just can't do two simultaneously users with this provider.
Also, try to set up some database aliases so you aren't specifying a database explicitly like that. In Firebird 3.0.3 check out databases.conf, where you can do something like:
mydatabasealias=/var/www/main.fdb
and your connection string would now be mydatabasealias instead of the whole path.
I am getting problem in Django project setting with POSTGRESQL.
Here is my setting.py database setting
DATABASES = {
'default':{
'ENGINE':'django.db.backends.postgresql_psycopg2', # '.postgresql_psycopg2', '.mysql', or '.oracle'
'NAME':'indivo', # Required to be non-empty string
'USER':'indivo', # Required to be non-empty string
'PASSWORD':'ritvik',
'HOST':'', # Set to empty string for localhost.
'PORT':'', # Set to empty string for default.
},
}
Now in postgres backend what I have done is .
rohit#rohit-desktop:~$ sudo su - postgres
postgres#rohit-desktop:~$ createuser --superuser indivo # create a super user indivo
postgres#rohit-desktop:~$ psql # open psql terminal
psql (9.1.8)
Type "help" for help.
postgres=# \password indivo # set the password ritvik
Enter new password:
Enter it again:
postgres=# \q #logout
postgres#rohit-desktop:~$ createdb -U indivo -O indivo indivo #create db indivo
Unfortunately when i am trying to syncdb I am getting the error .
psycopg2.OperationalError: FATAL: Peer authentication failed for user "indivo"
Please help me out what might I am doing wrong here .
I have similar problem and solved it with this answer by adding localhost to the database HOST settings in settings.py, so your database settings should look like this:
DATABASES = {
'default':{
'ENGINE':'django.db.backends.postgresql_psycopg2', # '.postgresql_psycopg2', '.mysql', or '.oracle'
'NAME':'indivo', # Required to be non-empty string
'USER':'indivo', # Required to be non-empty string
'PASSWORD':'ritvik',
'HOST':'localhost', # <- Changed from empty string to localhost
'PORT':'', # Set to empty string for default.
},
}
By default in many Linux distros, client authentication is set to "peer" for Unix socket connections to the DB. This is set in the pg_hba.conf config file for postgresql. The psycopg2 python library documentation states:
- *host*: database host address (defaults to UNIX socket if not provided)
So if you leave the host option blank, your script will try to connect and use the Unix username:password for authentication. To fix, you can either:
set the "host" option explicitly to 127.0.0.1 into the config code you pasted into your question.
Modify the pg_hba.conf file to use md5 for socket connections so it uses usernames and passwords stored in the postrgres DB user store (not recommended as this may break authentication for system users)
You need to set pg_hba.conf to use md5 authentication for the user, db and source IP of interest. See the client authentication chapter of the documentation.
Search for pg_hba.conf on Stack Overflow for tons more information.
I too was facing similar problem during setting up postgresql database with Django project.
My System is running RHEL CentOS 7 with postgresql version 10. I search a lot but couldnot find actually what is happening until I saw log at /var/lib/pqsql/10/data/log/postgresql*.log
2017-11-10 00:31:40.499 IST [14129] DETAIL: Connection matched pg_hba.conf line 85: "host all all ::1/128 ident"
So I just changed that particular line in file /var/lib/pgsql/10/data/pg_hba.conf :
from
host all all ::1/128 ident
to
host all all ::1/128 md5
with that I am able to create database & table using manage.py migrate
Thanks for all who has helped me to find this solution. especially #juliocesar and official documentation for postgresql client authentication