I've just spent a week on the problems recorded in this question: https://stackoverflow.com/questions/21315427/why-does-the-ca-information-need-to-be-in-a-tuple-for-mysqldb
Have now boiled it down to one problem. Here's a script that connects to the MySQL server I have on Amazon RDS:
#! /usr/bin/env python
import MySQLdb
ssl = ({'ca': '/home/James/Downloads/mysql-ssl-ca-cert-copy.pem'},)
conn = MySQLdb.connect(host='db.doopdoop.eu-west-n.rds.amazonaws.com', user='user', passwd='pass', ssl=ssl)
cursor = conn.cursor()
cursor.execute("SHOW STATUS LIKE 'Ssl_Cipher'")
print cursor.fetchone()
This gives me back ('Ssl_cipher', ''), which I gather means the connection is not encrypted. Have also tried this using Django's manage.py shell function, and got the same result. Why, then, am I getting no exceptions? Data is flowing, certainly, but apparently the security is just being ignored. Any help on where I'm going wrong here would be appreciated.
I have tried updating MySQL-Python to 1.2.5 with no success.
Possible workaround for your issue: Use a default file. It will look something like this:
[mysql]
host = db.doopdoop.eu-west-n.rds.amazonaws.com
user = user
password = pass
ssl-ca = /home/James/Downloads/mysql-ssl-ca-cert-copy.pem
Then change your connect() call to:
conn = MySQLdb.connect(read_default_file=options_path)
where options_path is the path to the file above. This also keeps authentication data out of your code.
Django settings will look like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
Ref: https://docs.djangoproject.com/en/dev/ref/databases/#connecting-to-the-database
Related
I'm using the mysql-connector-python library to connect and write to a MySQL 5.7 db. I've set the encoding to utf8mb4 with cursor.execute('SET CHARACTERS SET utf8mb4'), and even included it in my connect settings:
import mysql.connector
from mysql.connector import Error
sg_titles_db_settings = {
'user': <user>,
'password': <password>,
'host': <host>,
'port': <port>,
'database': <db>,
'charset': 'utf8'
}
def get_mysql_connection():
try:
db_connection = mysql.connector.connect(**sg_titles_db_settings)
return db_connection
except Error as e:
print("Error: ", e)
return False
But any non-latin (non-english letters), such as any eastern european alpha character or special symbols (♬ for example), is inserted as ?.
Here's the error I receive if I don't change the encoding:
1366 (HY000): Incorrect string value: '\xD0\x9E\xD1\x82\xD0\xB2...' for column...
I don't understand what I need to do in order to resolve this issue. Every article I stumble upon doesn't seem to help.
Thanks in advance!
That's strange that it's not working for you. I have a project where cyrillic worked for me and I had something roughly like:
import MySQLdb
db = MySQLdb.connect(host='', user='', passwd='', db='', use_unicode=True)
cur = db.cursor()
cur.execute('SET NAMES utf8mb4')
cur.execute("SET CHARACTER SET utf8mb4")
cur.execute("SET character_set_connection=utf8mb4")
# Actual database stuff goes here
db.commit()
cur.close()
db.close()
I notice you are using a different module than the one I typically use. Have you tried other mysql modules like MySQLdb?
In the MySQL server run this command SET character_set_results=utf8;, that should fix it. However, closing the server may not persist that change.
I'm working with docker and it does not persist. The only way to persist the encoding change is to include it in the docker-compose.yml file:
services:
<db name>:
environment:
LANG: C.UTF-8
I’m currently connecting to my database through my hard coding link in my config file, but I want to be able to dynamically generate this connection so I will be able to use multiple connections in the future.
SQLALCHEMY_DATABASE_URI = ‘postgresql+psycopg2://<blah>/database’
You could put the database connection parameters in an external file, eg connections_settings.ini
[credentials]
host=localhost
dbname=test
username=me
password=secret
and then read them with the configparser module, and create the connection url with string interpolation
import configparser
config = configparser.ConfigParser('connection_settings')['credentials']
connection_settings = {
'host': config['host'],
'dbname': config['dbname'],
'user': config['username'],
'password': config['password']
}
SQLALCHEMY_DATABASE_URI = f'postgresql+psycopg2://{host}/{dbname}'
I have a couple of services which query objects from the database.
Event.objects.filter
Connection.objects.filter
and other methods to retrieve different objects from MySQL database.
I come from JVM background, and I know that to setup a JDBC connection, you need a connector. A programmer can open a connection, query the database and close the connection. A programmer also can use Hibernate, which handles connection according to the configuration. Also it is possible to use pooled connections, so connections are not closed and removed, but stored in the pool untill they are needed.
However, I checked my teams Python Django code, and I did not find how db connection is configured. The only thing I got is which does not configure connections.
# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases
try:
import database_password
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': "mydb",
'USER': 'user',
'PASSWORD': database_password.password,
'HOST': '10.138.67.149',
'PORT': '3306'
}
}
Each thread maintains its own connection. See the docs for full details.
PostgreSQL + PgBouncer (connection pooler) + Django is a common setup. I'm not sure whether there's a similar connection pooler you could use with MySQL.
I have just setup a Django app on aws beanstalk with PostgreSQL 9.4.7. My database settings are as follows;
if 'RDS_DB_NAME' in os.environ:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
I have installed psycopg2.
AWS application installation completed successfully however when i opened by index page I received the following error.
Exception Type: RuntimeError
Exception Value:
Error creating new content types. Please make sure contenttypes is migrated before trying to migrate apps individually.
Further digging into the exception page I found this error
**Variable**
my_list
**Value**
Error in formatting: OperationalError: fe_sendauth: no password supplied
**request**
<WSGIRequest: GET '/new/'>
my_list is the variable defined in the index page view.
Further down the page I can see the DB values are updated correctly.
to check if there was some issue with the password I tried accessing aws RDS via pgadmin and was able to do so successfully so no issues there.
I have tried a lot of things but was not able to find a solution so need some assistance please.
EDIT: I have also checked the tables after connecting with pgadmin and can confirm that all the tables have been created through migrate command (created in aws .config file) on the aws database. Just thought of sharing this in case this is helpful.
EDIT 2: so on the error page if I go down to the settings section I can see both id and password reflecting (pass in ******** form). However in the Traceback section I can see that password is None but rest of the info is there and this error is;
/opt/python/run/venv/lib64/python2.7/site-packages/psycopg2/__init__.py in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
EDIT3: so I get everything from the db except for the password and I checked my env file found in /opt/python/current on aws ssh instance and found it had all the export settings correctly listed including the password.
To be more explicit, below I have mentioned the exception detail I get on traceback on the web page detail under
/opt/python/run/venv/lib64/python2.7/site-packages/psycopg2/init.py
in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
**Variable Value**
database 'xxx' - This is correct
items [('dbname', 'xxx'), This is correct
('user', 'xxxxx'), This is correct
('host', 'xxxxxxx.rds.amazonaws.com'), This is correct
('port', '5432')]
k 'port'
kwargs {}
connection_factory None
dsn 'dbname=xxxx user=xxxx host=xxxx.rds.amazonaws.com port=5432' This is correct
host 'xxxx.rds.amazonaws.com' This is correct
user 'xxxxx' This is correct
v '5432' async False
**password None** **THIS IT NOT CORRECT**
port '5432'
cursor_factory None
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-rds.html
In your case I woould try going for
Configure your Amazon RDS DB security group to allow access from the Amazon EC2 security group used by your Elastic Beanstalk
application
part
Update:
don't forget to backslash variables values like RDS_PASSWORD in os environment. Eg export RDS_PASSWORD="pa$$word" should become export RDS_PASSWORD="pa\$\$word"
to do this go to;
on ssh ec2 instance and go to file
/opt/python/current
. This dir has one app directory which holds the django app and an env file. "nano env" and it shows lot of export statements go to the RDS_PASSWORD line and add the blackslash pass to it.
export RDS_PASSWORD="pa\$\$word". Save the file.
Then restart the app on aws and it works like a charm.
I use Python 3.4 from the Anaconda distribution. Within this distribution, I found the pymysql library to connect to an existing MySQL database, which is located on another computer.
import pymysql
config = {
'user': 'my_user',
'passwd': 'my_passwd',
'host': 'my_host',
'port': my_port
}
try:
cnx = pymysql.connect(**config)
except pymysql.err.OperationalError :
sys.exit("Invalid Input: Wrong username/database or password")
I now want to write test code for my application, in which I want to create a very small database at the setUp of every test case, preferably in memory. However, when I try this out of the blue with pymysql, it cannot make a connection.
def setUp(self):
config = {
'user': 'test_user',
'passwd': 'test_passwd',
'host': 'localhost'
}
cnx = pymysql.connect(**config)
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I have been googling around, and found some things about SQLite and MySQLdb. I have the following questions:
Is sqlite3 or MySQLdb suitable for creating quickly a database in memory?
How do I install MySQLdb within the Anaconda package?
Is there an example of a test database, created in the setUp? Is this even a good idea?
I do not have a MySQL server running locally on my computer.
You can mock a mysql db using testing.mysqld (pip install testing.mysqld)
Due to some noisy error logs that crop up, I like this setup when testing:
import testing.mysqld
from sqlalchemy import create_engine
# prevent generating brand new db every time. Speeds up tests.
MYSQLD_FACTORY = testing.mysqld.MysqldFactory(cache_initialized_db=True, port=7531)
def tearDownModule():
"""Tear down databases after test script has run.
https://docs.python.org/3/library/unittest.html#setupclass-and-teardownclass
"""
MYSQLD_FACTORY.clear_cache()
class TestWhatever(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.mysql = MYSQLD_FACTORY()
cls.db_conn = create_engine(cls.mysql.url()).connect()
def setUp(self):
self.mysql.start()
self.db_conn.execute("""CREATE TABLE `foo` (blah)""")
def tearDown(self):
self.db_conn.execute("DROP TABLE foo")
#classmethod
def tearDownClass(cls):
cls.mysql.stop() # from source code we can see this kills the pid
def test_something(self):
# something useful
Both pymysql, MySQLdb, and sqlite will want a real database to connect too.
If you want just to test your code, you should just mock the pymysql module on the module you want to test, and use it accordingly
(in your test code: you can setup the mock object to return hardcoded results to predefined SQL statements)
Check the documentation on native Python mocking library at:
https://docs.python.org/3/library/unittest.mock.html
Or, for Python 2:
https://pypi.python.org/pypi/mock