I use Python 3.4 from the Anaconda distribution. Within this distribution, I found the pymysql library to connect to an existing MySQL database, which is located on another computer.
import pymysql
config = {
'user': 'my_user',
'passwd': 'my_passwd',
'host': 'my_host',
'port': my_port
}
try:
cnx = pymysql.connect(**config)
except pymysql.err.OperationalError :
sys.exit("Invalid Input: Wrong username/database or password")
I now want to write test code for my application, in which I want to create a very small database at the setUp of every test case, preferably in memory. However, when I try this out of the blue with pymysql, it cannot make a connection.
def setUp(self):
config = {
'user': 'test_user',
'passwd': 'test_passwd',
'host': 'localhost'
}
cnx = pymysql.connect(**config)
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I have been googling around, and found some things about SQLite and MySQLdb. I have the following questions:
Is sqlite3 or MySQLdb suitable for creating quickly a database in memory?
How do I install MySQLdb within the Anaconda package?
Is there an example of a test database, created in the setUp? Is this even a good idea?
I do not have a MySQL server running locally on my computer.
You can mock a mysql db using testing.mysqld (pip install testing.mysqld)
Due to some noisy error logs that crop up, I like this setup when testing:
import testing.mysqld
from sqlalchemy import create_engine
# prevent generating brand new db every time. Speeds up tests.
MYSQLD_FACTORY = testing.mysqld.MysqldFactory(cache_initialized_db=True, port=7531)
def tearDownModule():
"""Tear down databases after test script has run.
https://docs.python.org/3/library/unittest.html#setupclass-and-teardownclass
"""
MYSQLD_FACTORY.clear_cache()
class TestWhatever(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.mysql = MYSQLD_FACTORY()
cls.db_conn = create_engine(cls.mysql.url()).connect()
def setUp(self):
self.mysql.start()
self.db_conn.execute("""CREATE TABLE `foo` (blah)""")
def tearDown(self):
self.db_conn.execute("DROP TABLE foo")
#classmethod
def tearDownClass(cls):
cls.mysql.stop() # from source code we can see this kills the pid
def test_something(self):
# something useful
Both pymysql, MySQLdb, and sqlite will want a real database to connect too.
If you want just to test your code, you should just mock the pymysql module on the module you want to test, and use it accordingly
(in your test code: you can setup the mock object to return hardcoded results to predefined SQL statements)
Check the documentation on native Python mocking library at:
https://docs.python.org/3/library/unittest.mock.html
Or, for Python 2:
https://pypi.python.org/pypi/mock
Related
So I have been making websites using PHP with a MySQL database, PHPMyAdmin, and XAMPP. I am trying to switch from PHP to python. All of the tutorials seem to be using SQLite instead of MYSQL. As far as I understand, sqlite is serverless and cant hold certain data types like datetime, but I need datetime in my website. How would I connect to MySQL with a python Flask project or is there a different way I need to do this?
You need to use Client Library like PyMySQL
To install pymysql use:
pip install PyMySQL
And then use this function, it will return the DB object:
def make_connection():
try:
db = pymysql.connect(host='localhost',
user='root',
password='',
db='DatabaseName',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
except Exception as error:
print(error)
return db
As what #tirth-mehta mentioned, if you want to use connect without any ORM, you should a client library. and if you feel it's too painful to not forget to close connection in every function call, you could use a decorator like this:
DB_CONFIG = {'host':'127.0.0.1',
'database':'dbname',
'user':'root',
'password':''}
def connect(func):
def _connect(*args, **kwargs):
conn = pymysql.connect(**DB_CONFIG)
try:
rv = func(conn, *args, **kwargs)
except Exception as e:
print(e)
else:
conn.commit()
finally:
conn.close()
return rv
return _connect
I'm trying to unit test a class connection to a database. To avoid hardcoding a database, I'd like to assert that the mysql.connector.connect method is called with adequate values.
from mysql.connector import connect
from mysql.connector import Error
from discovery.database import Database
class MariaDatabase(Database):
def connect(self, username, password):
"""
Don't forget to close the connection !
:return: connection to the database
"""
try:
return connect(host=str(self.target),
database=self.db_name,
user=username,
password=password)
I've read documentation around mocks and similar problems (notably this one Python Unit Test : How to unit test the module which contains database operations?, which I thought would solve my problem, but mysql.connector.connect keeps being called instead of the mock).
I don't know what could I do to UTest this class
When running this code for connecting to a db through cmd - locally and on the actual server it works fine. But I have set it up on Jenkins and receive the error:
DatabaseError: file is encrypted or is not a database
It seems to be happening on this line:
self.cursor.execute(*args)
The database class is:
class DatabaseManager(object):
def __init__(self, db):
self.conn = sqlite3.connect(db)
self.cursor = self.conn.cursor()
def query(self, *args):
self.cursor.execute(*args)
self.conn.commit()
return self.cursor
def __del__(self):
self.conn.close()
The version of python sqlite3 and Command Line sqlite3 can be different. Create your database from the script i.e. code the DB initialization in the script rather than from CMD and it might solve the problem.
What is the value of *args ?? is it having the same values that you have when you run it on cmd and through jenkins, Did you check the path and location of DB related from jenkins location??
It's most probably a version mismatch between the SQLite CLI you're using and the one bundled with Python. You can have such mismatch on the same server, too. To be sure, you can use Python instead of the SQLite CLI to create your DB on the sever - provided you have all your SQLite init structure in path/to/your_sql.sql you can initialize path/to/your_database.db database with a script like:
import sqlite3
connection = sqlite3.connect("path/to/your_database.db")
cursor = connection.cursor()
with open("path/to/your_sql.sql", "r") as f:
cursor.execute(f.read())
connection.commit()
Then try to load that DB from your Jenkins job.
This is the code use to connect to our hive database , it ran fine a week back but now it seems to be be failing to even open a session and get a cursor to execute the queries. The issue was temporarily fixed when i explicitly added a cursor.close() method but now it's back again. I am unable to access the hive database using python
I have tried using pyhs2 and pyhive both the libraries fail to connect to the hive database. Nothing on the cluster has changed so far. what could the reason for this be ?
I know Hive is not a relational DB so the concept of cursors doesn't make sense but Is there any way the the Hive database remembers the cursors created using the pyhive library ?? if so how can i delete the currently unused cursors ??
Here is the code and exception it raises upon execution
from pyhive import hive
import contextlib
class Hive():
def __init__(self,host="[hostnamehere]",db="default",port="10000",auth="KERBEROS",kerberos_service_name="hive"):
self.host = host
self.db = db
self.port = port
self.auth = auth
self.kerberos_service_name = kerberos_service_name
def connect(self):
return(hive.connect(host=self.host, port=self.port, database=self.db, auth=self.auth, kerberos_service_name=self.kerberos_service_name))
def query_one(self,sql):
with contextlib.closing(self.connect()) as connection:
with contextlib.closing(connection.cursor()) as cursor:
cursor.execute(sql)
result = cursor.fetch_one()
cursor.close()
return(result)
if __name__ == "__main__":
connector = Hive()
print("running query")
print(connector.query_one("SELECT * FROM [tablenamehere]"))
raise OperationalError(response)
pyhive.exc.OperationalError: TOpenSessionResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:13:12', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:289', 'org.apache.hive.service.cli.CLIService:openSession:CLIService.java:199', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:319', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1257', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1242', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:562', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', 'java.lang.Thread:run:Thread.java:748', '*java.lang.RuntimeException:java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:15:2', 'org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:547', 'org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl.java:144', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:281', '*java.lang.RuntimeException:Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:21:6', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1566', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient::RetryingMetaStoreClient.java:92', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:138', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:110', 'org.apache.hadoop.hive.ql.metadata.Hive:createMetaStoreClient:Hive.java:3510', 'org.apache.hadoop.hive.ql.metadata.Hive:getMSC:Hive.java:3542', 'org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:528', '*java.lang.reflect.InvocationTargetException:null:25:4', 'sun.reflect.NativeConstructorAccessorImpl:newInstance0:NativeConstructorAccessorImpl.java:-2', 'sun.reflect.NativeConstructorAccessorImpl:newInstance:NativeConstructorAccessorImpl.java:62', 'sun.reflect.DelegatingConstructorAccessorImpl:newInstance:DelegatingConstructorAccessorImpl.java:45', 'java.lang.reflect.Constructor:newInstance:Constructor.java:423', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1564', '*org.apache.hadoop.hive.metastore.api.MetaException:GC overhead limit exceeded:30:4', 'org.apache.hadoop.hive.metastore.RetryingHMSHandler::RetryingHMSHandler.java:82', 'org.apache.hadoop.hive.metastore.RetryingHMSHandler:getProxy:RetryingHMSHandler.java:91', 'org.apache.hadoop.hive.metastore.HiveMetaStore:newRetryingHMSHandler:HiveMetaStore.java:6463', 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient::HiveMetaStoreClient.java:206', 'org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient::SessionHiveMetaStoreClient.java:76', '*java.lang.OutOfMemoryError:GC overhead limit exceeded:0:-1'], sqlState=None, errorCode=0, errorMessage='Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient'), serverProtocolVersion=7, sessionHandle=None, configuration=None)
I've just spent a week on the problems recorded in this question: https://stackoverflow.com/questions/21315427/why-does-the-ca-information-need-to-be-in-a-tuple-for-mysqldb
Have now boiled it down to one problem. Here's a script that connects to the MySQL server I have on Amazon RDS:
#! /usr/bin/env python
import MySQLdb
ssl = ({'ca': '/home/James/Downloads/mysql-ssl-ca-cert-copy.pem'},)
conn = MySQLdb.connect(host='db.doopdoop.eu-west-n.rds.amazonaws.com', user='user', passwd='pass', ssl=ssl)
cursor = conn.cursor()
cursor.execute("SHOW STATUS LIKE 'Ssl_Cipher'")
print cursor.fetchone()
This gives me back ('Ssl_cipher', ''), which I gather means the connection is not encrypted. Have also tried this using Django's manage.py shell function, and got the same result. Why, then, am I getting no exceptions? Data is flowing, certainly, but apparently the security is just being ignored. Any help on where I'm going wrong here would be appreciated.
I have tried updating MySQL-Python to 1.2.5 with no success.
Possible workaround for your issue: Use a default file. It will look something like this:
[mysql]
host = db.doopdoop.eu-west-n.rds.amazonaws.com
user = user
password = pass
ssl-ca = /home/James/Downloads/mysql-ssl-ca-cert-copy.pem
Then change your connect() call to:
conn = MySQLdb.connect(read_default_file=options_path)
where options_path is the path to the file above. This also keeps authentication data out of your code.
Django settings will look like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
Ref: https://docs.djangoproject.com/en/dev/ref/databases/#connecting-to-the-database