Develop Reference

How to set connection with database in the api Flask?

I'm trying to create the api on my linux PC. At this moment I have support for some basic requests which were done just for testing. My api works in cooperation with uswgi+nginx+flask. And now I'm trying to add connection to the database. For this purpose I had installed MySQL and created database. But I don't understand how to connect from the api to database. For example here is code of the script which can connect to the DB but it works separately of the api:
try:
connection = mysql.connector.connect(host='localhost',
database='tired_db',
user='test',
password='pw')
if connection.is_connected():
mycursor = connection.cursor()
mycursor.execute("SHOW TABLES")
for x in mycursor:
print(x)
return connection
except Error as e:
print("Error while connecting to MySQL", e)
finally:
if connection.is_connected():
mycursor.close()
connection.close()
print("MySQL connection is closed")
and it works correctly. I thought that maybe I can call this connection like some metaclass:
import mysql.connector
from mysql.connector import Error
class DbProvider(type):
#property
def my_data(cls):
try:
connection = mysql.connector.connect(host='localhost',
database='tired_db',
user='test',
password='pw')
if connection.is_connected():
mycursor = connection.cursor()
mycursor.execute("SHOW TABLES")
for x in mycursor:
print(x)
return connection
except Error as e:
print("Error while connecting to MySQL", e)
finally:
if connection.is_connected():
mycursor.close()
connection.close()
print("MySQL connection is closed")
class MyClass(metaclass=DbProvider):
pass
if __name__ == "__main__":
MyClass.my_data
but I think that such stuff can be done with more efficient way. For example here is some request in the api:
#app.route("/api/login", methods = ['POST'])
def logIn():
return "all is ok"
and the idea is that for example I have to connect during this request to the DB and check whether a user exists or not and if all is ok generate+save some token to the database. I don't understand whether it is important to keep connection alive during all api uptime or only during requests. And also is it important to close connection after an every request or we have to keep alive it forever. And also how to call connection from separate class, or I have to have all stuff in one file together with api calls.

You can write a connection manager class that connects to the database, performs operations, and closes the connection.
import mysql.connector
from mysql.connector import Error
class ConnectionManager:
def __init__(self, host, database, user, password):
self.host = host
self.database = database
self.user = user
self.password = password
def __enter__(self):
try:
self.connection = mysql.connector.connect(
host=self.host,
database=self.database,
user=self.user,
password=self.password
)
if self.connection.is_connected():
self.cursor = self.connection.cursor()
return self.cursor
except Error as e:
print("Error while connecting to MySQL", e)
def __exit__(self, type, value, traceback):
if self.connection.is_connected():
self.cursor.close()
self.connection.close()
print("MySQL connection is closed")
#app.route("/api/login", methods = ['POST'])
def logIn():
with ConnectionManager('localhost', 'tired_db', 'test', 'pw') as cursor:
cursor.execute("SELECT * FROM users WHERE username = %s", ('test_user',))
result = cursor.fetchone()
if result:
# generate and save token
return "all is ok"
else:
return "user not found"

Python : threading.Thread block

import pymysql
from multiprocessing.pool import ThreadPool
def process(item):
host = str(item).strip()
db = None
try:
db = pymysql.connect(host=host, user='root', passwd='root', port=3306, connect_timeout=10,
write_timeout=10)
r = open("success.txt", 'a')
r.write(host)
r.write("\n")
pass
except Exception as msg:
print(msg)
pass
finally:
if db:
db.close()
pass
if __name__ == '__main__':
t = ThreadPool(100)
t.map_async(process,open('3306.txt').readlines())
t.close()
t.join()
I want to verify in batches whether the MySQL account passwords of some IPs are valid, but my code will have the problem of sub-thread blocking. In fact, the last IP has been verified, but the main thread has not ended. I don't know how to modify it. You can provide Some modifications and suggestions?

sqlalchemy engine.connect() stalled

I use sqlachemy to connect to a remote database but I do not know the type (can be PostgreSQL, MariaDB, etc.). I try them in a loop and I keep the first working driver:
for driver in drivers:
try:
uri = get_uri_from_driver(driver)
engine = create_engine(uri, echo=False)
print('Try connection')
con = engine.engine.connect()
# Try to get some lines
return engine
except Exception:
continue
return None
In some case the con = engine.engine.connect() does not end and it happens when you try the MySQL driver to connect to something which is not MySQL (Oracle).
Questions:
How can I set a timeout to this?
If I cannot, is there any other way to achieve this ? (I will for example base the test order with the default port but I would like to be able to kill the connect() after some seconds.
EDIT:
This code is in a Django so I cannot use signal/alarm because of multi-threading.

This can be done with a generic timeout solution like in:
What should I do if socket.setdefaulttimeout() is not working?
import signal
class Timeout():
"""Timeout class using ALARM signal"""
class TimeoutException(Exception): pass
def __init__(self, sec):
self.sec = sec
def __enter__(self):
signal.signal(signal.SIGALRM, self.raise_timeout)
signal.alarm(self.sec)
def __exit__(self, *args):
signal.alarm(0) # disable alarm
def raise_timeout(self, *args):
raise Timeout.TimeoutException()
# In your example
try:
uri = get_uri_from_driver(driver)
engine = create_engine(uri, echo=False)
print('Try connection')
with Timeout(10):
con = engine.engine.connect()
# Try to get some lines
return engine
except Exception:
continue

Accessing a MySQL connection pool from Python multiprocessing

I'm trying to set up a MySQL connection pool and have my worker processes access the already established pool instead of setting up a new connection each time.
I'm confused if I should pass the database cursor to each process, or if there's some other way to do this? Shouldn't MySql.connector do the pooling automatically? When I check my log files, many, many connections are opened and closed ... one for each process.
My code looks something like this:
PATH = "/tmp"
class DB(object):
def __init__(self):
connected = False
while not connected:
try:
cnxpool = mysql.connector.pooling.MySQLConnectionPool(pool_name = "pool1",
**config.dbconfig)
self.__cnx = cnxpool.get_connection()
except mysql.connector.errors.PoolError:
print("Sleeping.. (Pool Error)")
sleep(5)
except mysql.connector.errors.DatabaseError:
print("Sleeping.. (Database Error)")
sleep(5)
self.__cur = self.__cnx.cursor(cursor_class=MySQLCursorDict)
def execute(self, query):
return self.__cur.execute(query)
def isValidFile(self, name):
return True
def readfile(self, fname):
d = DB()
d.execute("""INSERT INTO users (first_name) VALUES ('michael')""")
def main():
queue = multiprocessing.Queue()
pool = multiprocessing.Pool(None, init, [queue])
for dirpath, dirnames, filenames in os.walk(PATH):
full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
filenames)
full_path_fnames = filter(is_valid_file, full_path_fnames)
pool.map(readFile, full_path_fnames)
if __name__ == '__main__':
sys.exit(main())

First, you're creating a different connection pool for each instance of your DB class. The pools having the same name doesn't make them the same pool
From the documentation:
It is not an error for multiple pools to have the same name. An application that must distinguish pools by their pool_name property should create each pool with a distinct name.
Besides that, sharing a database connection (or connection pool) between different processes would be a bad idea (and i highly doubt it would even work correctly), so each process using it's own connections is actually what you should aim for.
You could just initialize the pool in your init initializer as a global variable and use that instead.
Very simple example:
from multiprocessing import Pool
from mysql.connector.pooling import MySQLConnectionPool
from mysql.connector import connect
import os
pool = None
def init():
global pool
print("PID %d: initializing pool..." % os.getpid())
pool = MySQLConnectionPool(...)
def do_work(q):
con = pool.get_connection()
print("PID %d: using connection %s" % (os.getpid(), con))
c = con.cursor()
c.execute(q)
res = c.fetchall()
con.close()
return res
def main():
p = Pool(initializer=init)
for res in p.map(do_work, ['select * from test']*8):
print(res)
p.close()
p.join()
if __name__ == '__main__':
main()
Or just use a simple connection instead of a connection pool, as only one connection will be active in each process at a time anyway.
The number of concurrently used connections is implicitly limited by the size of the multiprocessing.Pool.

#!/usr/bin/python
# -*- coding: utf-8 -*-
import time
import mysql.connector.pooling
dbconfig = {
"host":"127.0.0.1",
"port":"3306",
"user":"root",
"password":"123456",
"database":"test",
}
class MySQLPool(object):
"""
create a pool when connect mysql, which will decrease the time spent in
request connection, create connection and close connection.
"""
def __init__(self, host="172.0.0.1", port="3306", user="root",
password="123456", database="test", pool_name="mypool",
pool_size=3):
res = {}
self._host = host
self._port = port
self._user = user
self._password = password
self._database = database
res["host"] = self._host
res["port"] = self._port
res["user"] = self._user
res["password"] = self._password
res["database"] = self._database
self.dbconfig = res
self.pool = self.create_pool(pool_name=pool_name, pool_size=pool_size)
def create_pool(self, pool_name="mypool", pool_size=3):
"""
Create a connection pool, after created, the request of connecting
MySQL could get a connection from this pool instead of request to
create a connection.
:param pool_name: the name of pool, default is "mypool"
:param pool_size: the size of pool, default is 3
:return: connection pool
"""
pool = mysql.connector.pooling.MySQLConnectionPool(
pool_name=pool_name,
pool_size=pool_size,
pool_reset_session=True,
**self.dbconfig)
return pool
def close(self, conn, cursor):
"""
A method used to close connection of mysql.
:param conn:
:param cursor:
:return:
"""
cursor.close()
conn.close()
def execute(self, sql, args=None, commit=False):
"""
Execute a sql, it could be with args and with out args. The usage is
similar with execute() function in module pymysql.
:param sql: sql clause
:param args: args need by sql clause
:param commit: whether to commit
:return: if commit, return None, else, return result
"""
# get connection form connection pool instead of create one.
conn = self.pool.get_connection()
cursor = conn.cursor()
if args:
cursor.execute(sql, args)
else:
cursor.execute(sql)
if commit is True:
conn.commit()
self.close(conn, cursor)
return None
else:
res = cursor.fetchall()
self.close(conn, cursor)
return res
def executemany(self, sql, args, commit=False):
"""
Execute with many args. Similar with executemany() function in pymysql.
args should be a sequence.
:param sql: sql clause
:param args: args
:param commit: commit or not.
:return: if commit, return None, else, return result
"""
# get connection form connection pool instead of create one.
conn = self.pool.get_connection()
cursor = conn.cursor()
cursor.executemany(sql, args)
if commit is True:
conn.commit()
self.close(conn, cursor)
return None
else:
res = cursor.fetchall()
self.close(conn, cursor)
return res
if __name__ == "__main__":
mysql_pool = MySQLPool(**dbconfig)
sql = "select * from store WHERE create_time < '2017-06-02'"
p = Pool()
for i in range(5):
p.apply_async(mysql_pool.execute, args=(sql,))
Code above creates a connection pool at the beginning, and get connections from it in execute(), once the connection pool has been created, the work is to remain it, since the pool is created only once, it will save the time to request for a connection every time you would like to connect to MySQL.
Hope it helps!

You created multiple DB object instance. In mysql.connector.pooling.py, pool_name is only a attribute to let you make out which pool it is. There is no mapping in the mysql pool.
So, you create multiple DB instance in def readfile(), then you will have several connection pool.
A Singleton is useful in this case.
(I spent several hours to find it out. In Tornado framework, each http get create a new handler, which leads to making a new connection.)

There may be synchronization issues if you're going to reuse MySQLConnection instances maintained by a pool, but just sharing a MySQLConnectionPool instance between worker processes and using connections retrieved by calling the method get_connection() would be okay, because a dedicated socket would be created for each MySQLConnection instance.
import multiprocessing
from mysql.connector import pooling
def f(cnxpool: pooling.MySQLConnectionPool) -> None:
# Dedicate connection instance for each worker process.
cnx = cnxpool.get_connection()
...
if __name__ == '__main__':
cnxpool = pooling.MySQLConnectionPool(
pool_name='pool',
pool_size=2,
)
p0 = multiprocessing.Process(target=f, args=(cnxpool,))
p1 = multiprocessing.Process(target=f, args=(cnxpool,))
p0.start()
p1.start()

Making sure a query gets executed

I have a server which executes properly 9/10 times, but sometimes I get the error “Lost connection to MySQL server during query” and then the whole process stops/freezes.
I'm trying to let a MySQL query execute again if it fails using a function for this, unfortunately it seems that the code 'stops' or get stuck whenever I get an error. Am I doing something wrong?
Here string is something like: SELECT xx FROM xxx WHERE xxx
try:
db = MySQLdb.connect (host = "",
user = "",
passwd = "",
db = "" )
except MySQLdb.Error, e:
print("Error %d: %s" %(e.args[0], e.args[1]))
sys.exit(1);
cursor = db.cursor()
def mysql_handling(string):
while 1:
try:
cursor.execute(string)
if 'SELECT' not in string:
db.commit()
break
except:
mysql_error_tracking(string)
mysql_error_tracking is a function to monitor what queries fail most of the time (not relevant).

Add a timeout to your connection method:
db = MySQLdb.connect (host = HOST, user = USER, passwd = PASS, db = DB,
connect_timeout = TIMEOUT)
Do this in your except block so that you do a re-connect.
To make your life easier, place your connection code in a separate function so you can re-use it. Something like this (untested):
def get_cursor()
try:
db = MySQLdb.connect (host = HOST, user = USER, passwd = PASS, db = DB,
connect_timeout = TIMEOUT)
except MySQLdb.Error, e:
print("Error %d: %s" %(e.args[0], e.args[1]))
sys.exit(1);
return db.cursor()
def mysql_handling(cursor, string):
while True:
try:
cursor.execute(string)
if 'SELECT' not in string:
db.commit()
break
except MySQLdb.MySQLError:
cursor.close()
mysql_error_tracking(string)
cursor = get_cursor()
def main():
mysql_handling(get_cursor(), string)