I'm using Python script to check if user requested exists.
using:
import MySQLdb
from flask import Flask, request, abort
app = Flask(__name__)
try:
db = MySQLdb.connect('xxx1','my_username','my_password','my_db_name')
db1 = MySQLdb.connect('xxx2','my_username','my_password','my_db_name')
db2=
db3=
except MySQLdb.OperationalError as e:
print "Caught an exception : " + str(.message)
#app.route('/')
#app.route('/<path:path>')
def page(path = ''):
user = request.args.get('user', None)
if not mac:
abort (403)
cursor = db.cursor()
query = 'Select ID from f_member where Name=%s'
db.commit()
cursor execute(query, (user, ))
row = cursor.fetchone()
cursor.close()
#cursor.db1 here
if row == None and row1 == None:
abort (403)
return 'OK', 200
if __name__ == '__main__':
app.run(host=host, port=port)
Then i have 5 nginx servers with this:
location = /auth {
proxy_pass http://xxx.xxx$request_uri;
proxy_pass_request_body off;
proxy_set_header Content_Lenght "";
proxy_set_header X-Real-IP $remote_addr;
So the thing is, this script checks if user is found in one
of the databases, if true then access the page.
Problem is my user list is now getting up to 5k users. and when i run the .py script it runs so fast (even with errors 403 people who are trying to connect), then broken pipe starts to show up.
Seems like it is getting overloaded, is there a better way to handle my script so it runs better and more efficient?
You may use a dictionary for username/id map in your Python program. Basically, when the program starts it will make a query for all the users and populate the map. Afterwards, every 20 seconds or so it will make a query to get the "changes" in f_member to update the dictionary. Lookup for username happens always in this map. If a user name is not found in the map, then only it makes a DB query (and if the user detail is found on DB, update the local map as well).
If you don't have millions of users in the table, this approach will work. Otherwise use a LRU cache.
So, after a lengthy comment thread, it appears like your flask instances might be competing for DB resources. There's also another hypothesis that saving your connections off in global scope could have some bad side effects (I could be wrong about this, but I'd be concerned about timeouts, not closing the connections, etc). Here's how I might rewrite it:
import MySQLdb
from flask import Flask, request, abort
app = Flask(__name__)
def get_db_connection_args():
try:
db_args = { 'host':'xxx1', 'user':'my_username', 'passwd':'my_password', 'db':'my_db_name' }
db1_args = { 'host':'xxx2', 'user':'my_username', 'passwd':'my_password', 'db':'my_db_name' }
db2_args = { 'host':'xxx3', 'user':'my_username', 'passwd':'my_password', 'db':'my_db_name' }
db3_args = { 'host':'xxx4', 'user':'my_username', 'passwd':'my_password', 'db':'my_db_name' }
except MySQLdb.OperationalError as e:
print "Caught an exception : " + str(.message)
return (db_args, db1_args, db2_args, db3_args)
#app.route('/')
#app.route('/<path:path>')
def page(path = ''):
user = request.args.get('user', None)
#I don't know what mac is...but it was in your original code.
if not mac:
abort (403)
found = False
db_connection_args = get_db_connection_args()
for db_connection_arg_dict in db_connection_args:
if not found:
db_conn = MySQLdb.connect(**db_connection_arg_dict)
try:
cursor = db_conn.cursor()
cursor.execute('Select ID from f_member where Name=%s', (user, ))
row = cursor.fetchone()
if row:
found = True
finally:
db_conn.close()
if found:
return 'OK', 200
abort (403)
if __name__ == '__main__':
app.run(host=host, port=port)
Related
I am implementing a basic HTTP server that interrogates a database via PostGRESQL.
In order to handle multiple requests I need to create a thread in the server for each one, and I am currently doing this.
class MyServer(ThreadingMixIn, HTTPServer):
def __init__(self, server_address, handler_class):
super().__init__(server_address=server_address, RequestHandlerClass=handler_class)
self.KEEP_ALIVE = True
def run(self):
while self.KEEP_ALIVE:
self.handle_request()
and, for example, my do_DELETE method is done like this:
def do_DELETE(self):
# Get ID from the request path
config_id = self.path[2:]
# Connect to DB
connection_db, cursor = self.connect_to_DB()
with connection_db:
# Check if the config to delete exists
cursor.execute('SELECT * FROM configuration WHERE id=%s', (config_id, ))
res = cursor.fetchall()
# If the config to delete exists, the operation is done. Otherwise an error message is sent back
if len(res) != 0:
cursor.execute('DELETE FROM configuration WHERE id=%s', (config_id,))
response_code = 200
answer = OPERATION_SUCCESSFUL
else:
# Send error message back
response_code = 400
answer = NO_SUCH_ID_ERROR
content_type = 'text/plain'
self.reply(response_code=response_code, content_type=content_type, answer=answer)
I would like to make it so that, in the do_DELETE method, the part with the two queries to the DB is atomic, i.e. it should lock on a resource visible by all the threads... but I have no idea on how to do it. Can you help me?
I am working on a project to work on a website that connects to a mysql server using flask and sqlAlchemy (Hosted on AWS RDS) and I am following this tutorial but when I try to do (/api/bar) I get this error. When I just do my localhost:8080 it shows "Hello World" perfectly fine.
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([WinError 10061] No connection could be made because the target machine actively refused it)") (Background on this error at: http://sqlalche.me/e/e3q8)
config.py
database_url = "mysql+pymysql://username:password#localhost:3306/barbeerdrinker"
Here is my __init__.py
from flask import Flask
from flask import jsonify
from flask import make_response
from flask import request
import json
from barbeerdrinker import database
#Flask application
app = Flask(__name__)
#app.route("/")
def hello_world():
return "Hello World"
#app.route("/api/bar", methods=["GET"])
def get_bars():
return jsonify(database.get_bars())
#app.route("/api/bar/<name>", methods=["GETS"])
def find_bar(name):
try:
if name is None:
raise ValueError("Bar is not specified")
bar = database.find_bar(name)
if bar is None:
return make_response("No bar found within given name", 404)
return jsonify(bar)
except ValueError as e:
return make_response(str(e), 400)
except Exception as e:
return make_response(str(e), 500)
#app.route("/api/beers_cheaper_than", methods=["POST"])
def find_beers_cheaper_than():
body = json.loads(request.data)
max_price = body['maxPrice']
return jsonify(database.filter_beers(max_price))
database.py
from sqlalchemy import create_engine
from sqlalchemy import sql
from barbeerdrinker import config
engine = create_engine(config.database_url)
def get_bars():
with engine.connect() as con:
rs = con.execute("SELECT name, address, city, opening, closing, phoneNum FROM bars")
return [dict(row) for row in rs]
def find_bar(name):
with engine.connect() as con:
query = sql.text("SELECT * FROM bars WHERE name = :name;")
rs = con.execute(query, name=name)
result = rs.first()
if result is None:
return None
return dict(result)
def filter_beers(max_price):
with engine.connect() as con:
query = sql.text("SELECT * FROM sells WHERE price < : max_price;")
rs = con.execute(query, max_price=max_price)
results = [dict(row) for row in rs]
for r in results:
r['price'] = float(r['price'])
return results
**Edit: So it seems like the problem is not an issue with my code but a Windows error. One solution I tried to do was to open up the required ports through my firewall to no avail.
I just figured it out, turns out the issue is not a windows issue. The problem is within my config.py:
Instead of:
database_url = "mysql+pymysql://username:password#localhost:3306/barbeerdrinker"
It should be:
database_url = "mysql+pymysql://username:password#**AWSENDPOINT/HOSTNAME**:3306/barbeerdrinker"
I am currently using AWS Lambda (Python 3.6) to talk to a MySQL database. I also have Slack commands triggering the queries to the database. On occasion, I have noticed that I can change things directly through MySQL Workbench and then trigger a query through Slack which returns old values. I currently connect to MySQL outside of the python handler like this:
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
The MySQL connection is done outside of any definition at the very top of my program. When Slack sends a command, I call another definition that then queries MySQL. This works okay sometimes, but other times can send my old data that has not updated. The whole layout is like this:
imports
SQL connections
SQL query definitions
handler definition
I tried moving the MySQL connection portion inside of the handler, but then the SQL query definitions do not recognize my cursor (out of scope, I guess).
So my question is, how do I handle this MySQL connection? Is it best to keep the MySQL connection outside of any definitions? Should I open and close the connection each time? Why is my data stale? Will Lambda ALWAYS run the entire routine or can it try to split the load between servers (I swear I read somewhere that I cannot rely on Lambda to always read my entire routine; sometimes it just reads the handler)?
I'm pretty new to all this, so any suggestions are much appreciated. Thanks!
Rest of the code if it helps:
################################################################################################################################################################################################################
# Slack Lambda handler.
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# IMPORTS
###############
import sys
import os
import pymysql
import urllib
import math
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Grab data from AWS environment.
###############
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Attempt SQL connection.
###############
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
################################################################################################################################################################################################################
# Define the URL of the targeted Slack API resource.
SLACK_URL = "https://slack.com/api/chat.postMessage"
################################################################################################################################################################################################################
# Function Definitions.
###############
def get_userExistance(user):
statement = f"SELECT 1 FROM slackDB.users WHERE userID LIKE '%{user}%' LIMIT 1"
cursor.execute(statement, args=None)
userExists = cursor.fetchone()
return userExists
def set_User(user):
statement = f"INSERT INTO `slackDB`.`users` (`userID`) VALUES ('{user}');"
cursor.execute(statement, args=None)
conn.commit()
return
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Slack command interactions.
###############
def lambda_handler(data, context):
# Slack challenge answer.
if "challenge" in data:
return data["challenge"]
# Grab the Slack channel data.
slack_event = data['event']
slack_userID = slack_event['user']
slack_text = slack_event['text']
channel_id = slack_event['channel']
slack_reply = ""
# Check sql connection.
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except pymysql.OperationalError:
connected = 0
else:
connected = 1
# Ignore bot messages.
if "bot_id" in slack_event:
slack_reply = ""
else:
# Start data sift.
if slack_text.startswith("!addme"):
if get_userExistance(slack_userID):
slack_reply = f"User {slack_userID} already exists"
else:
slack_reply = f"Adding user {slack_userID}"
set_user(slack_userID)
# We need to send back three pieces of information:
data = urllib.parse.urlencode(
(
("token", BOT_TOKEN),
("channel", channel_id),
("text", slack_reply)
)
)
data = data.encode("ascii")
# Construct the HTTP request that will be sent to the Slack API.
request = urllib.request.Request(
SLACK_URL,
data=data,
method="POST"
)
# Add a header mentioning that the text is URL-encoded.
request.add_header(
"Content-Type",
"application/x-www-form-urlencoded"
)
# Fire off the request!
urllib.request.urlopen(request).read()
# Everything went fine.
return "200 OK"
################################################################################################################################################################################################################
All of the code outside the lambda handler is only run once per container. All code inside the handler is run every time the lambda is invoked.
A lambda container lasts for between 10 and 30 minutes depending on usage. A new lambda invocation may or may not run on an already running container.
It's possible you are invoking a lambda in a container that is over 5 minutes old where your connection has timed out.
I'm currently running an flask python application running on an NGINX server with the help of uWGI. The static pages are always reachable but the pages that use a connection (MySQL) time out after 2 minutes. What's going on? They become simply become unavailable.
Things I have tried:
Not using a global
working with pools
turning off firewall
.
# using python version 2.7.10
from flask import Flask, jsonify, request, session
import mysql.connector.pooling
#Make a connection with the DB
dbconfig = {
"host" : "12.34.5.78",
"database": "db",
"user": "user",
"password": "pass"
}
conn = mysql.connector.connect(pool_name = "mypool",
pool_size = 6,
**dbconfig)
#Define the root
app = Flask(__name__)
#Landings page
#app.route('/')
def index():
return "Hello World."
# return all resources by name
#app.route('/resources', methods=['GET'])
def allResourceNames():
conn1 = mysql.connector.connect(pool_name="mypool")
reader = conn1.cursor()
query = ("SELECT name FROM resources")
reader.execute(query)
resources = []
for name in reader:
resources.append({'name' : name[0]})
reader.close()
conn1.close()
return jsonify({"resources" : resources})
if __name__ == "__main__":
app.run(debug=True)
According to MySQL Python Connector Doc
you may want to set connection_timeout as connect option. E.g.,
conn = mysql.connector.connect(pool_name = "mypool",
pool_size = 6, connection_timeout=3600,
**dbconfig)
I'm attempting to develop a web app using tornado/torndb and am running into some issues with my database interactions. I've written a "Database" class which wraps torndb in order to provide some common database functionality for my web app. When invoking any of the methods from the Database class I've written there seems to be a problem with the connection to the database:
"ERROR:root:Error connecting to MySQL on localhost"
My constructor opens the connection so I'm a bit confused as to why I see this message after the connection has been opened. I expect this is a scoping and/or GC issue that I am not understanding. The goal is to to create the Database object once and thus just have that
single connection persist throughout the life of the server, the db is stored
The following code snippet does work as expected which led me to the scoping or GC issue possibly:
#!/usr/bin/python
import torndb
class Database:
def __init__(self):
try:
self.__dbh = torndb.Connection(
'localhost',
'mydb',
user = 'myuser',
password = 'mypass')
except Exception as e:
print e
def user_add(self, user, email, passwd):
insert = "INSERT INTO users (username, email, passwd) VALUES " + \
"(%s, %s, %s)" % (user, email, passwd)
rowid = 0
try:
rowid = self.__dbh.execute(insert)
except Exception as e:
print e
if rowid is 0:
return (False, 'User exists');
return (True, None)
if __name__ == "__main__":
print 'Testing'
raw_input('Hit enter to connect to the DB')
d = Database();
users = []
raw_input('Hit enter to create some users')
for i in range(5):
users.append(str(i))
d.user_add(users[i], users[i], users[i])
<- snip ->
The issue is when I try to create a Database object from anywhere other than the main of the module that defines the Database class, for example:
import tornado.ioloop
import tornado.httpserver
import tornado.web
from register import Register
from logon import Logon
from db import Database
class Application(tornado.web.Application):
def __init__(self):
resources = [
(r"/logon", Logon),
(r"/register", Register)
]
self.db = Database()
tornado.web.Application.__init__(self, resources)
try:
self.db.user_add('testuser', 'testemail', 'password')
except Exception as e:
print e
if __name__ == "__main__":
app = Application()
# Start the server.
server = tornado.httpserver.HTTPServer(app)
server.listen(8080)
tornado.ioloop.IOLoop.instance().start()
The above when executed prints (due to the call to self.__dbh.execute()):
ERROR:root:Error connecting to MySQL on localhost
Some other bits of information:
I can connect to the db from the console without any issues.
I'm following this example https://github.com/facebook/tornado/blob/master/demos/blog/blog.py#L60
torndb version is 2.4.1, torndb version is LATEST (pulled using pip).
Questions:
Why is there a difference when I create my Database object in the main of the module that defines the class compared to creating the Database object anywhere else?
The problem was due to the string arguments passed to the query not being escaped, with the following change it works:
def user_add(self, user, email, passwd):
insert = "INSERT INTO users (username, email, passwd) VALUES " + \
"(\'%s\', \'%s\', \'%s\')" % (user, email, passwd)
rowid = 0
try:
rowid = self.__dbh.execute(insert)
except Exception as e:
print e
if rowid is 0:
return (False, 'User exists');
return (True, None)