Im using python 3 and i have this loop, which iterate on this list -
self.settings["db"]["host"] = ["db-0", "db-1"]
My problem is that it seems to send in return self.conn the first option all the time,
db-0 instead of trying with db-1
I have 2 db container servers, and when i stop one of them - for example db-0 it should try psycopg2.connect with db-1
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
return self.conn
except Exception:
pass
if loop has not succeeded i dont want it to return self.conn, only if the try worked.
I also tried :
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
except Exception:
pass
return self.conn
You are looping within a try.
Do it the other way around,
push the try down within the loop.
The DbC contract the current code is attempting to offer
is to return a valid connection.
Let's make that more explicit, by writing a very simple helper.
We will spell out its contract in a docstring.
def first_true(iterable, default=False, pred=None):
# from https://docs.python.org/3/library/itertools.html
return next(filter(pred, iterable), default)
def get_conn_or_none(self, server):
"""Returns DB connection to server, or None if that's not possible."""
try:
return psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
except Exception:
return None
Now db_conn is simply:
def db_conn(self):
return first_true(map(self.get_conn_or_none, self.settings["db"]["host"]))
That uses the same logic as in your question.
You may want to have db_conn additionally check whether
all connection attempts failed, and raise fatal error in that case.
BTW, it's very odd that you're apparently storing a list of server hostnames
in self.settings["db"]["host"], given that an individual user / pw / db is
stored in the other fields.
Consider renaming that list to self.servers.
You can try to check the connection status before returning it:
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
**if self.conn.isOk()**
return self.conn
except Exception:
pass
This code worked for me:
def db_conn(self):
for server in self.settings["db"]["host"]:
try:
self.print_info("TRYING", server)
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
)
except:
self.print_info("SERVER DOWN", server)
continue
return self.conn
continue will continue the rest of the code if i get exception ( failed connection)
then it goes back to for loop with the second item in list.
Related
I'm having difficulties connecting to my Neo4j Database in VS Code.
I started with the boiler plate code that is given on the Neo4J documentation:
from neo4j import GraphDatabase
import logging
from neo4j.exceptions import ServiceUnavailable
class Neo4jConnection:
def __init__(self, uri, user, pwd):
self.__uri = uri
self.__user = user
self.__pwd = pwd
self.__driver = None
try:
self.__driver = GraphDatabase.driver(self.__uri, auth=(self.__user, self.__pwd))
except Exception as e:
print("Failed to create the driver:", e)
def close(self):
if self.__driver is not None:
self.__driver.close()
def query(self, query, db=None):
assert self.__driver is not None, "Driver not initialized!"
session = None
response = None
try:
session = self.__driver.session(database=db) if db is not None else self.__driver.session()
response = list(session.run(query))
except Exception as e:
print("Query failed:", e)
finally:
if session is not None:
session.close()
return response
Then I connected to my database:
conn = Neo4jConnection(uri="neo4j+s://7022d007.databases.neo4j.io", user="neo4j", pwd="****")
Then I attempted to call for neo4j to run a task in the database:
query_string = '''
CALL db.schema.visualization()
'''
conn.query(query_string, db='MARA')
Which then failed and gave me the error:
Unable to retrieve routing information
Query failed: Unable to retrieve routing information
This could be due to the certificate error. It worked for me after changing the certificate to self signed SSL. You could try using neo4j+ssc://{IP_address}:{Port} as the link to DB.
I'm writing some Python code to connect to our Snowflake Database and getting an error in my Python (I'm new to Python so i've likely done something wrong here).
I'm getting <class 'NameError'> TestConnection.py 98 inside my script, which has the relevant parts below
def connection(obj):
try:
global cur, engine
cur = obj.cursor_connection().cursor()
engine = obj.engine_connection().engine.connect()
except Exception as e:
print(e)
sql = "select current_warehouse(), current_database(), current_schema();"
try:
print("Cursor connection successful")
except Exception as e:
print(e)
try:
print("Engine connection successful")
except Exception as e:
print(e)
return cur, engine
try:
#Setup Connection with both cursor and the engine
db, schema= login.db, login.schema
obj = connect(db, schema)
cur , engine = connection(obj)
The line I'm getting the error on is the cur,engine = connection(obj) part.
I had a previous error before (UnboundLocalError) but putting global cur, engine inside the connection function fixed that, but getting NameError now.
What am I doing wrong here?
Thanks
try this
import snowflake.connector
ctx = snowflake.connector.connect(
user='<user_name>',
password='<password>',
account='<account_identifier>'
)
cs = ctx.cursor()
Ok, I've fixed this now, it was indentation on the functions and class.
I ended up with
class connect:
def __init__
def cursor_connection
def engine_connection
def connection
and that has it working whereas having def_connection tabbed in was breaking it.
I'm newbie in Python. I'm trying to use Python to connect MySQL Server. I wrote like guides from MySQL official page, it was OK. But, when I create a connector class, it raised the error "MySQL Connection not available"
Here is my class
import mysql.connector
from mysql.connector import errorcode
## BEGIN MySQL Connector Class
class MySQLConnector :
configs = {
"user":"root",
"password":"",
"host":"127.0.0.1",
"database":"python_db",
"raise_on_warnings": True
}
cursor = None
connection = None
## BEGIN Constructor
def __init__(self, configs = {}) :
if(any(configs)!=False) :
self.configs = configs
## END Constructor
## BEGIN Open
def open(self) :
try:
self.connection = mysql.connector.connect(self.configs)
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with your user name or password")
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exists")
else:
print(err)
finally:
self.connection.close()
return self.connection
## END Open
## BEGIN close connection
def close(self) :
self.cursor.close()
self.connection.close()
## END close connection
## BEGIN execute
def execute(self, query) :
if(self.connection == None) :
print("Connection is None")
return
self.cursor = self.connection.cursor()
if(self.cursor!=None) :
self.cursor.execute(query)
else:
print("Cursor is 'None'")
## END execute
## END MySQL Connector Class
## BEGIN RUN
objConnect = MySQLConnector()
objConnect.open()
objConnect.execute("SELECT * FROM User")
Please show me the way to solution and explained me why my code has error.
Thanks!
EDITED
Finally, mata and alecxe help me to solve this problem, I don't know which solution to be choosen. I summary here for someone has mistake like me:
1. Remove the finally statement.
2. Using ** in self.connection = mysql.connector.connect(**self.configs)
Even if you correct the error alecxe pointed out, your code still won't work.
The finally block ensures that each connection is closed before it is returned, no matter wheather there was an exception or not, so the open method only returns closed connections.
You are passing a dictionary object self.configs into mysql.connector.connect, though, according to docs, you should pass to it user, password and other arguments. Looks like you need to unpack configs:
self.connection = mysql.connector.connect(**self.configs)
Hope this is it.
I want my code to automatically try multiple ways to create a database connection. As soon as one works, the code needs to move on (i.e. it shouldn't try to other ways anymore). If they all fail well, then the script can just blow up.
So in - what I thought was, but most likely isn't - a stroke of genius I tried this:
import psycopg2
from getpass import getpass
# ouch, global variable, ooh well, it's just a simple script eh
CURSOR = None
def get_cursor():
"""Create database connection and return standard cursor."""
global CURSOR
if not CURSOR:
# try to connect and get a cursor
try:
# first try the bog standard way: db postgres, user postgres and local socket
conn = psycopg2.connect(database='postgres', user='postgres')
except psycopg2.OperationalError:
# maybe user pgsql?
conn = psycopg2.connect(database='postgres', user='pgsql')
except psycopg2.OperationalError:
# maybe it was postgres, but on localhost? prolly need password then
conn = psycopg2.connect(database='postgres', user='postgres', host='localhost', password=getpass())
except psycopg2.OperationalError:
# or maybe it was pgsql and on localhost
conn = psycopg2.connect(database='postgres', user='pgsql', host='localhost', password=getpass())
# allright, nothing blew up, so we have a connection
# now make a cursor
CURSOR = conn.cursor()
# return existing or new cursor
return CURSOR
But it seems that the second and subsequent except statements aren't catching the OperationalErrors anymore. Probably because Python only catches an exception once in a try...except statement?
Is that so? If not: is there anything else I'm doing wrong? If so: how do you do something like this then? Is there a standard idiom?
(I know there are ways around this problem, like having the user specify the connection parameters on the command line, but that's not my question ok :) )
EDIT:
I accepted retracile's excellent answer and I took in gnibbler's comment for using the for..else construct. The final code became (sorry, I'm not really following the max characters per line limit from pep8):
EDIT 2: As you can see from the comment on the Cursor class: I don't really know how to call this kind of class. It's not really a singleton (I can have multiple different instances of Cursor) but when calling get_cursor I do get the same cursor object everytime. So it's like a singleton factory? :)
import psycopg2
from getpass import getpass
import sys
class UnableToConnectError(Exception):
pass
class Cursor:
"""Cursor singleton factory?"""
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
# try to connect and get a cursor
attempts = [
{'database': 'postgres', 'user': 'postgres'},
{'database': 'postgres', 'user': 'pgsql'},
{'database': 'postgres', 'user': 'postgres', 'host': 'localhost', 'password': None},
{'database': 'postgres', 'user': 'pgsql', 'host': 'localhost', 'password': None},
]
for attempt in attempts:
if 'password' in attempt:
attempt['password'] = getpass(stream=sys.stderr) # tty and stderr are default in 2.6, but 2.5 uses sys.stdout, which I don't want
try:
conn = psycopg2.connect(**attempt)
attempt.pop('password', None)
sys.stderr.write("Succesfully connected using: %s\n\n" % attempt)
break # no exception raised, we have a connection, break out of for loop
except psycopg2.OperationalError:
pass
else:
raise UnableToConnectError("Unable to connect: exhausted standard permutations of connection dsn.")
# allright, nothing blew up, so we have a connection
# now make a cursor
self.CURSOR = conn.cursor()
# return existing or new cursor
return self.CURSOR
get_cursor = Cursor()
Approximately:
attempts = [
{ 'database'='postgres', 'user'='pgsql', ...},
{ 'database'='postgres', 'user'='postgres', 'host'='localhost', 'password'=getpass()},
...
]
conn = None
for attempt in attempts:
try:
conn = psycopg2.connect(**attempt)
break
except psycopg2.OperationalError:
pass
if conn is None:
raise a ruckus
CURSOR = conn.cursor()
Now, if you don't want to call getpass() unless it is necessary, you'd want to check if 'password' in attempt: attempt['password'] = getpass() or so.
Now about that global....
class MyCursor:
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
<insert logic here>
return self.CURSOR
get_cursor = MyCursor()
... though I think there are a couple of other ways to accomplish the same thing.
Bringing it all together:
class MyCursor:
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
attempts = [
{'database'='postgres', 'user'='postgres'},
{'database'='postgres', 'user'='pgsql'},
{'database'='postgres', 'user'='postgres', 'host'='localhost', 'password'=True},
{'database'='postgres', 'user'='pgsql', 'host'='localhost', 'password'=True},
]
conn = None
for attempt in attempts:
if 'password' in attempt:
attempt['password'] = getpass()
try:
conn = psycopg2.connect(**attempt)
break # that didn't throw an exception, we're done
except psycopg2.OperationalError:
pass
if conn is None:
raise a ruckus # nothin' worked
self.CURSOR = conn.cursor()
return self.CURSOR
get_cursor = MyCursor()
Note: completely untested
You're close. Probably the best thing to do in this case is nesting the second and subsequent attempts in the except block. Thus the critical part of your code would look like:
if not CURSOR:
# try to connect and get a cursor
try:
# first try the bog standard way: db postgres, user postgres and local socket
conn = psycopg2.connect(database='postgres', user='postgres')
except psycopg2.OperationalError:
# maybe user pgsql?
try:
conn = psycopg2.connect(database='postgres', user='pgsql')
except psycopg2.OperationalError:
# maybe it was postgres, but on localhost? prolly need password then
try:
conn = psycopg2.connect(database='postgres', user='postgres', host='localhost', password=getpass())
except psycopg2.OperationalError:
# or maybe it was pgsql and on localhost
conn = psycopg2.connect(database='postgres', user='pgsql', host='localhost', password=getpass())
I came across PHP way of doing the trick:
my_bool reconnect = 1;
mysql_options(&mysql, MYSQL_OPT_RECONNECT, &reconnect);
but no luck with MySQLdb (python-mysql).
Can anybody please give a clue? Thanks.
I solved this problem by creating a function that wraps the cursor.execute() method since that's what was throwing the MySQLdb.OperationalError exception. The other example above implies that it is the conn.cursor() method that throws this exception.
import MySQLdb
class DB:
conn = None
def connect(self):
self.conn = MySQLdb.connect()
def query(self, sql):
try:
cursor = self.conn.cursor()
cursor.execute(sql)
except (AttributeError, MySQLdb.OperationalError):
self.connect()
cursor = self.conn.cursor()
cursor.execute(sql)
return cursor
db = DB()
sql = "SELECT * FROM foo"
cur = db.query(sql)
# wait a long time for the Mysql connection to timeout
cur = db.query(sql)
# still works
I had problems with the proposed solution because it didn't catch the exception. I am not sure why.
I have solved the problem with the ping(True) statement which I think is neater:
import MySQLdb
con=MySQLdb.Connect()
con.ping(True)
cur=con.cursor()
Got it from here: http://www.neotitans.com/resources/python/mysql-python-connection-error-2006.html
If you are using ubuntu Linux there was a patch added to the python-mysql package that added the ability to set that same MYSQL_OPT_RECONNECT option (see here). I have not tried it though.
Unfortunately, the patch was later removed due to a conflict with autoconnect and transations (described here).
The comments from that page say:
1.2.2-7 Published in intrepid-release on 2008-06-19
python-mysqldb (1.2.2-7) unstable; urgency=low
[ Sandro Tosi ]
* debian/control
- list items lines in description starts with 2 space, to avoid reformat
on webpages (Closes: #480341)
[ Bernd Zeimetz ]
* debian/patches/02_reconnect.dpatch:
- Dropping patch:
Comment in Storm which explains the problem:
# Here is another sad story about bad transactional behavior. MySQL
# offers a feature to automatically reconnect dropped connections.
# What sounds like a dream, is actually a nightmare for anyone who
# is dealing with transactions. When a reconnection happens, the
# currently running transaction is transparently rolled back, and
# everything that was being done is lost, without notice. Not only
# that, but the connection may be put back in AUTOCOMMIT mode, even
# when that's not the default MySQLdb behavior. The MySQL developers
# quickly understood that this is a terrible idea, and removed the
# behavior in MySQL 5.0.3. Unfortunately, Debian and Ubuntu still
# have a patch right now which *reenables* that behavior by default
# even past version 5.0.3.
I needed a solution that works similarly to Garret's, but for cursor.execute(), as I want to let MySQLdb handle all escaping duties for me. The wrapper module ended up looking like this (usage below):
#!/usr/bin/env python
import MySQLdb
class DisconnectSafeCursor(object):
db = None
cursor = None
def __init__(self, db, cursor):
self.db = db
self.cursor = cursor
def close(self):
self.cursor.close()
def execute(self, *args, **kwargs):
try:
return self.cursor.execute(*args, **kwargs)
except MySQLdb.OperationalError:
self.db.reconnect()
self.cursor = self.db.cursor()
return self.cursor.execute(*args, **kwargs)
def fetchone(self):
return self.cursor.fetchone()
def fetchall(self):
return self.cursor.fetchall()
class DisconnectSafeConnection(object):
connect_args = None
connect_kwargs = None
conn = None
def __init__(self, *args, **kwargs):
self.connect_args = args
self.connect_kwargs = kwargs
self.reconnect()
def reconnect(self):
self.conn = MySQLdb.connect(*self.connect_args, **self.connect_kwargs)
def cursor(self, *args, **kwargs):
cur = self.conn.cursor(*args, **kwargs)
return DisconnectSafeCursor(self, cur)
def commit(self):
self.conn.commit()
def rollback(self):
self.conn.rollback()
disconnectSafeConnect = DisconnectSafeConnection
Using it is trivial, only the initial connect differs. Extend the classes with wrapper methods as per your MySQLdb needs.
import mydb
db = mydb.disconnectSafeConnect()
# ... use as a regular MySQLdb.connections.Connection object
cursor = db.cursor()
# no more "2006: MySQL server has gone away" exceptions now
cursor.execute("SELECT * FROM foo WHERE bar=%s", ("baz",))
you can separate the commit and the close for the connection...that's not cute but it does it.
class SqlManager(object):
"""
Class that handle the database operation
"""
def __init__(self,server, database, username, pswd):
self.server = server
self.dataBase = database
self.userID = username
self.password = pswd
def Close_Transation(self):
"""
Commit the SQL Query
"""
try:
self.conn.commit()
except Sql.Error, e:
print "-- reading SQL Error %d: %s" % (e.args[0], e.args[1])
def Close_db(self):
try:
self.conn.close()
except Sql.Error, e:
print "-- reading SQL Error %d: %s" % (e.args[0], e.args[1])
def __del__(self):
print "close connection with database.."
self.conn.close()
I had a similar problem with MySQL and Python, and the solution that worked for me was to upgrade MySQL to 5.0.27 (on Fedora Core 6; your system may work fine with a different version).
I tried a lot of other things, including patching the Python libraries, but upgrading the database was a lot easier and (I think) a better decision.
In addition to Liviu Chircu solution ... add the following method to DisconnectSafeCursor:
def __getattr__(self, name):
return getattr(self.cursor, name)
and the original cursor properties like "lastrowid" will keep working.
You other bet it to work around dropped connections yourself with code.
One way to do it would be the following:
import MySQLdb
class DB:
conn = None
def connect(self):
self.conn = MySQLdb.connect()
def cursor(self):
try:
return self.conn.cursor()
except (AttributeError, MySQLdb.OperationalError):
self.connect()
return self.conn.cursor()
db = DB()
cur = db.cursor()
# wait a long time for the Mysql connection to timeout
cur = db.cursor()
# still works