I'm writing some Python code to connect to our Snowflake Database and getting an error in my Python (I'm new to Python so i've likely done something wrong here).
I'm getting <class 'NameError'> TestConnection.py 98 inside my script, which has the relevant parts below
def connection(obj):
try:
global cur, engine
cur = obj.cursor_connection().cursor()
engine = obj.engine_connection().engine.connect()
except Exception as e:
print(e)
sql = "select current_warehouse(), current_database(), current_schema();"
try:
print("Cursor connection successful")
except Exception as e:
print(e)
try:
print("Engine connection successful")
except Exception as e:
print(e)
return cur, engine
try:
#Setup Connection with both cursor and the engine
db, schema= login.db, login.schema
obj = connect(db, schema)
cur , engine = connection(obj)
The line I'm getting the error on is the cur,engine = connection(obj) part.
I had a previous error before (UnboundLocalError) but putting global cur, engine inside the connection function fixed that, but getting NameError now.
What am I doing wrong here?
Thanks
try this
import snowflake.connector
ctx = snowflake.connector.connect(
user='<user_name>',
password='<password>',
account='<account_identifier>'
)
cs = ctx.cursor()
Ok, I've fixed this now, it was indentation on the functions and class.
I ended up with
class connect:
def __init__
def cursor_connection
def engine_connection
def connection
and that has it working whereas having def_connection tabbed in was breaking it.
Related
I have a python application that is reading from mysql/mariadb, uses that to fetch data from an api and then inserts results into another table.
I had setup a module with a function to connect to the database and return the connection object that is passed to other functions/modules. However, I believe this might not be a correct approach. The idea was to have a small module that I could just call whenever I needed to connect to the db.
Also note, that I am using the same connection object during loops (and within the loop passing to the db_update module) and call close() when all is done.
I am also getting some warnings from the db sometimes, those mostly happen at the point where I call db_conn.close(), so I guess I am not handling the connection or session/engine correctly. Also, the connection id's in the log warning keep increasing, so that is another hint, that I am doing it wrong.
[Warning] Aborted connection 351 to db: 'some_db' user: 'some_user' host: '172.28.0.3' (Got an error reading communication packets)
Here is some pseudo code that represents the structure I currently have:
################
## db_connect.py
################
# imports ...
from sqlalchemy import create_engine
def db_connect():
# get env ...
db_string = f"mysql+pymysql://{db_user}:{db_pass}#{db_host}:{db_port}/{db_name}"
try:
engine = create_engine(db_string)
except Exception as e:
return None
db_conn = engine.connect()
return db_conn
################
## db_update.py
################
# imports ...
def db_insert(db_conn, api_result):
# ...
ins_qry = "INSERT INTO target_table (attr_a, attr_b) VALUES (:a, :b);"
ins_qry = text(ins_qry)
ins_qry = ins_qry.bindparams(a = value_a, b = value_b)
try:
db_conn.execute(ins_qry)
except Exception as e:
print(e)
return None
return True
################
## main.py
################
from sqlalchemy import text
from db_connect import db_connect
from db_update import db_insert
def run():
try:
db_conn = db_connect()
if not db_conn:
return False
except Exception as e:
print(e)
qry = "SELECT *
FROM some_table
WHERE some_attr IN (:some_value);"
qry = text(qry)
search_run_qry = qry.bindparams(
some_value = 'abc'
)
result_list = db_conn.execute(qry).fetchall()
for result_item in result_list:
## do stuff like fetching data from api for every record in the query result
api_result = get_api_data(...)
## insert into db:
db_ins_status = db_insert(db_conn, api_result)
## ...
db_conn.close
run()
EDIT: Another question:
a) Is it ok in a loop, that does an update on every iteration to use the same connection, or would it be wiser to instead pass the engine to the run() function and call db_conn = engine.connect() and db_conn.close() just before and after each update?
b) I am thinking about using ThreadPoolExecutor instead of the loop for the API calls. Would this have implications on how to use the connection, i.e. can I use the same connection for multiple threads that are doing updates to the same table?
Note: I am not using the ORM feature mostly because I have a strong DWH/SQL background (though not so much as DBA) and I am used to writing even complex sql queries. I am thinking about switching to just using PyMySQL connector for that reason.
Thanks in advance!
Yes you can return/pass connection object as parameter but what is the aim of db_connect method, except testing connection ? As I see there is no aim of this db_connect method therefore I would recommend you to do this as I done it before.
I would like to share a code snippet from one of my project.
def create_record(sql_query: str, data: tuple):
try:
connection = mysql_obj.connect()
db_cursor = connection.cursor()
db_cursor.execute(sql_query, data)
connection.commit()
return db_cursor, connection
except Exception as error:
print(f'Connection failed error message: {error}')
and then using this one as for another my need
db_cursor, connection, query_data = fetch_data(sql_query, query_data)
and after all my needs close the connection with this method and method call.
def close_connection(connection, db_cursor):
"""
This method used to close SQL server connection
"""
db_cursor.close()
connection.close()
and calling method
close_connection(connection, db_cursor)
I am not sure can I share my github my check this link please. Under model.py you can see database methods and to see how calling them check it main.py
Best,
Hasan.
I am developing a web app using Angular 9 for frontend and Flask in python 3.8 for backend, everything was going well till i tried to connect the backend with the data base, where the code apparently its ok, flask start running succesfully but when i tried to use my endpoint for authentication, flask throws the error:
**local variable 'conn' referenced before assignment**
i have been checking some forums and so on but i dont understand what is going on. Thanks in advance for your help.
import pymysql
database_name = "dbx80"
database_user = "user_auth"
database_password = "Jan2019#"
def conection_database(db):
try:
conn = (pymysql.connect(
unix_socket="/cloudsql/servicisEX:us-central1:dbx43",
port = 3306,
user = database_user,
password = database_password,
database = database_name,
charset="utf8"
))
print(f"Connection {database_name} Succesful")
except Exception as e:
print(f"Error connecting to {database_name}")
return conn
Converting comment to an answer.
return conn is called no matter the result of the try except block.
If you put the return statement inside the try block, the error will not occur.
Example:
import pymysql
database_name = "dbx80"
database_user = "user_auth"
database_password = "Jan2019#"
def conection_database(db):
try:
conn = (pymysql.connect(
unix_socket="/cloudsql/servicisEX:us-central1:dbx43",
port = 3306,
user = database_user,
password = database_password,
database = database_name,
charset="utf8"
))
print(f"Connection {database_name} Succesful")
return conn
except Exception as e:
print(f"Error connecting to {database_name}")
I am trying to write function for updating database table columns (here column name will differ various point of time.) So while doing so, i am getting "operation error, Database locked". Could anyone suggest reason behind this.
For more clarification i am providing code as below:
def UpdateDB_ForFields(DB_name, DB_tableName, dev_name, devindex, colList, colValList):
try:
conn = None
cloudDBlockList[devindex].acquire()
pool_sema.acquire()
conn = sqlite3.connect("XYZ.db",timeout=20)
for i in range(len(colList)):
cursor = conn.execute("""UPDATE XYZ_settings SET {0} = ? WHERE {1} = ?;""".format(colList[i],"devName"),(colValList[i],dev_name))
conn.commit()
conn.close()
pool_sema.release()
cloudDBlockList[devindex].release()
except Exception as ex:
template = "An exception of type {0} occurred. Arguments:\n{1!r}"
message = template.format(type(ex).__name__, ex.args)
print(message)
logging.error(message)
if conn == None:
print("Error")
else:
conn.close()
pool_sema.release()
cloudDBlockList[devindex].release()
I am getting error while updating db as database locked. Please suggest the solution.
Im using python 3 and i have this loop, which iterate on this list -
self.settings["db"]["host"] = ["db-0", "db-1"]
My problem is that it seems to send in return self.conn the first option all the time,
db-0 instead of trying with db-1
I have 2 db container servers, and when i stop one of them - for example db-0 it should try psycopg2.connect with db-1
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
return self.conn
except Exception:
pass
if loop has not succeeded i dont want it to return self.conn, only if the try worked.
I also tried :
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
except Exception:
pass
return self.conn
You are looping within a try.
Do it the other way around,
push the try down within the loop.
The DbC contract the current code is attempting to offer
is to return a valid connection.
Let's make that more explicit, by writing a very simple helper.
We will spell out its contract in a docstring.
def first_true(iterable, default=False, pred=None):
# from https://docs.python.org/3/library/itertools.html
return next(filter(pred, iterable), default)
def get_conn_or_none(self, server):
"""Returns DB connection to server, or None if that's not possible."""
try:
return psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
except Exception:
return None
Now db_conn is simply:
def db_conn(self):
return first_true(map(self.get_conn_or_none, self.settings["db"]["host"]))
That uses the same logic as in your question.
You may want to have db_conn additionally check whether
all connection attempts failed, and raise fatal error in that case.
BTW, it's very odd that you're apparently storing a list of server hostnames
in self.settings["db"]["host"], given that an individual user / pw / db is
stored in the other fields.
Consider renaming that list to self.servers.
You can try to check the connection status before returning it:
def db_conn(self):
try:
for server in self.settings["db"]["host"]:
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
connect_timeout=5,
)
**if self.conn.isOk()**
return self.conn
except Exception:
pass
This code worked for me:
def db_conn(self):
for server in self.settings["db"]["host"]:
try:
self.print_info("TRYING", server)
self.conn = psycopg2.connect(
host=server,
user=self.settings["db"]["user"],
password=self.settings["db"]["password"],
database=self.settings["db"]["database"],
)
except:
self.print_info("SERVER DOWN", server)
continue
return self.conn
continue will continue the rest of the code if i get exception ( failed connection)
then it goes back to for loop with the second item in list.
I'm newbie in Python. I'm trying to use Python to connect MySQL Server. I wrote like guides from MySQL official page, it was OK. But, when I create a connector class, it raised the error "MySQL Connection not available"
Here is my class
import mysql.connector
from mysql.connector import errorcode
## BEGIN MySQL Connector Class
class MySQLConnector :
configs = {
"user":"root",
"password":"",
"host":"127.0.0.1",
"database":"python_db",
"raise_on_warnings": True
}
cursor = None
connection = None
## BEGIN Constructor
def __init__(self, configs = {}) :
if(any(configs)!=False) :
self.configs = configs
## END Constructor
## BEGIN Open
def open(self) :
try:
self.connection = mysql.connector.connect(self.configs)
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with your user name or password")
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exists")
else:
print(err)
finally:
self.connection.close()
return self.connection
## END Open
## BEGIN close connection
def close(self) :
self.cursor.close()
self.connection.close()
## END close connection
## BEGIN execute
def execute(self, query) :
if(self.connection == None) :
print("Connection is None")
return
self.cursor = self.connection.cursor()
if(self.cursor!=None) :
self.cursor.execute(query)
else:
print("Cursor is 'None'")
## END execute
## END MySQL Connector Class
## BEGIN RUN
objConnect = MySQLConnector()
objConnect.open()
objConnect.execute("SELECT * FROM User")
Please show me the way to solution and explained me why my code has error.
Thanks!
EDITED
Finally, mata and alecxe help me to solve this problem, I don't know which solution to be choosen. I summary here for someone has mistake like me:
1. Remove the finally statement.
2. Using ** in self.connection = mysql.connector.connect(**self.configs)
Even if you correct the error alecxe pointed out, your code still won't work.
The finally block ensures that each connection is closed before it is returned, no matter wheather there was an exception or not, so the open method only returns closed connections.
You are passing a dictionary object self.configs into mysql.connector.connect, though, according to docs, you should pass to it user, password and other arguments. Looks like you need to unpack configs:
self.connection = mysql.connector.connect(**self.configs)
Hope this is it.