I am writing a python script that does the following as a part a transaction.
Creates a new database.
Creates new tables using a schema.sql file.
Copies the data from the master DB to this new DB using insert into select * from master.table_name... like SQL statements.
Commits the txn, in the else block, if everything goes right. Rollback the txn, in the except block, if something goes wrong.
Close the connection in the finally block.
However, while testing, I found out that rollback isn't working. If an exception is raised after DB is created, even after rollback, DB is created. If an exception is raised after inserting data into a few tables with some tables remaining, calling rollback in the except block does not revert the inserted data. The script looks like this:
import mysql.connector
try:
conn = mysql.connector.connect(host='localhost', port=3306,
user=USERNAME, password=PASSWORD,
autocommit=False)
cursor.execute("START TRANSACTION;")
cursor.execute(f"DROP DATABASE IF EXISTS {target_db_name};")
cursor.execute(f"CREATE DATABASE {target_db_name};")
cursor.execute(f"USE {target_db_name};")
with open(SCHEMA_LOCATION) as f:
schema_query = f.read()
commands = schema_query.split(";")
for command in commands:
cursor.execute(command)
for query in QUERIES:
cursor.execute(f"{query}{org_Id};")
except Exception as error:
conn.rollback()
else:
conn.commit() # cursor.execute("COMMIT")
finally:
if conn.is_connected():
cursor.close()
conn.close()
Below are the details of the setup
Python3
mysql-connector-python==8.0.32
MySQL 5.7
Storage Engine: InnoDB
Related
I am trying to execute stored procedure by using pyodbc in databricks, after executing SP I tried to commit the connection but, commit is not happening. Here I am giving my code, please help me out from this issue.
import pyodbc
#### Connecting Azure SQL
def db_connection():
try:
username = "starsusername"
password = "password-db"
server = "server-name"
database_name = "db-name2"
port = "db-port"
conn=pyodbc.connect('Driver={ODBC Driver 17 for SQL server};SERVER=tcp:'+server+','+port+';DATABASE='+ database_name +';UID='+ username +';PWD='+ password)
cursor=conn.cursor()
return cursor, conn
except Exception as e:
print("Faild to Connect AZURE SQL: \n"+str(e))
cursor, conn = db_connection()
# conn1.autocommit=True
cursor.execute("delete from db.table_name")
cursor.execute("insert into db.table_name(BUSINESS_DATE) values('2021-10-02')")
cursor.execute("exec db.SP_NAME '20211023'")
conn.commit()
conn.close()
here I am commiting connection after SP excution. deletion and insertion is not happening at all. and I tried with cursor.execute("SET NOCOUNT ON; exec db.SP_NAME '20211023'") but it's also not working.
Thanks in Advance
If you check this document on pyodbc, you will find that -
To call a stored procedure right now, pass the call to the execute method using either a format your database recognizes or using the ODBC call escape format. The ODBC driver will then reformat the call for you to match the given database.
Note that after connection is set up or done, try doing conn.autocommit = True before calling your SP and it will help. By default it is false.
Executing the Stored Procedure.
You will be able to execute your stored procedure if you follow the below code snippet.
cursor = conn.cursor()
conn.autocommit = True
executesp = """EXEC yourstoredprocedure """
cursor.execute(executesp)
conn.commit()
Delete the Records in SQL Server
You can delete record as shown in the below example.
...#just an example
cursor.execute('''
DELETE FROM product
WHERE product_id in (5,6)
''')
conn.commit()
Don’t forget to add conn.commit() at the end of the code, to ensure that the command would get executed.
Insert record in SQL Server
The below snippet show how we can do the same.
...#just an example
cursor.execute("INSERT INTO EMP (EMPNO, ENAME, JOB, MGR) VALUES (535, 'Scott', 'Manager', 545)")
conn.commit()
I will suggest you to read the for following document for more information.
Delete Record Documentation.
Insert Record Document
I tried a lot however I am unable to copy data available as json file in S3 bucket(I have read only access to the bucket) to Redshift table using python boto3. Below is the python code which I am using to copy the data. Using the same code I was able to create the tables in which I am trying to copy.
import configparser
import psycopg2
from sql_queries import create_table_queries, drop_table_queries
def drop_tables(cur, conn):
for query in drop_table_queries:
cur.execute(query)
conn.commit()
def create_tables(cur, conn):
for query in create_table_queries:
cur.execute(query)
conn.commit()
def main():
try:
config = configparser.ConfigParser()
config.read('dwh.cfg')
# conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
conn = psycopg2.connect(
host=config.get('CLUSTER', 'HOST'),
database=config.get('CLUSTER', 'DB_NAME'),
user=config.get('CLUSTER', 'DB_USER'),
password=config.get('CLUSTER', 'DB_PASSWORD'),
port=config.get('CLUSTER', 'DB_PORT')
)
cur = conn.cursor()
#drop_tables(cur, conn)
#create_tables(cur, conn)
qry = """copy DWH_STAGE_SONGS_TBL
from 's3://udacity-dend/song-data/A/A/A/TRAAACN128F9355673.json'
iam_role 'arn:aws:iam::xxxxxxx:role/MyRedShiftRole'
format as json 'auto';"""
print(qry)
cur.execute(qry)
# execute a statement
# print('PostgreSQL database version:')
# cur.execute('SELECT version()')
#
# # display the PostgreSQL database server version
# db_version = cur.fetchone()
# print(db_version)
print("Executed successfully")
cur.close()
conn.close()
# close the communication with the PostgreSQL
except Exception as error:
print("Error while processing")
print(error)
if __name__ == "__main__":
main()
I don't see any error in the Pycharm console but I see Aborted status in the redshift query console. I don't see any reason why it has been aborted(or I don't know where to look for that)
Other thing that I have noticed is when I run the copy statement in Redshift query editor , it runs fine and data gets moved into the table. I tried to delete and recreate the cluster but no luck. I am not able to figure what I am doing wrong. Thank you
Quick read - it looks like you haven't committed the transaction and the COPY is rolled back when the connection closes. You need to either change the connection configuration to be in "autocommit" or add an explicit "commit()".
I would like to run below postgresql function queries in python but I keep getting error message says
cursor = connection.cursor()
cursor.execute("SELECT
ST_Distance_sphere(st_makepoint(32.836956,39.925018)
,st_makepoint(28.990174,41.036857))")
df = cursor.fetchall()
df
InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
How can I fix this? Thank you.
According to psycopg docs, you probably have an error in your command (SQL).
There was a problem in the previous command to the database, which
resulted in an error. The database will not recover automatically from
this condition: you must run a rollback() before sending new commands
to the session (if this seems too harsh, remember that PostgreSQL
supports nested transactions using the SAVEPOINT command).
I highly recommend using try/except/finally clause in your database connections. Or use with statement.
Here is an example from http://www.postgresqltutorial.com/postgresql-python/transaction/:
conn = psycopg2.connect(dsn)
# transaction 1
with conn:
with conn.cursor() as cur:
cur.execute(sql)
# transaction 2
with conn:
with conn.cursor() as cur:
cur.execute(sql)
conn.close()
I'm still using Flask-mysql.
I'm getting the database context (the mysql variable) just fine, and can query on the database / get results. It's only the insert that is not working: it's not complaining (throwing Exceptions). It returns True from the insert method.
This should be done inserting the record when it commits, but for some reason, as I watch the MySQL database with MySQL Workbench, nothing is getting inserted into the table (and it's not throwing exceptions from the insert method):
I'm passing in this to insertCmd:
"INSERT into user(username, password) VALUES ('test1','somepassword');"
I've checked the length of the column in the database, and copied the command into MySQL Workbench (where it successfully inserts the row into the table).
I'm at a loss. The examples I've seen all seem to follow this format, and I have a good database context. You can see other things I've tried in the comments.
def insert(mysql, insertCmd):
try:
#connection = mysql.get_db()
cursor = mysql.connect().cursor()
cursor.execute(insertCmd)
mysql.connect().commit()
#mysql.connect().commit
#connection.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
You need to keep a handle to the connection; you keep overriding it in your loop.
Here is a simplified example:
con = mysql.connect()
cursor = con.cursor()
def insert(mysql, insertCmd):
try:
cursor.execute(insertCmd)
con.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
If mysql is your connection, then you can just commit on that, directly:
def insert(mysql, insertCmd):
try:
cursor = mysql.cursor()
cursor.execute(insertCmd)
mysql.commit()
return True
except Exception as e:
print("Problem inserting into db: " + str(e))
return False
return False
Apparently, you MUST separate the connect and cursor, or it won't work.
To get the cursor, this will work: cursor = mysql.connect().cursor()
However, as Burchan Khalid so adeptly pointed out, any attempt after that to make a connection object in order to commit will wipe out the work you did using the cursor.
So, you have to do the following (no shortcuts):
connection = mysql.connect()
cursor = connection.cursor()
cursor.execute(insertCmd)
connection.commit()
I'm having a problem where I have a SQL statement that when run in my MySQL Workbench executes properly, but when run with python's mysql package function cursor.execute() doesn't work. The problem SQL statement is:
REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;
The statement is supposed to copy all data from mmm_master into mmm_dev. The following the python code that I'm using to execute the query:
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
def examine(cursor, cnx):
try:
qry = cursor.execute("REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;")
except mysql.connector.Error as err:
print("Failed to select everything")
exit(1)
MySQL Python libraries are PEP 249-compliant:
.commit () Commit any pending transaction to the database.
Note that if the database supports an auto-commit feature, this must
be initially off. An interface method may be provided to turn it back
on.
Database modules that do not support transactions should implement
this method with void functionality.
Call cnx.commit()
Call examine(). Debug with print():
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
def examine(cursor):
print("[DEBUG] 1: before query")
try:
qry = cursor.execute("REPLACE INTO mmm_dev.samp_wp_links SELECT * FROM mmm_master.samp_wp_links;")
print("[DEBUG] 2: after query")
except mysql.connector.Error as err:
print("Failed to select everything %s" % err)
exit(1)
print("[DEBUG] 3: success")
examine(cursor)
You have to do a commit.
mydb.commit()
in order to save the changes.