Develop Reference

How to restore MySQL database using Python

I have a sql file generated during database backup process and I want to load all database content from that sql file to a different MySQL database (secondary database).
I have created a python function to load the whole database in that sql file but when I execute the function, I get an error
'str' object is not callable
Below is python script
def load_database_dump_to_secondary_mysql(file_path='db_backup_file.sql'):
query = f'source {file_path}'
try:
connection = mysql_hook.get_conn() # connection to secondary db
cursor = connection.cursor(query)
print('LOAD TO MYSQL COMPLETE')
except Exception as xerror:
print("LOAD ERROR: ", xerror)
NB: mysql_hook is an airflow connector that contains MySQL DB connection info such as Host, user/passwd, Database name. Also, I don't have connection to the primary database, I'm only receiving sql dump file.
What I'm I missing?

source is a client builtin command: https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html
It's not an SQL query that MySQL's SQL parser understands.
So you can't execute source using cursor.execute(), because that goes directly to the dynamic SQL interface.
You must run it using the MySQL command-line client as a subprocess:
subprocess.run(['mysql', '-e', f'source {file_path}'])
You might need other options to the mysql client, such as user, password, host, etc.

try this
import mysql.connector as m
# database which you want to backup
db = 'geeksforgeeks'
connection = m.connect(host='localhost', user='root',
password='123', database=db)
cursor = connection.cursor()
# Getting all the table names
cursor.execute('SHOW TABLES;')
table_names = []
for record in cursor.fetchall():
table_names.append(record[0])
backup_dbname = db + '_backup'
try:
cursor.execute(f'CREATE DATABASE {backup_dbname}')
except:
pass
cursor.execute(f'USE {backup_dbname}')
for table_name in table_names:
cursor.execute(
f'CREATE TABLE {table_name} SELECT * FROM {db}.{table_name}')

How to commit stored procedure execution by using pyodbc

I am trying to execute stored procedure by using pyodbc in databricks, after executing SP I tried to commit the connection but, commit is not happening. Here I am giving my code, please help me out from this issue.
import pyodbc
#### Connecting Azure SQL
def db_connection():
try:
username = "starsusername"
password = "password-db"
server = "server-name"
database_name = "db-name2"
port = "db-port"
conn=pyodbc.connect('Driver={ODBC Driver 17 for SQL server};SERVER=tcp:'+server+','+port+';DATABASE='+ database_name +';UID='+ username +';PWD='+ password)
cursor=conn.cursor()
return cursor, conn
except Exception as e:
print("Faild to Connect AZURE SQL: \n"+str(e))
cursor, conn = db_connection()
# conn1.autocommit=True
cursor.execute("delete from db.table_name")
cursor.execute("insert into db.table_name(BUSINESS_DATE) values('2021-10-02')")
cursor.execute("exec db.SP_NAME '20211023'")
conn.commit()
conn.close()
here I am commiting connection after SP excution. deletion and insertion is not happening at all. and I tried with cursor.execute("SET NOCOUNT ON; exec db.SP_NAME '20211023'") but it's also not working.
Thanks in Advance

If you check this document on pyodbc, you will find that -
To call a stored procedure right now, pass the call to the execute method using either a format your database recognizes or using the ODBC call escape format. The ODBC driver will then reformat the call for you to match the given database.
Note that after connection is set up or done, try doing conn.autocommit = True before calling your SP and it will help. By default it is false.
Executing the Stored Procedure.
You will be able to execute your stored procedure if you follow the below code snippet.
cursor = conn.cursor()
conn.autocommit = True
executesp = """EXEC yourstoredprocedure """
cursor.execute(executesp)
conn.commit()
Delete the Records in SQL Server
You can delete record as shown in the below example.
...#just an example
cursor.execute('''
DELETE FROM product
WHERE product_id in (5,6)
''')
conn.commit()
Don’t forget to add conn.commit() at the end of the code, to ensure that the command would get executed.
Insert record in SQL Server
The below snippet show how we can do the same.
...#just an example
cursor.execute("INSERT INTO EMP (EMPNO, ENAME, JOB, MGR) VALUES (535, 'Scott', 'Manager', 545)")
conn.commit()
I will suggest you to read the for following document for more information.
Delete Record Documentation.
Insert Record Document

CREATE PROCEDURE fails silently when called from pyodbc

I am unable to create a SQL Server stored procedure using Python's pyodbc. The command executes correctly and I get no error message however the stored procedure does not appear on the server
import pyodbc
host = 'myServer'
database = 'model'
conn = pyodbc.connect(
r'DRIVER={SQL Server Native Client 11.0};' +
r'SERVER=' + host + ';' +
r'DATABASE=' + database + ';' +
r'Trusted_Connection=yes'
)
cursor = conn.cursor()
sql = """
CREATE OR ALTER PROCEDURE [dbo].[Test] AS
SELECT 1
"""
cursor.execute(sql)
conn.close()

pyodbc connections default to having autocommit disabled as specified in Python's DB API 2.0 spec. In that mode, any changes to the database must be committed by calling commit() on the Connection.
If you want a connection with autocommit enabled, see this answer for details.

Python to SQL Server Insert

I'm trying to follow the method for inserting a Panda data frame into SQL Server that is mentioned here as it appears to be the fastest way to import lots of rows.
However I am struggling with figuring out the connection parameter.
I am not using DSN , I have a server name, a database name, and using trusted connection (i.e. windows login).
import sqlalchemy
import urllib
server = 'MYServer'
db = 'MyDB'
cxn_str = "DRIVER={SQL Server Native Client 11.0};SERVER=" + server +",1433;DATABASE="+db+";Trusted_Connection='Yes'"
#cxn_str = "Trusted_Connection='Yes',Driver='{ODBC Driver 13 for SQL Server}',Server="+server+",Database="+db
params = urllib.parse.quote_plus(cxn_str)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
conn = engine.connect().connection
cursor = conn.cursor()
I'm just not sure what the correct way to specify my connection string is. Any suggestions?

I have been working with pandas and SQL server for a while and the fastest way I found to insert a lot of data in a table was in this way:
You can create a temporary CSV using:
df.to_csv('new_file_name.csv', sep=',', encoding='utf-8')
Then use pyobdc and BULK INSERT Transact-SQL:
import pyodbc
conn = pyodbc.connect(DRIVER='{SQL Server}', Server='server_name', Database='Database_name', trusted_connection='yes')
cur = conn.cursor()
cur.execute("""BULK INSERT table_name
FROM 'C:\\Users\\folders path\\new_file_name.csv'
WITH
(
CODEPAGE = 'ACP',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)""")
conn.commit()
cur.close()
conn.close()
Then you can delete the file:
import os
os.remove('new_file_name.csv')
It was a second to charge a lot of data at once into SQL Server. I hope this gives you an idea.
Note: don't forget to have a field for the index. It was my mistake when I started to use this lol.

Connection string parameter values should not be enclosed in quotes so you should use Trusted_Connection=Yes instead of Trusted_Connection='Yes'.

Create a schema in SQL Server using pyodbc

I am using pyodbc to read from a SQL Server database and create analogous copies of the same structure in a different database somewhere else.
Essentially:
for db in source_dbs:
Execute('create database [%s]' % db) # THIS WORKS.
for schema in db:
# The following result in an error starting with:
# [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Execute('create schema [%s].[%s]' % (db, schema)
# Incorrect syntax near '.'
Execute('use [%s]; create schema [%s]' %(db, schema)
# CREATE SCHEMA' must be the first statement in a query batch.
In this example, you can assume that Execute creates a cursor using pyodbc and executes the argument SQL string.
I'm able to create the empty databases, but I can't figure out how to create the schemas within them.
Is there a solution, or is this a limitation of using pyodbc with MS SQL Server?
EDIT: FWIW - I also tried to pass the database name to Execute, so I could try to set the database name in the connection string. This doesn't work either - it seems to ignore the database name completely.

Python database connections usually default to having transactions enabled (autocommit == False) and SQL Server tends to dislike certain DDL commands being executed in a transaction.
I just tried the following and it worked for me:
import pyodbc
connStr = (
r"Driver={SQL Server Native Client 10.0};"
r"Server=(local)\SQLEXPRESS;"
r"Trusted_connection=yes;"
)
cnxn = pyodbc.connect(connStr, autocommit=True)
crsr = cnxn.cursor()
crsr.execute("CREATE DATABASE pyodbctest")
crsr.execute("USE pyodbctest")
crsr.execute("CREATE SCHEMA myschema")
crsr.close()
cnxn.close()