I currently have code that executes queries on data stored on a SQL Server database, such as the following:
import pyodbc
conn = pyodbc.connect(
r'DRIVER={SQL Server};'
r'SERVER=SQL2SRVR;'
r'DATABASE=DBO732;'
r'Trusted_Connection=yes;'
)
sqlstr = '''
SELECT Company, Street_Address, City, State
FROM F556
WHERE [assume complicated criteria statement here]
'''
crsr = conn.cursor()
for row in crsr.execute(sqlstr):
print(row.Company, row.Street_Address, row.City, row.State)
I can't find documentation online of whether pyodbc can (or is by default) running my queries on the SQL Server (as passthrough queries), or whether (if pyodbc can't do that) there is another way (maybe sqlalchemy or similar?) of doing that. Any insight?
Or is there a way to execute passthrough queries directly from Pandas?
If you are working with pandas and SQL Server then you should already have created a SQLAlchemy Engine object (usually named engine). To execute a raw DML statement you can use the construct
with engine.begin() as conn:
conn.execute("UPDATE table_name SET column_name ...")
print("table updated")
Related
I am currently developing a program in python that interacts with multiple database. I am using pyodbc to connect, and execute queries. One of the database is an azure database. I noticed sometimes the sent data is not updated in the database although the program run successfully and no error was thrown. Is there any practices that i should follow to make sure this doesn't happen or is this related to my code or db connection issue? I am a beginner. Would appreciate everyone's help thank you!
Also is the .commit() line should be run after every sql run?
The program should be updating a row of data in the database based on a condition, this particular query sometimes doesn't take effect, but no error was thrown. I also executed multiple queries after that, no issue was found for the next queries. It is successfully executed.
the query is a simple query which is
UPDATE DraftVReg SET VRStatus = 'Potential Duplicate Found' WHERE RowID = ?
I tried to reproduce your scenario on my end and was able to update the SQL row in the Azure SQL DB with Pyodbc module.
Yes, Its very necessary to use
conn.commit
to commit your changes inside a database after you perform operations such as Update or Insert inside Azure SQL DB programmatically.
1) Fetch Data with Select statement.
I was able to fetch the Table’s data successfully with Select * from ‘Tablename’ query inside pyodbc code before I try UPDATE statement.
import pyodbc
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};''SERVER=tcp:sqlservernamesql.database.windows.net,1433;''DATABASE=databasename; UID=siliconuser;PWD=Password;')
#conn.commit()
cursor = conn.cursor()
cursor.execute('Select * FROM StudentReviews')
#conn.commit()
for i in cursor:
print(i)
cursor.close()
conn.close()
Result:-
2) UPDATE the rows require conn.commit()
Code :-
import pyodbc
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};''SERVER=tcp:siliconserversql.database.windows.net,1433;''DATABASE=silicondb; UID=userid; PWD=Password;')
cursor = conn.cursor()
#cursor.execute('Select * FROM StudentReviews')
cursor.execute("UPDATE StudentReviews SET ReviewTime = ('7') WHERE ReviewText = ('SQL DB')")
conn.commit()
cursor.close()
conn.close()
Result:-
Update statement Executed successfully and the Table Row was updated in Azure SQL, Refer Below :-
3) With autocommit=true
Thank you #Gord thompson for the comment and suggestion!
Code :-
import pyodbc
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};''SERVER=tcp:siliconserversql.database.windows.net,1433;''DATABASE=silicondb; UID=username; PWD=Password;', autocommit=True)
#conn.commit()
cursor = conn.cursor()
cursor.execute("UPDATE StudentReviews SET ReviewTime = ('8') WHERE ReviewText = ('SQL DB')")
cursor.close()
conn.close()
Results :- With autocommit=true, You do not need to add conn.commit everytime you update the SQL DB.
I have a sql file generated during database backup process and I want to load all database content from that sql file to a different MySQL database (secondary database).
I have created a python function to load the whole database in that sql file but when I execute the function, I get an error
'str' object is not callable
Below is python script
def load_database_dump_to_secondary_mysql(file_path='db_backup_file.sql'):
query = f'source {file_path}'
try:
connection = mysql_hook.get_conn() # connection to secondary db
cursor = connection.cursor(query)
print('LOAD TO MYSQL COMPLETE')
except Exception as xerror:
print("LOAD ERROR: ", xerror)
NB: mysql_hook is an airflow connector that contains MySQL DB connection info such as Host, user/passwd, Database name. Also, I don't have connection to the primary database, I'm only receiving sql dump file.
What I'm I missing?
source is a client builtin command: https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html
It's not an SQL query that MySQL's SQL parser understands.
So you can't execute source using cursor.execute(), because that goes directly to the dynamic SQL interface.
You must run it using the MySQL command-line client as a subprocess:
subprocess.run(['mysql', '-e', f'source {file_path}'])
You might need other options to the mysql client, such as user, password, host, etc.
try this
import mysql.connector as m
# database which you want to backup
db = 'geeksforgeeks'
connection = m.connect(host='localhost', user='root',
password='123', database=db)
cursor = connection.cursor()
# Getting all the table names
cursor.execute('SHOW TABLES;')
table_names = []
for record in cursor.fetchall():
table_names.append(record[0])
backup_dbname = db + '_backup'
try:
cursor.execute(f'CREATE DATABASE {backup_dbname}')
except:
pass
cursor.execute(f'USE {backup_dbname}')
for table_name in table_names:
cursor.execute(
f'CREATE TABLE {table_name} SELECT * FROM {db}.{table_name}')
I am using pyodbc to read from a SQL Server database and create analogous copies of the same structure in a different database somewhere else.
Essentially:
for db in source_dbs:
Execute('create database [%s]' % db) # THIS WORKS.
for schema in db:
# The following result in an error starting with:
# [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Execute('create schema [%s].[%s]' % (db, schema)
# Incorrect syntax near '.'
Execute('use [%s]; create schema [%s]' %(db, schema)
# CREATE SCHEMA' must be the first statement in a query batch.
In this example, you can assume that Execute creates a cursor using pyodbc and executes the argument SQL string.
I'm able to create the empty databases, but I can't figure out how to create the schemas within them.
Is there a solution, or is this a limitation of using pyodbc with MS SQL Server?
EDIT: FWIW - I also tried to pass the database name to Execute, so I could try to set the database name in the connection string. This doesn't work either - it seems to ignore the database name completely.
Python database connections usually default to having transactions enabled (autocommit == False) and SQL Server tends to dislike certain DDL commands being executed in a transaction.
I just tried the following and it worked for me:
import pyodbc
connStr = (
r"Driver={SQL Server Native Client 10.0};"
r"Server=(local)\SQLEXPRESS;"
r"Trusted_connection=yes;"
)
cnxn = pyodbc.connect(connStr, autocommit=True)
crsr = cnxn.cursor()
crsr.execute("CREATE DATABASE pyodbctest")
crsr.execute("USE pyodbctest")
crsr.execute("CREATE SCHEMA myschema")
crsr.close()
cnxn.close()
I am trying to create a database using pyodbc, however, I cannot find it seems to be paradox as the pyodbc needs to connect to a database first, and the new database is created within the linked one. Please correct me if I am wrong.
In my case, I used following code to create a new database
conn = pyodbc.connect("driver={SQL Server};server= serverName; database=databaseName; trusted_connection=true")
cursor = conn.cursor()
sqlcommand = """
CREATE DATABASE ['+ #IndexDBName +'] ON PRIMARY
( NAME = N'''+ #IndexDBName+''', FILENAME = N''' + #mdfFileName + ''' , SIZE = 4000KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB )
LOG ON
( NAME = N'''+ #IndexDBName+'_log'', FILENAME = N''' + #ldfFileName + ''' , SIZE = 1024KB , MAXSIZE = 100GB , FILEGROWTH = 10%)'
"""
cursor.execute(sqlcommand)
cursor.commit()
conn.commit()
The above code works without errors, however, there is no database created.
So how can I create a database using pyodbc?
Thanks a lot.
If you try to create a database with the default autocommit value for the connection, you should receive an error like the following. If you're not seeing this error message, try updating the SQL Server native client for a more descriptive message:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0]
[SQL Server]CREATE DATABASE statement not allowed within multi-statement transaction.
(226) (SQLExecDirectW)')
Turn on autocommit for the connection to resolve:
conn = pyodbc.connect("driver={SQL Server};server=serverName; database=master; trusted_connection=true",
autocommit=True)
Note two things:
autocommit is not part of the connection string, it is a separate keyword passed to the connect function
specify the initial connection database context is the master system database
As an aside, you may want to check the #IndexDBName, #mdfFileName, and #ldfFileName are being appropriately set in your T-SQL. With the code you provided, a database named '+ #IndexDBName +' would be created.
The accepted answer did not work for me but I managed to create a database using the following code on Ubuntu:
conn_str = r"Driver={/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.9.so.1.1};" + f"""
Server={server_ip};
UID=sa;
PWD=passwd;
"""
conn = pyodbc.connect(conn_str, autocommit=True)
cursor = conn.cursor()
cursor.execute(f"CREATE DATABASE {db_name}")
Which uses the default "master database" when connecting. You can check if the dataset is created by this query:
SELECT name FROM master.sys.databases
I'm tasked with obtaining data from two MS SQL databases on the same server so i can run a single query that uses info from both databases simultaneously. I am trying to achieve this in python 2.7 with pyodbc 3.0.7. My query would look like this:
Select forcast.WindGust_Forecast, forcast.Forecast_Date, anoSection.SectionName, refTable.WindGust
FROM [EO1D].[dbo].[Dashboard_Forecast] forcast
JOIN [EO1D].[dbo].[Dashboard_AnoSections] anoSection
ON forcast.Section_ID = anoSection.Record_ID
JOIN [EO1D].[dbo].[Dashboard_AnoCircuits] anoCircuits
ON anoSection.Circuit_Number = anoCircuits.Circuit_Number
JOIN [FTSAutoCaller].[dbo].[ReferenceTable] refTable
ON anoCircuits.StationCode = refTable.StationCode
Where refTable.Circuit IS NOT NULL and refTable.StationCode = 'sil'
the typical connection for pyodbc looks like:
cnxn = pyodbc.connect('DRIVER{SQLServer};SERVER=SQLSRV01;DATABASE=DATABASE;UID=USER;PWD=PASSWORD')
Which would only allow access to the database name provided.
how would I go about setting up a connection that allows me access to both databases so this query can be ran. The two database names in my case are EO1D and FTSAutoCaller.
you're overthinking it. If you setup the connection as you did above, and then simply pass the sql along to a cursor it should work.
import pyodbc
conn_string = '<removed>'
conn = pyodbc.connect(conn_string)
cur = conn.cursor()
query = 'select top 10 * from table1 t1 inner join database2..table2 t2 on t1.id = t2.id'
cur.execute(query)
and you are done (tested in my own environment, clearly the connection string and query were different, but it did work.)
The query takes care of its self although I only referenced one of the tables in the connection the query didnt have an issue connecting to both of the database. Not 100% sure but im assuming it worked because of the prefixed in "[ ]"