I have written a small tool which migrates data from SQL Server to Postgres.
To make things work, I did it quick and dirty using string concats since I had a lot of other issues to solve and I did not want to bother with the SQL at the time. But now that everything is sorted out, I want to do things porperly in the SQL department.
The unsafe quick and dirty version :
import pyodbc
# this is the bad example DON'T do this
def getDataFromTable(self,table):
"""
Gets all data from the specified Table.
table -- Table name as string
"""
cursor = self.cursor
SQL = f"""SELECT * FROM {table}""" ## DON'T do this
cursor.execute(SQL)
rows = cursor.fetchall()
records = []
for row in rows:
records.append(list(row))
return records
This works perfectly fine, but is a SQL injection waiting to happen.
I want to build something like this (I have omitted the unchanged parts):
...
cursor = self.cursor
SQL = f"""SELECT * FROM ?""" # Use parameters insted of string concats
cursor.execute(SQL, table) # pass parameters to the execute method.
rows = cursor.fetchall()
...
This looks nice an safe, but also does not work. Following error pops up :
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Die #P1-Tabellenvariable muss deklariert werden. (1087) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Anweisung(en) konnte(n) nicht vorbereitet werden. (8180)')
It's in German but roughly translates to: table variables must be declared, statement could not be prepared.
How can I pass a variable into the execute method to take the identifiers place safely?
You can not have a variable for table name in SQL Server. You can generate a string in python and then execute it.
table_name = 'table1'
query = "Select * from %s" % table_name
Or you can use Dynamic SQL.
Related
I am trying to execute below SQL query by introducing 2 variables in Python but getting error,
Original Query:
ALTER DATABASE db1 MODIFY (SERVICE_OBJECTIVE = 'DW300');
I want to use variables for db1 and 'DW300'. I have tried below statement in Python and got error.
dwu = 'DW100'
sqlpool = 'db1'
cursor.execute("""ALTER DATABASE ? MODIFY (SERVICE_OBJECTIVE = ?) """, (sqlpool, dwu))
Error: pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '#P1'. (102) (SQLExecDirectW)")
I use sqlite and it accepts this answe.
try the use of fstrings
f"""Alter Database {sqlpool} modify (service objective = {dwu}) """
I am trying to read data from SQL server into pandas data frame. Below is the code.
def get_data(size):
con = pyodbc.connect(r'driver={SQL Server}; server=SPROD_RPT01; database=Reporting')
cur = con.cursor()
db_cmd = "select distinct top %s * from dbo.KrishAnalyticsAllCalls" %size
res = cur.execute(db_cmd)
sql_out = pd.read_sql_query(db_cmd, con, chunksize=10**6)
frames = [chunk for chunk in sql_out]
df_sql = pd.concat(frames)
return df_sql
df = get_data(5000000)
I am getting following error:
pandas.io.sql.DatabaseError: Execution failed on sql 'select distinct
top 500000 * from dbo.KrishAnalyticsAllCalls': ('HY000', '[HY000]
[Microsoft][ODBC SQL Server Driver]Connection is busy with results for
another hstmt (0) (SQLExecDirectW)')
I had executed the function before and interrupted the execution with ctrl+k as I wanted to make a change in the function. Now, after making the change when I'm trying to execute the function I am getting the above error.
How can I kill that connection/IPython Kernel since I don't know of any IPython Kernel running executing the query in the function?
I was facing the same issue. This was fixed when I used fetchall() function. The following the code that I used.
import pypyodbc as pyodbc
def connect(self, query):
con = pyodbc.connect(self.CONNECTION_STRING)
cursor = con.cursor()
print('Connection to db successful')
cmd = (query)
results = cursor.execute(cmd).fetchall()
df = pd.read_sql(query, con)
return df, results
Using cursor.execute(cmd).fetchall() instead of cursor.execute(cmd) resolved it.
Hope this helps.
The issue is due to cursor being executed just before the pd.read_sql_query() command .
Pandas is using the connection and SQL String to get the data . DB Cursor is not required .
#res = cur.execute(db_cmd)
sql_out = pd.read_sql_query(db_cmd, con, chunksize=10**6)
print(sql_out)
Most likely you haven't connected to the SQL server yet. Or, you connected in a previous instance for a different SQL query that was run. Either way, you need to re-establish the connection.
import pyodbc as pyodbc
conn = pyodbc.connect('Driver={YOUR_DRIVER};''Server=YOUR_SERVER;''Database=YOUR_DATABASE;''Trusted_Connection=yes')
Then execute your SQL:
sql = conn.cursor()
sql.execute("""ENTER YOUR SQL""")
Then transform into Pandas:
df = pd.DataFrame.from_records(sql.fetchall(),columns=[desc[0] for desc in sql.description])
I am using pyodbc to read from a SQL Server database and create analogous copies of the same structure in a different database somewhere else.
Essentially:
for db in source_dbs:
Execute('create database [%s]' % db) # THIS WORKS.
for schema in db:
# The following result in an error starting with:
# [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Execute('create schema [%s].[%s]' % (db, schema)
# Incorrect syntax near '.'
Execute('use [%s]; create schema [%s]' %(db, schema)
# CREATE SCHEMA' must be the first statement in a query batch.
In this example, you can assume that Execute creates a cursor using pyodbc and executes the argument SQL string.
I'm able to create the empty databases, but I can't figure out how to create the schemas within them.
Is there a solution, or is this a limitation of using pyodbc with MS SQL Server?
EDIT: FWIW - I also tried to pass the database name to Execute, so I could try to set the database name in the connection string. This doesn't work either - it seems to ignore the database name completely.
Python database connections usually default to having transactions enabled (autocommit == False) and SQL Server tends to dislike certain DDL commands being executed in a transaction.
I just tried the following and it worked for me:
import pyodbc
connStr = (
r"Driver={SQL Server Native Client 10.0};"
r"Server=(local)\SQLEXPRESS;"
r"Trusted_connection=yes;"
)
cnxn = pyodbc.connect(connStr, autocommit=True)
crsr = cnxn.cursor()
crsr.execute("CREATE DATABASE pyodbctest")
crsr.execute("USE pyodbctest")
crsr.execute("CREATE SCHEMA myschema")
crsr.close()
cnxn.close()
I try to do an update query using a left join on another mdb.
Into a cursor for the first MDB, I execute this query:
update table as ori
left join (select *
from param in "E:/Jeter/param_141114.mdb"
where zone = '1H005') param
on ori.dep_sur = param.dsu_co
set ori.texture = param.textu where mid(ori.type,4,1) in ('0','7','8')
When I launch this query from Microsoft Access, no problem, the query is applied.
When I launch this query from python 2.7 with pyodbc, here my result translated from french:
ProgrammingError ('42000', "[42000] [Microsoft] [ODBC Microsoft Access
Driver] The database engine can not find [E: /Jeter/param_141114.mdb]
'Make sure the name. parameter or alias is valid, he does not
understand character or incorrect punctuation and that it is not too
long. (-1002) (SQLExecDirectW) ")
Some ideas?
Apparently the SELECT ... FROM TableName IN "FileName" ... syntax is not available to ODBC queries from external applications. However, I just tried the following variation and it worked for me (Python 2.7 and pyodbc):
sql = """
update tableau as ori
left join (select *
from [C:/__tmp/test.mdb].param
where zone = '1H005') param
on ori.dep_sur = param.dsu_co
set ori.texture = param.textu
"""
crsr = db.execute(sql)
crsr.commit()
i'm trying to upload an image to MS SQL web-server in Linux(raspbian) environment using python language. so far i had able connect to MS Sql and also i had create a table. And im using pyodbc.
#! /user/bin/env python
import pyodbc
dsn = 'nicedcn'
user = myid
password = mypass
database = myDB
con_string = 'DSN=%s;UID=%s;PWD=%s;DATABASE=%s;' % (dsn, user, password, database)
cnxn = pyodbc.connect(con_string)
cursor = cnxn.cursor()
string = "CREATE TABLE Database1([image name] varchar(20), [image] varbinary(max))"
cursor.execute(string)
cnxn.commit()
this part complied without any error. that means i have successfully created a table isn't? or is there any issue?
i try to upload image as this way.
with open('new1.jpg','rb') as f:
bindata = f.read()
cursor.execute("insert into Database1(image name, image) values (?,?)", 'new1', bindata)
cnxn.commit()
i get the error on this part. and it pyodbc.ProgrammingError: ('42000', '[42000] [FreeTDS] [SQL Server] Satement(s) could not be prepared. (8180) (SQLParamData)')
can some one help me please. thank you
Your parameters must be passed in as one sequence, not as two separate arguments. A tuple will do nicely here:
cursor.execute(
"insert into Database1([image name], image) values (?,?)",
('new1', pyodbc.Binary(bindata)))
Note that you also need to quote the image name column correctly, and wrap the data in a pyodbc.Binary() object; this will produce the right datatype for your Python version (bytearray or bytes).