With python, I am trying to read a table from SQL Server, populate one field and load the resulting table back to SQL Server as a new table. I'm using "petl".
I can read the table with no problems, I populate the field successfully, but then I get an error when I try to load the resulting table back to SQL Server.
This is the python code I'm using:
import petl as etl
import pyodbc
def populate_field(z, row):
...
con = pyodbc.connect('Trusted_Connection=yes', driver='{SQL Server}', server=r'my_server', database='my_db')
query = r'SELECT TOP 10 * FROM my_table'
tab = etl.fromdb(con, query)
tab = etl.convert(tab, 'my_field', populate_field, pass_row=True)
etl.todb(tab, con, 'my_new_table')
And this is the error message I get on "etl.todb(tab, con, 'my_new_table')"
Error: ('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)')
Related
First I query my cosmos data and put it into a dataframe. Then I connect to my database and try using to_sql:
params = urllib.parse.quote_plus(r'Driver=SQL Server;Server={SERVERNAME},1433;Database=cosmosTest;Trusted_Connection=yes;TrustServerCertificate=no;Connection Timeout=0;')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str,echo=True)
conn = engine.connect()
df.to_sql('Document', conn, if_exists='replace', index = False)
When I try and run my script I run into this error message
('HY004', '[HY004] [Microsoft][ODBC SQL Server Driver]Invalid SQL data type (0) (SQLBindParameter)')
I'm not sure if I'm receiving this error due to something I'm doing wrong or if the data I'm querying from cosmos doesn't play well with SQL Server. I've checked a couple other posts about this issue, but they didn't really pertain to me. Any suggestions on what I can try here?
I can do a df.to_slq on my local instance of SQL Server just fine. I am getting stuck when trying to do the same df.to_sll using Python and Azure SQL Server. I thought it would essentially be done like this.
import urllib.parse
params = urllib.parse.quote_plus(
'Driver=%s;' % '{ODBC Driver 17 for SQL Server}' +
'Server=%s,1433;' % 'ryan-server.database.windows.net' +
'Database=%s;' % 'ryan_sql_db' +
'Uid=%s;' % 'UN' +
'Pwd={%s};' % 'PW' +
'Encrypt=no;' +
'TrustServerCertificate=no;'
)
from sqlalchemy.engine import create_engine
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
connection = engine.connect()
connection
all_data.to_sql('health', engine, if_exists='append', chunksize=100000, method=None,index=False)
That is giving me this error.
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n (10060) (SQLExecDirectW); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10060)')
[SQL: INSERT INTO health ([0], [Facility_BU_ID], [Code_Type], [Code], [Description], [UB_Revenue_Code], [UB_Revenue_Description], [Gross_Charge], [Cash_Charge], [Min_Negotiated_Rate], [Max_Negotiated_Rate], etc., etc., etc.
I found this link today:
https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-sql-server?view=sql-server-ver15
I tried to do something similar, like this.
import pyodbc
import pandas as pd
df = all_data
# server = 'myserver,port' # to specify an alternate port
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute(all_data.to_sql('health', cnxn, if_exists='append', chunksize=100000, method=None,index=False))
cnxn.commit()
cursor.close()
When I run that, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
What I'm really hoping to to is df.to_sql, not Insert Into. I am working in Spyder and trying to send the data from my local machine to the cloud.
I read the two links below, and got it working.
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-set-database-firewall-rule-azure-sql-database?view=azuresqldb-current
https://www.virtual-dba.com/blog/firewalls-database-level-azure-sql/
Basically, you need to open your command window on your local machine, enter 'ipconfig', and grab two IP addresses. Then, enter those into SQL Server in Azure.
EXECUTE sp_set_database_firewall_rule
N'health',
'192.0.1.1',
'192.0.0.5';
Finally, run the small script below, in SQL Server, to confirm that the changes were made correctly.
USE [ryan_sql_db]
GO
SELECT * FROM sys.database_firewall_rules
ORDER BY modify_date DESC
I'm ingesting data from Salesforce to Azure SQL Database using Python with pyodbc.
I make a first connexion with Salesforce as shown bellow:
cnxn = pyodbc.connect('DRIVER={Devart ODBC Driver for Salesforce};User ID=xxx;Password=xxx;Security Token=xxx')
Then I import Salesforce data, as shown bellow:
cursor = cnxn.cursor()
cursor.execute("select * from X where Y > 'VALUE'")
row = cursor.fetchall()
After that I make a second connexion with the destination which is Azure SQL Database, as shown bellow:
cnxn = pyodbc.connect('DRIVER={Devart ODBC Driver for SQL Azure};Server=xxx;Database=xxx;Port=1433;User ID=xxx;Password=xxx')
Until now, everything is working fine. But when I try to insert the output that I got from Salesforce (in variable row) I face data type problems, from which we can cite:
Tabulations "\t"
Back to the line sign (in Azure SQL Database is CHAR(13) + CHAR(10) +)
Charaters that contains quote (e.g. "big data's technology")
Here is how I launch the insertion query:
cursor.executemany('INSERT INTO dbo.Account (Column_a,Column_b,Column_c) VALUES (?,?,?,?)', row)
cursor.commit()
Here is the first error I get:
pyodbc.Error: ('HY000', '[HY000] [Devart][ODBC][Microsoft SQL Azure]Statement(s) could not be prepared.\r\nMust declare the scalar variable "#_39".\r\nLine 1: Specified scale 14 is invalid. (0) (SQLExecDirectW)')
This issue was apparently caused by a defect in the
Devart ODBC Driver for SQL Azure
Using Microsoft's
ODBC Driver 17 for SQL Server
solved the problem .
I am trying to read data from SQL server into pandas data frame. Below is the code.
def get_data(size):
con = pyodbc.connect(r'driver={SQL Server}; server=SPROD_RPT01; database=Reporting')
cur = con.cursor()
db_cmd = "select distinct top %s * from dbo.KrishAnalyticsAllCalls" %size
res = cur.execute(db_cmd)
sql_out = pd.read_sql_query(db_cmd, con, chunksize=10**6)
frames = [chunk for chunk in sql_out]
df_sql = pd.concat(frames)
return df_sql
df = get_data(5000000)
I am getting following error:
pandas.io.sql.DatabaseError: Execution failed on sql 'select distinct
top 500000 * from dbo.KrishAnalyticsAllCalls': ('HY000', '[HY000]
[Microsoft][ODBC SQL Server Driver]Connection is busy with results for
another hstmt (0) (SQLExecDirectW)')
I had executed the function before and interrupted the execution with ctrl+k as I wanted to make a change in the function. Now, after making the change when I'm trying to execute the function I am getting the above error.
How can I kill that connection/IPython Kernel since I don't know of any IPython Kernel running executing the query in the function?
I was facing the same issue. This was fixed when I used fetchall() function. The following the code that I used.
import pypyodbc as pyodbc
def connect(self, query):
con = pyodbc.connect(self.CONNECTION_STRING)
cursor = con.cursor()
print('Connection to db successful')
cmd = (query)
results = cursor.execute(cmd).fetchall()
df = pd.read_sql(query, con)
return df, results
Using cursor.execute(cmd).fetchall() instead of cursor.execute(cmd) resolved it.
Hope this helps.
The issue is due to cursor being executed just before the pd.read_sql_query() command .
Pandas is using the connection and SQL String to get the data . DB Cursor is not required .
#res = cur.execute(db_cmd)
sql_out = pd.read_sql_query(db_cmd, con, chunksize=10**6)
print(sql_out)
Most likely you haven't connected to the SQL server yet. Or, you connected in a previous instance for a different SQL query that was run. Either way, you need to re-establish the connection.
import pyodbc as pyodbc
conn = pyodbc.connect('Driver={YOUR_DRIVER};''Server=YOUR_SERVER;''Database=YOUR_DATABASE;''Trusted_Connection=yes')
Then execute your SQL:
sql = conn.cursor()
sql.execute("""ENTER YOUR SQL""")
Then transform into Pandas:
df = pd.DataFrame.from_records(sql.fetchall(),columns=[desc[0] for desc in sql.description])
I am using pyodbc in python 2.7 with MS SQL Server 2008R.
Here is my code for creating a database (the SQL code which works fine in SQL alone, but crash when executed in python)
SQL_command = """
IF EXISTS(SELECT * FROM sys.databases WHERE [name] = 'NewDatabase')
DROP DATABASE NewDatabase
"""
conn.cursor.execute(SQL_command)
SQL_command = """
CREATE DATABASE NewDatabase
ON
(
NAME = 'NewDatabase_data'
, FILENAME='D:\MSSQL\DATA\NewDatabase_data.mdf'
, SIZE = 4096KB
, FILEGROWTH = 4096KB
)
LOG ON
(
NAME = 'NewDatabase_log'
, FILENAME='D:\MSSQL\LOG\NewDatabase_log.ldf'
, SIZE = 4096KB
, FILEGROWTH = 10%
)
COLLATE SQL_Latin1_General_CP1_CI_AS
"""
conn.cursor.execute(SQL_command)
SQL_command = """
ALTER DATABASE
NewDatabase
SET RECOVERY SIMPLE
"""
conn.cursor.execute(SQL_command)
However, I've got following error message:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]CREATE DATABASE statement not allowed within
multi-statement transaction. (226) (SQLExecDirectW)')
May I know what is wrong with my code?
Many thanks.
====================================================================================
So, after taking the advice from #Matthias, I've executed commit after the drop database, then the error message became:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]CREATE FILE encountered operating system
error 123(failed to retrieve text for this error. Reason: 15105) while
attempting to open or create the physical file
'D:\MSSQL\DATA\NewDatabase_data.mdf'. (5123)
(SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL
Server]CREATE DATABASE failed. Some file names listed could not be
created. Check related errors. (1802)")