I am attempting to great a temporary table in an SQL database and populate the table from a pandas dataframe. I am receiving an error when using the df.to_sql to populate the temp table. Thank you for the assistance.
import pandas as pd
from sqlalchemy import create_engine
import pandas.io.sql as psql
import urllib
params = urllib.quote_plus("DRIVER={SQL Server};SERVER=ServerAddressHere;DATABASE=DatabaseNameHere;Trusted_Connection=yes")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection = engine.connect()
resoverall = connection.execute('''SELECT DISTINCT
a.CountryRegionID AS ISO_Short,
b.Name
FROM
CustTable AS a
LEFT JOIN AddressCountryRegion AS b
ON b.CountryRegionID = a.CountryRegionID''')
Countries= pd.DataFrame(resoverall.fetchall())
Countries.columns = resoverall.keys()
Countries= pd.Countries['ISO_Short'].str.upper()
Countries= pd.DataFrame(data=Countries)
temp = connection.execute('''
create table #tempTable
(
ISO_Short varchar(5)
)
''')
Countries.to_sql('Countries',engine)
The error I'm receiving is:
ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]CREATE TABLE permission denied in database 'databasename'. (262) (SQLExecDirectW)") [SQL: u'\nCREATE TABLE [Countries] (\n\t[index] BIGINT NULL, \n\t[ISO_Short] VARCHAR(max) NULL\n)\n\n'
UPDATE:
The other option I thought of is to use Pyodbc and convert Countries to a dictionary and then pass the dictionary values into the temporary table. Using this method, everything works until I try and pass the dictionary to the temp table. I have the following code using this approach:
import pandas as pd
import pyodbc
import pandas.io.sql as psql
cnxn = pyodbc.connect('''DRIVER={SQL Server};SERVER=telsmith;
DATABASE=DatabaseNameHere;Trusted_Connection=yes;''')
cursor = cnxn.cursor()
Countries= '''
SELECT DISTINCT
a.CountryRegionID AS ISO_Short,
b.Name
FROM
CustTable AS a
LEFT JOIN AddressCountryRegion AS b
ON b.CountryRegionID = a.CountryRegionID
'''
Countries= psql.read_sql(Countries, cnxn)
Countries= Countries['ISO_Short'].str.upper()
Countries= pd.DataFrame(data=Countries)
Countriesdict = Countries.to_dict()
Temp = '''
create table #tempTable
(
ISO_Short varchar(5)
)
'''
cnxn.commit()
# This is where I run into difficulty
placeholders = ', '.join(['%s'] * len(Countriesdict ))
columns = ', '.join(Countriesdict .keys())
sql = "INSERT INTO #tempTable VALUES ( %s )" % (placeholders)
cursor.execute(sql, Countriesdict.values())
This might sound little dumb but look at the error:
ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]CREATE TABLE permission denied in database 'databasename'. (262) (SQLExecDirectW)") [SQL: u'\nCREATE TABLE [Countries] (\n\t[index] BIGINT NULL, \n\t[ISO_Short] VARCHAR(max) NULL\n)\n\n'
Do you have any database called databasename? Since It can't find database, it can't create table. I ran the same code and it worked just fine. I believe that's the reason
Not strictly a SQLAlchemy concern. You need to obtain "CREATE TABLE" permissions on the server from your DBA for some username and password, and use those credentials to access your DB. Try including "UID=uname;PWD=pword;" in your params for some set of permissioned credentials.
I might have a solution that worked for me:
from sqlalchemy import create_engine
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server};SERVER=10.233.6.52;DATABASE=databaseName;UID=xxx;PWD=Welcome1!")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection = engine.connect()
df.to_sql('tempTable',engine)
Related
I'm using Jupyter notebook to insert a pandas dataframe into a mysql table. But I can't execute the first SQL statement without getting this error: ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DROP TABLE IF EXISTS avila;CREATE TABLE avila( intercolumnar_distance FLOAT(12),' at line 1")
This is the code I am using to try to insert the dataframe 'avila'.
import pymysql
import pymysql.cursors as sql
# Connect to the database
connection = pymysql.connect(host='localhost',
user='root',
password='')
curs = connection.cursor(pymysql.cursors.DictCursor)
# Create sql database for dataset
sql = 'DROP DATABASE IF EXISTS avila; CREATE DATABASE avila;'
curs.execute(sql)
sql = "USE avila;DROP TABLE IF EXISTS avila;CREATE TABLE avila( intercolumnar_distance FLOAT(12), upper_margin FLOAT(10), lower_margin FLOAT(10), exploitation FLOAT(10), row_num FLOAT(10), modular_ratio FLOAT(10), interlinear_spacing FLOAT(10), weight FLOAT(10), peak_number FLOAT(10), modratio_to_interlinspacing FLOAT(10), monk CHAR(5) );"
curs.execute(sql)
# Insert avila DataFrame into MySQL using sql alchemy
from sqlalchemy import create_engine
engine = create_engine("mysql+pymysql://{user}:{pw}#localhost/{db}"
.format(user="root",
pw="XXXXX",
db="AVILA"))
avila.to_sql('avila', con = engine, if_exists = 'append')
I can't advance past the first sql statement because of the error.
I am trying, for the first time ever, to send data from a data frame in Spyder to Azure SQL Server...I think it's called Synapse. I created a small table in the database and when I run the code below, I see the results I expect to see.
import pyodbc
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
driver= '{ODBC Driver 17 for SQL Server}'
with pyodbc.connect('DRIVER='+driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password) as conn:
with conn.cursor() as cursor:
cursor.execute("SELECT * From Order_Table")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
So, the connection is fine. I guess I am just stuck on the syntax to push a data frame to SQL Server in Azure. I tested the code below.
import pyodbc
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
all_data.to_sql('health', conn, if_exists='replace', index=True)
When I run that code, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)")
Try to do it by importing Pandas and pyodbc.
Below are few basic steps which we follow usually:
Connect to SQL Server.
Install all your python packages in your local.
Load the data into CSV.
Later you can use below Python script to load it from dataframe.
import pyodbc
import pandas as pd
df = pd.read_csv("c:\\user\\username\department.csv")
server = 'yourservername'
database = 'AdventureWorks'
username = 'username'
password = 'yourpassword'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
#Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute("INSERT INTO HumanResources.DepartmentTest (DepartmentID,Name,GroupName) values(?,?,?)", row.DepartmentID, row.Name, row.GroupName)
cnxn.commit()
cursor.close()
After completing the configs you can run below command to get the data from SQL:
SELECT count(*) from HumanResources.DepartmentTest;
Refer to this official doc for detailed explanation.
Hey I'm trying to create table on MySQL using Python, the user s0566293 has all the grants on the table but i still get 1044 (42000) access denied, any suggestions?.
Here is the code:
import mysql.connector
db_connection = mysql.connector.connect(
host="141.45.91.40",
user="s0566293",
passwd="6172I9wf",
database="dbproject"
)
db_cursor = db_connection.cursor()
#Here creating database table as student'
db_cursor.execute("CREATE TABLE fil (filName INT, Genre VARCHAR(255))")
#Get database table'
db_cursor.execute("SHOW TABLES")
for table in db_cursor:
print(table)
I'm new to Python and I wanted to ask you for help.
I want to put the data of a view in SQL Server into a table of my database in MySQL, when I try to give the following error:
Execution failed on sql: SELECT name FROM sqlite_master WHERE
type='table' AND name=?; not all arguments converted during string
formatting unable to rollback
Using Python version 3.7
Below is the code I use:
import pymysql.cursors
import pyodbc
import pandas as pd
# SQL Server Connection
connection = pyodbc.connect("DSN=SQLServer") #autocommit=True
try:
with connection.cursor() as cursor:
result = "SELECT * FROM dw.dbo.vW_sale"
df = pd.read_sql_query("SELECT * FROM dw.dbo.vW_sale", connection)
cursor.execute(result)
table = cursor.fetchall()
print(table)
finally:
connection.close()
# MySQL connection
cnx = pymysql.connect(host='test',
user='test',
password='test',
db='dw')
try:
with cnx.cursor() as cursor:
mysql = "select *from ft_sale_test"
cursor.execute(mysql)
result = cursor.fetchall()
#print(result)
finally:
cnx.close()
# using if_exists to handle the table that already exists
The error happens right here
df.to_sql(con=cnx, name= 'ft_sale_test', if_exists= 'replace')
I have an SQL database and am wondering what command you use to just get a list of the table names within that database.
To be a bit more complete:
import MySQLdb
connection = MySQLdb.connect(
host = 'localhost',
user = 'myself',
passwd = 'mysecret') # create the connection
cursor = connection.cursor() # get the cursor
cursor.execute("USE mydatabase") # select the database
cursor.execute("SHOW TABLES") # execute 'SHOW TABLES' (but data is not returned)
now there are two options:
tables = cursor.fetchall() # return data from last query
or iterate over the cursor:
for (table_name,) in cursor:
print(table_name)
SHOW tables
15 chars
show tables will help. Here is the documentation.
It is also possible to obtain tables from a specific scheme with execute the single query with the driver below.
python3 -m pip install PyMySQL
import pymysql
# Connect to the database
conn = pymysql.connect(host='127.0.0.1',user='root',passwd='root',db='my_database')
# Create a Cursor object
cur = conn.cursor()
# Execute the query: To get the name of the tables from a specific database
# replace only the my_database with the name of your database
cur.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'my_database'")
# Read and print tables
for table in [tables[0] for tables in cur.fetchall()]:
print(table)
output:
my_table_name_1
my_table_name_2
my_table_name_3
...
my_table_name_x