I'm trying to upload a dataframe in SQL server using pandas (to_sql) function, I get the below error
[SQL Server Native Client 11.0]Invalid character value for cast
specification (0) (SQLExecDirectW)')
I checked for variables' names and types and they are exactly the same in the SQL database and pandas dataframe.
How can I fix this?
Thanks
df.to_sql(raw_table, connDB, if_exists='append', index=False )
plz try this , this code use to juypter note book and SQL workbench
import mysql.connector
from mysql.connector import Error
from sqlalchemy import create_engine
import pandas as pd
mydata = pd.read_csv("E:\\Hourly_Format\\upload.csv")
engine = create_engine("mysql://root:admin#localhost/pythondb", pool_size=10, max_overflow=20)
mydata.to_sql(name='emp',con=engine,if_exists='append', index=False)
jupyter :-
workbench :-
Related
I am using python to connect to DB2 Database
I have installed ibm_db and ibm_dbi packages and imported in to the code
import ibm_db
import ibm_db_dbi
1)created a connection string as conn_str
conn_str='database=pydev;hostname=host.test.com;port=portno;protocol=tcpip;uid=db2inst1;pwd=secret'
ibm_db_conn = ibm_db.connect(conn_str,'','')
conn = ibm_db_dbi.Connection(ibm_db_conn)
2)Now i need to read a DB2 table which is in under schemas called as "BRUD" into python pandas
could any one please help me in getting the connection for this
I'm not sure about sql syntax, but resolution looks like:
df = pd.read_sql('SELECT * FROM BRUD.table_name', conn)
I have just started learning SQL and I'm having some difficulties to import my sql file in python.
The .sql file is in my desktop, as well is my .py file.
That's what I tried so far:
import codecs
from codecs import open
import pandas as pd
sqlfile = "countries.sql"
sql = open(sqlfile, mode='r', encoding='utf-8-sig').read()
pd.read_sql_query("SELECT name FROM countries")
But I got the following message error:
TypeError: read_sql_query() missing 1 required positional argument: 'con'
I think I have to create some kind of connection, but I can't find a way to do that. Converting my data to an ordinary pandas DataFrame would help me a lot.
Thank you
This is the code snippet taken from https://www.dataquest.io/blog/python-pandas-databases/ should help.
import pandas as pd
import sqlite3
conn = sqlite3.connect("flights.db")
df = pd.read_sql_query("select * from airlines limit 5;", conn)
Do not read database as an ordinary file. It has specific binary format and special client should be used.
With it you can create connection which will be able to handle SQL queries. And can be passed to read_sql_query.
Refer to documentation often https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql_query.html
You need a database connection. I don't know what SQL flavor are you using, but suppose you want to run your query in SQL server
import pyodbc
con = pyodbc.connect(driver='{SQL Server}', server='yourserverurl', database='yourdb', trusted_connection=yes)
then pass the connection instance to pandas
pd.read_sql_query("SELECT name FROM countries", con)
more about pyodbc here
And if you want to query an SQLite database
import sqlite3
con = sqlite3.connect('pathto/example.db')
More about sqlite here
I'm looking at the documentation here.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html
I keep getting this error.
'DataFrame' object has no attribute 'to_sql'
Below, is all my code. I don't see what's wrong here. What is going on?
import pandas as pd
from sqlalchemy import create_engine
import urllib
import pyodbc
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};SERVER=server_name.database.windows.net;DATABASE=my_db;UID=my_id;PWD=my_pw")
myeng = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
df.to_sql(name="dbo.my_table", con=myeng, if_exists='append', index=False)
As it turns out, the object wasn't an actual dataframe that Pandas could interpret. This fixed the problem.
# convert pyspark.sql DF to Pandas DF
df = df.toPandas()
I have imported into pandas a SQL query by doing the following:
import pandas as pd
import numpy as np
import pyodbc
con= pyodbc.connect(
'Trusted_Connection=yes',
driver = '{SQL Server Native Client 11.0}',
server = 'SERVER',
database = 'DATABASE')
Receivables = pd.read_sql_query("select * from receivables",con)
Which works fine, but most columns are now of type "object", some has been recognized as float. Is there no method for just keeping the column type from the SQL server, where they are already defined correctly.
try to use read_sql function from pandas.
I am able to generate an engine as follows:
import pandas as pd
import sqlalchemy as sa
url = sa.url.URL(drivername='vertica+pyodbc',
username='username',
password='****',
host='vertica')
engine = sa.create_engine(url)
I can read data, e.g.
pd.read_sql_query("SELECT * FROM my_schema.tablename", engine)
However when I try to write data:
import numpy as np
df = pd.DataFrame(np.random.randn(10, 5))
df.to_sql("my_schema.random_table", engine)
I get the error
Permission denied for schema public\n (4367) (SQLExecDirectW)') u'\nCREATE TABLE "my_schema.random_table"
I believe that the issue is that "my_schema.random_table" is being wrapped by quotes, making Vertica think I want to create this table in the base (public) schema instead of my_schema. Is there a way I can specify the schema I want in the ODBC url?
pandas.DataFrame.to_sql has a schema parameter. Try using that instead of specifying the schema as a prefix to the table name.
df.to_sql('random_table', engine, schema='my_schema')