I am getting the above error when trying to import a pandas dataframe:
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine('mysql://username:password#localhost/dbname')
c = getsomedata()
fields = ['user_id', 'timestamp', 'text']
c1 = c[fields].reset_index()
c1.to_sql(name='comments', con=engine, if_exists='replace', index=False)
There are lots of questions with this MySql issue - but how to address with a pandas import?
The solution for me was very simple: Use the chunksize option:
c1.to_sql(name='comments', con=engine, chunksize=1000, if_exists='replace', index=False)
^^^^^^^^^^^^^^^
Probably related to this issue with overly large packets.
Related
I have excel file which has 7 columns and daily informations. I'm trying to connect excel file to pgAdmin table. But not achieving to this, it's not working. Please, help me.
My database name is "exceltodatabase". Table name is "daily". It has 7 columns. Hostname is "localhost" . Port is: 5432. Password is: 1234
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql+psycopg2://postgres:1234#localhost/exceltodatabase')
with pd.ExcelFile('C:/Users/Administrator/PycharmProjects/TelegramBot/ActivateBot/masters.xlsx') as xls:
df = pd.read_excel(xls)
df.to_sql(name='daily', con=engine, if_exists='append', index=False)
I am trying to import some data from the database (Postgre SQL) to work with them in Python. I tried with the code below, which seems quite similar to the ones I've found on the internet.
import psycopg2
import sqlalchemy as db
import pandas as pd
engine = db.create_engine('database specifications')
connection = engine.connect()
metadata = db.MetaData()
data = db.Table(tabela, metadata, schema=shema, autoload=True, autoload_with=engine)
query = db.select([data])
ResultProxy = connection.execute(query)
ResultSet = ResultProxy.fetchall()
df = pd.DataFrame(ResultSet)
However, it returns data without column names. What did I forget?
It turned out the only thing needed is adding
columns = data.columns.keys()
df.columns = columns
There is a great debate about that in this thread.
I'm looking at the documentation here.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html
I keep getting this error.
'DataFrame' object has no attribute 'to_sql'
Below, is all my code. I don't see what's wrong here. What is going on?
import pandas as pd
from sqlalchemy import create_engine
import urllib
import pyodbc
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};SERVER=server_name.database.windows.net;DATABASE=my_db;UID=my_id;PWD=my_pw")
myeng = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
df.to_sql(name="dbo.my_table", con=myeng, if_exists='append', index=False)
As it turns out, the object wasn't an actual dataframe that Pandas could interpret. This fixed the problem.
# convert pyspark.sql DF to Pandas DF
df = df.toPandas()
import pymysql
import pandas as pd
db = pymysql.connect('localhost', 'testuser', 'test123', 'world')
df1 = pd.read_sql('select * from country limit 5', db)
df1
I need to create a table name with country2 and update the df1 out to country2
Use Pandas to_sql (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html). This should work for you:
import mymysql
from sqlalchemy import create_engine
sql_table_name = 'country2'
engine = create_engine("mysql://testuser:test123#localhost:0/world") # creat engine
df1.to_sql(sql_table_name, engine) # add to table
Definitely check out SqlAlchemy. Use SqlAlchemy to write a Mysql interation class. SqlAlchemy enables using python to connect database. Encoding your dataframe into a upsert sql string. And then use cursor.execute(query_string) to do the upsert.
engine = sqlalchemy.create_engine(
'mysql+mysqlconnector://user:pwd#hostname/db_name',
connect_args={'auth_plugin': 'mysql_native_password'})
sample_sql_database = df.to_sql('table_name', con=engine)
There is an option to "append" the contends from data frame or "replace" also
sample_sql_database = df.to_sql('table_name', engine, if_exists='replace')
sample_sql_database = df.to_sql('table_name', engine, if_exists='append')
Reference :
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html
I'm trying to upload a dataframe in SQL server using pandas (to_sql) function, I get the below error
[SQL Server Native Client 11.0]Invalid character value for cast
specification (0) (SQLExecDirectW)')
I checked for variables' names and types and they are exactly the same in the SQL database and pandas dataframe.
How can I fix this?
Thanks
df.to_sql(raw_table, connDB, if_exists='append', index=False )
plz try this , this code use to juypter note book and SQL workbench
import mysql.connector
from mysql.connector import Error
from sqlalchemy import create_engine
import pandas as pd
mydata = pd.read_csv("E:\\Hourly_Format\\upload.csv")
engine = create_engine("mysql://root:admin#localhost/pythondb", pool_size=10, max_overflow=20)
mydata.to_sql(name='emp',con=engine,if_exists='append', index=False)
jupyter :-
workbench :-