So I'm trying to use the to_sql command to write records stored in a DataFrame to a SQL database using sqlalchemy.
I have mySQL installed
I have a dataframe called df
What I've tried:
First I created a schema in mySQL workbench called "task_db", then:
import pandas as pd
import sqlalchemy
import pymysql
import sqlalchemy as db
engine = db.create_engine("mysql+pymysql://myusername:mypassword#3306/task_db")
df.to_sql("result", engine, schema=None, if_exists="fail", index=True, index_label=None, chunksize=None, dtype=None, method=None)
In the errors it mentions several times "Can't connect to MySQL server on '3306'", but that is the localhost I got showing on mySQL workbench.
You didn't declare this localhost anywhere. Try this connection string: mysql+pymysql://myusername:mypassword#localhost/task_db?host=localhost?port=3306
I guess there is a connectivity issue. because the same code was working fine in my old laptop but not working in new laptop.
Below are my line of code.
import numpy as np
import pandas as pd
import pandasql
import sqlalchemy
engine = sqlalchemy.create_engine('postgresql://prd*****:Au*****$#10.31.13.6:****/redshiftdb')
query1 = 'Select * from pres_sandbox.abias_atb limit 10'
e = pd.read_sql_query(query1,engine)
e.to_csv('C:\\Users\\jawed.sheikh\\Desktop\\ATTRIBUTES UPLOAD\\Attribute_22072022qq.csv', index = False) #Date change
print("File has been created in 'ATTRIBUTES UPLOAD' folder")
Also I tried below lines to get output but not able to extract the data.
import mysql.connector as sql
db_connection = sql.connect(host='10.31.13.6', port= '****', database='redshiftdb', user='prd******', password='A*****$',
connect_timeout=1000)
db_cursor = db_connection.cursor()
db_cursor.execute('Select * from pres_sandbox.abias_atb limit 10')
table_rows = db_cursor.fetchall()
df = pd.DataFrame(table_rows)
print(df)
From first query I am getting below error:
OperationalError: (psycopg2.OperationalError) could not translate host name "********#10.31.13.6" to address: Unknown server error (password removed)
(Background on this error at: https://sqlalche.me/e/14/e3q8)
From second query I am getting below error:
InterfaceError: 2013: Lost connection to MySQL server during query
Note:All the password and host are correct.
I am trying to connect to oracle-db using the odbc connection string, I am able to make the connection using pyodbc
import pyodbc
import pandas as pd
connection_string = 'DRIVER={Oracle};DBQ=X.X.X.X/YY/dbname;UID=someuser;PWD=XXXXXX'
cnxn = pyodbc.connect(connection_string)
but I am not able to connect using SQLAlchemy.
from sqlalchemy.engine import create_engine
params = urllib.parse.quote_plus(connection_string)
db_engine = create_engine(f"cx-Oracle+pyodbc:///?odbc_connect={params}")
Context: I'd like to send a concatenated data frame (I joined several dataframes from individual stock data) into a MySQL database, however, I can't seem to create a table and send the data there
Problem: When I run this code df.to_sql(name='stockdata', con=con, if_exists='append', index=False) (source: Writing a Pandas Dataframe to MySQL), I keep getting this error: pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': not all arguments converted during string formatting.
I'm new to MySQL as well so any help is very welcome! Thank you
from __future__ import print_function
import pandas as pd
from datetime import date, datetime, timedelta
import numpy as np
import yfinance as yf
import mysql.conector
import pymysql as pymysql
import pandas_datareader.data as web
from sqlalchemy import create_engine
import yahoo_fin.stock_info as si
######################################################
# PyMySQL configuration
user = '...'
passw = '...'
host = '...'
port = 3306
database = 'stockdata'
con.cursor().execute("CREATE DATABASE IF NOT EXISTS {0} ".format(database))
con = pymysql.connect(host=host,
port=port,
user=user,
passwd=passw,
db=database,
charset='utf8')
df.to_sql(name='stockdata', con=con, if_exists='append', index=False)
.to_sql() expects the second argument to be either a SQLAlchemy Connectable object (Engine or Connection) or a DBAPI Connection object. If it is the latter then pandas assumes that it is a SQLite connection.
You need to use SQLAlchemy to create an engine object
engine = create_engine("mysql+pymysql://…")
and pass that to to_sql()
trying to write pandas dataframe to MySQL table using to_sql. Previously been using flavor='mysql', however it will be depreciated in the future and wanted to start the transition to using SQLAlchemy engine.
sample code:
import pandas as pd
import mysql.connector
from sqlalchemy import create_engine
engine = create_engine('mysql+mysqlconnector://[user]:[pass]#[host]:[port]/[schema]', echo=False)
cnx = engine.raw_connection()
data = pd.read_sql('SELECT * FROM sample_table', cnx)
data.to_sql(name='sample_table2', con=cnx, if_exists = 'append', index=False)
The read works fine but the to_sql has an error:
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master
WHERE type='table' AND name=?;': Wrong number of arguments during
string formatting
Why does it look like it is trying to use sqlite? What is the correct use of a sqlalchemy connection with mysql and specifically mysql.connector?
I also tried passing the engine in as the connection as well, and that gave me an error referencing no cursor object.
data.to_sql(name='sample_table2', con=engine, if_exists = 'append', index=False)
>>AttributeError: 'Engine' object has no attribute 'cursor'
Using the engine in place of the raw_connection() worked:
import pandas as pd
import mysql.connector
from sqlalchemy import create_engine
engine = create_engine('mysql+mysqlconnector://[user]:[pass]#[host]:[port]/[schema]', echo=False)
data.to_sql(name='sample_table2', con=engine, if_exists = 'append', index=False)
Not clear on why when I tried this yesterday it gave me the earlier error.
Alternatively, use pymysql package...
import pymysql
from sqlalchemy import create_engine
cnx = create_engine('mysql+pymysql://[user]:[pass]#[host]:[port]/[schema]', echo=False)
data = pd.read_sql('SELECT * FROM sample_table', cnx)
data.to_sql(name='sample_table2', con=cnx, if_exists = 'append', index=False)
Using pymysql and sqlalchemy, this works for Pandas v0.22:
import pandas as pd
import pymysql
from sqlalchemy import create_engine
user = 'yourUserName'
passw = 'password'
host = 'hostName' # either localhost or ip e.g. '172.17.0.2' or hostname address
port = 3306
database = 'dataBaseName'
mydb = create_engine('mysql+pymysql://' + user + ':' + passw + '#' + host + ':' + str(port) + '/' + database , echo=False)
directory = r'directoryLocation' # path of csv file
csvFileName = 'something.csv'
df = pd.read_csv(os.path.join(directory, csvFileName ))
df.to_sql(name=csvFileName[:-4], con=mydb, if_exists = 'replace', index=False)
"""
if_exists: {'fail', 'replace', 'append'}, default 'fail'
fail: If table exists, do nothing.
replace: If table exists, drop it, recreate it, and insert data.
append: If table exists, insert data. Create if does not exist.
"""
I know in the title of the question is included the word SQLAlchemy, however I see in the questions and answers the need to import pymysql or mysql.connector, and also is possible to do the job with pymysql, withouth calling SQLAlchemy.
import pymysql
user = 'root'
passw = 'my-secret-pw-for-mysql-12ud' # In previous posts variable "pass"
host = '172.17.0.2'
port = 3306
database = 'sample_table' # In previous posts similar to "schema"
conn = pymysql.connect(host=host,
port=port,
user=user,
passwd=passw,
db=database)
data.to_sql(name=database, con=conn, if_exists = 'append', index=False, flavor = 'mysql')
I think this solution could be good althought it is not using SQLAlchemy.