I wonder if there's anyways to upload a dataframe and create a new table in Exasol? import_from_pandas assumes the table already exists. Do we need to run a SQL separately to create the table? for other databases, to_sql can just create the table if it doesn't exist.
Yes, As you mentioned import_from_pandas requires a table. So, you need to create a table before writing to it. You can run a SQL create table ... script by connection.execute before using import_from_pandas. Also to_sql needs a table since based on the documentation it will be translated to a SQL insert command.
Pandas to_sql allows to create a new table if it does not exist, but it needs an SQLAlchemy connection, which is not supported for Exasol out of the box. However, there seems to be a SQLAlchemy dialect for Exasol you could use (haven't tried it yet): sqlalchemy-exasol.
Alternatively, I think you have to use a create table statement and then populate the table via pyexasol's import_from_pandas.
Related
hello someone can help me I must create a program in Python that connects to my database, read the database records and create a pivot table with Pandas with that data
You might be looking for pandas.read_sql
You can use con parameter to pass your database connection string.
Here's a link to connect SQL to Python. Depending on the database you use, library will vary. Search for your particular database to python connection.
https://www.geeksforgeeks.org/how-to-connect-python-with-sql-database/
This will give you a 5 min crash course with Pandas and how to use pivot tables.
https://medium.com/bhavaniravi/python-pandas-tutorial-92018da85a33#:~:text=%20Get%20Started%20With%20Pandas%20In%205%20mins,DataFrame%20and%20understood%20its%20structure%2C%20let%E2%80%99s...%20More%20
Now I am doing something freaky here... I want to ingest data from a pandas dataframe into an in-memory OLTP database on Microsoft SQL Server 2019. The table is not existing yet and I want to create it on the fly based on the pandas dataframe.
For that, I modify the SQL create statement that pandas generates before it usually inserts data so that I will actually create the table as an in-memory table. The create statement works fine when used directly in Microsoft SQL Server Management Studio.
However, when I use SQLAlchemy to run the create statement from within my Python script, I receive the following error message:
DDL statements ALTER, DROP and CREATE inside user transactions are not supported with memory optimized tables.
What does this mean? What is a user transaction? What could I try to make this work?
Thanks
I found out that the cause was due to the autocommit flag being set to False by default. After setting it to True, everything works as expected
I am using Python to_sql function to insert data in a database table from Pandas dataframe.
I am able to insert data in database table but I want to know in my code how many records are inserted .
How to know record count of inserts ( i do not want to write one more query to access database table to get record count)?
Also, is there a way to see logs for this function execution. like what were the queries executed etc.
There is no way to do this, since python cannot know how many of the records being inserted were already in the table.
I'm pretty new to database and server related tasks. I currently have two tables stored in a MSsql database on a server and I'm trying to use python package sqlalchemy to pull some of the data to my local machine. The first table has default schema dbo, and I was able to use the Connect String
'mssql+pyodbc://<username>:<password>#<dsnname>'
to inspect the table, but the other table has a customized schema, and I don't see any information about the table when I use the previous commands. I assume it is because now the second table has different schema and the python package can't find it anymore.
I was looking at automap hoping the package offers a way to deal with customized schema, but many concepts described in there I don't quite understand and I'm not trying to alter the database just pulling data so not sure if it's the right way, any suggestions?
Thanks
In case of automap you should pass the schema argument when preparing reflectively:
AutomapBase.prepare(reflect=True, schema='myschema')
If you wish to reflect both the default schema and your "customized schema" using the same automapper, then first reflect both schemas using the MetaData instance and after that prepare the automapper:
AutomapBase.metadata.reflect()
AutomapBase.metadata.reflect(schema='myschema')
AutomapBase.prepare()
If you call AutomapBase.prepare(reflect=True, ...) consecutively for both schemas, then the automapper will recreate and replace the classes from the 1st prepare because the tables already exist in the metadata. This will then raise warnings.
I have a script that downloads data from an API and turns all of this info into a CSV. I need this data to be into a table in MySQL (I already created it and established the connection with MySQL Connector). Is there anyway to do this?
Pandas.DataFrame has a method to_sql which writes a dataframe into a sql table.
Simplest way to use the method is to create a connection with sqlalchemy.(You will need to install mysql-python) and use .to_sql to read the data into the table.
from sqlalchemy import create_engine
engine = create_engine('mysql://username:password#host:port/database') #change to connect your mysql
#if you want to append the data to an existing table
df.to_sql(name='SQL Table name',con=engine,if_exists='append',index=False)
#if you want to create a new table
df.to_sql(name='New SQL Table name',con=engine,if_exists='fail',index=False)
Please note that you will need to use param dtype to define the dtype of the table columns if you have created the table before hand.