hello someone can help me I must create a program in Python that connects to my database, read the database records and create a pivot table with Pandas with that data
You might be looking for pandas.read_sql
You can use con parameter to pass your database connection string.
Here's a link to connect SQL to Python. Depending on the database you use, library will vary. Search for your particular database to python connection.
https://www.geeksforgeeks.org/how-to-connect-python-with-sql-database/
This will give you a 5 min crash course with Pandas and how to use pivot tables.
https://medium.com/bhavaniravi/python-pandas-tutorial-92018da85a33#:~:text=%20Get%20Started%20With%20Pandas%20In%205%20mins,DataFrame%20and%20understood%20its%20structure%2C%20let%E2%80%99s...%20More%20
Related
Now I am doing something freaky here... I want to ingest data from a pandas dataframe into an in-memory OLTP database on Microsoft SQL Server 2019. The table is not existing yet and I want to create it on the fly based on the pandas dataframe.
For that, I modify the SQL create statement that pandas generates before it usually inserts data so that I will actually create the table as an in-memory table. The create statement works fine when used directly in Microsoft SQL Server Management Studio.
However, when I use SQLAlchemy to run the create statement from within my Python script, I receive the following error message:
DDL statements ALTER, DROP and CREATE inside user transactions are not supported with memory optimized tables.
What does this mean? What is a user transaction? What could I try to make this work?
Thanks
I found out that the cause was due to the autocommit flag being set to False by default. After setting it to True, everything works as expected
Is there a simple way to download all tables from a postgresql database into pandas? For example can pandas just load from the .sql file? All the solutions I found on the line suggest connecting to the database and using select from commands, which seems far more complicated.
You have to connect to a database anyway. You can find out table names from odbc cursor and then use pandas.read_table for names ex. pypyodbc finding names:
allnames=cursor.tables( schema='your_schema').fetchall()
-- without view and indexes below
tabnames=[el[2] for el in allnames if el[3]=='TABLE']
I am trying to find a way, either in R or Python, to use a dataframe as a table in an Oracle SQL statement.
It is impractical, for my objective, to:
Create a string out of a column and use that as a criteria (more than a 1k, which is the limit)
Create a new table in the database and use that (don't have access)
Download the entire contents of the table and merge in pandas (millions of records in the database and would bog down the db and my system)
I have found packages that will allow you to "register" a dataframe and have it act as a "table/view" to allow queries against it, but it will not allow them to be used in a query with a different connection string. Can anyone point me in the right direction? Either to allow two different connections in the same SQL statement (to Oracle and a package like DuckDB) to permit an inner join or direct link to the dataframe and allow that to be used as a table in a join?
SAS does this so effortlessly and I don't want to go back to SAS because the other functionality is not as good as Python / R, but this is a dealbreaker if I can't do database extractions.
Answering my own question here -- after much research.
In short, this cannot be done. A series of criteria, outside of a list or concat, you cannot create a dataframe in python or R and pass it through a query into a SQL Server or Oracle database. It's unfortunate, but if you don't have permissions to write to temporary tables in the Oracle database, you're out of options.
I wonder if there's anyways to upload a dataframe and create a new table in Exasol? import_from_pandas assumes the table already exists. Do we need to run a SQL separately to create the table? for other databases, to_sql can just create the table if it doesn't exist.
Yes, As you mentioned import_from_pandas requires a table. So, you need to create a table before writing to it. You can run a SQL create table ... script by connection.execute before using import_from_pandas. Also to_sql needs a table since based on the documentation it will be translated to a SQL insert command.
Pandas to_sql allows to create a new table if it does not exist, but it needs an SQLAlchemy connection, which is not supported for Exasol out of the box. However, there seems to be a SQLAlchemy dialect for Exasol you could use (haven't tried it yet): sqlalchemy-exasol.
Alternatively, I think you have to use a create table statement and then populate the table via pyexasol's import_from_pandas.
I have a script that downloads data from an API and turns all of this info into a CSV. I need this data to be into a table in MySQL (I already created it and established the connection with MySQL Connector). Is there anyway to do this?
Pandas.DataFrame has a method to_sql which writes a dataframe into a sql table.
Simplest way to use the method is to create a connection with sqlalchemy.(You will need to install mysql-python) and use .to_sql to read the data into the table.
from sqlalchemy import create_engine
engine = create_engine('mysql://username:password#host:port/database') #change to connect your mysql
#if you want to append the data to an existing table
df.to_sql(name='SQL Table name',con=engine,if_exists='append',index=False)
#if you want to create a new table
df.to_sql(name='New SQL Table name',con=engine,if_exists='fail',index=False)
Please note that you will need to use param dtype to define the dtype of the table columns if you have created the table before hand.