I'm using Python, Pandas and SQLite3. I've read data from a CSV into a Dataframe and then used to_sql to store the data from Pandas in a SQLite3 database table.
I didn't set an index so SQLite3 used an auto index for each row.
Is it possible for me to use that auto index when I want to delete a row from my Python code?
I want to use something like "DELETE FROM prod_backup WHERE id=4;" but I'm getting an error saying no column called id. When I use a database browser, the auto index column doesn't actually have a column name so I'm not sure how to actually reference it.
Any help would be greatly appreciated. Thanks
Related
I read from an API the following data into a pandas dataframe:
Now, I want to write this data into a MySQL-DB-table, using pandas to_sql:
In MySQL, the column is set up correctly, but has not written the values:
Then I looked in the debugger to show me the dataframe:
I thought it would maybe a formatting issue, and added the following lines:
In the debugger, it looks now fine:
But now, in the database, it wants to write the index column as text
... and interrupts the execution with an error:
Is there a way to get this going, aka to write df index data as date into a MySQL DB using pandas to_SQL in connection with a sqlalchemy engine?
Edit:
Table schema:
DataFrame Header:
It seems you are using Date column as primary key. I would suggest not to use that as primary key instead you should use Date + Ticker as primary key.
i am creating a database using SQLAlchemy and I Need to do Migration to my data as i am using df_sql function for converting my csv into dataframe and then to tables in sqlalchemy. As i do this i need to do Migration to add new column and values inside it and assign Primary and foreign key Features. I saw someting related to Alembic and flask but am not sure how to upgrade it as also am working on Jupyter. Any ideas of how i can update delete and assign keys to my tables would be very helpful. Done until the table creation.
metadata.tables.keys()
dict_keys(['table1', 'table2'])
I also tried directly to create a temp table and copy ist values and assinging Primary key but am getting error with my column names as it has Special characters so i cant create duplicate too. Rename property too doesnt work
Column: date
Column: time_stamp
Column: timeslices[5].profilerDataProcess[8]_C0[us]
Column: timeslices[4].profilerDataProcess[54]_C0[us]
Column: timeslices[4]profilerDataProcess[50]_C0[us]
Column: timeslices[4].profilerDataProcess[49]_C0[us]
Column: timeslices[0].profilerDataProcess[14]_C0[us]
I created a table importing data from a csv file into a SQL Server table. The table contains about 6000 rows that are all float. I am trying to insert a new row using INSERT (I am using Python/Spyder and SQL Server Management Studio) and it does insert the row but not at the bottom of the table but towards the middle. I have no idea why it does that. This is the code that I am using:
def create (conn):
print ("Create")
cursor = conn.cursor()
cursor.execute ("insert into PricesTest
(Price1,Price2,Price3,Price4,Price5,Price6,Price7,Price8,Price9,Price10,Price
11,Price12) values (?,?,?,?,?,?,?,?,?,?,?,?);",
(46,44,44,44,44,44,44,44,44,44,44,44))
conn.commit()
read (conn)
Any idea why this is happening? What I should add to my code to "force" that row to be added at the bottom of the table? Many thanks.
I managed to sort it out following different suggestions posted here. Basically I was conceptually wrong to think that tables in MS SQL have an order. I am now working with the data in my table using the ORDER BY dates (I added dates as my first column) and works well. Many thanks all for your help!!
The fact is that the new rows are inserted without any order by default because the server has no rule to order the newly inserted rows (there is no primary key defined). You should have created an identity column before importing your data (even you can do it now):
Id Int IDENTITY(1,1) primary key
This will ensure all rows will be added at the end of the table.
More info on the data type you could use on w3school : https://www.w3schools.com/sql/sql_datatypes.asp
I have a MySQL table with a JSON column.
I am able to write into this JSON field following either answer to this question,
but I have not yet found a solution to be able to update the json field with the same method.
The solution I tried:
out_df.to_sql('my_table', my_sql_alchemy_engine, if_exists='append')
can go 2 ways:
if there are no unique keys in my table, the rows are simply added at the end of the table, with the JSON field containing only the values I wanted to make an update with
If there are unique fields, i get an IntegrityError
Is there any way to make that work, or should I find another solution to update my JSON fields ?
I'm using PySpark to write a DataFrame to a PostgreSQL database via JDBC command below. How can I get the inserted row id ? which is set as identity column with auto-increment.
I'm using below command, not a for-loop inserting each row separately.
df.write.jdbc(url=url, table="table1", mode=mode, properties=properties)
I know I can use monotonicallyIncreasingId and set the IDs within Spark, but I'm looking for an alternative where the DB handles the assignment, but I want to get he IDs back to use in other DataFrames.
I didn't find this in the documentation.
The easiest way will be to query the table that you created and read that into a new data frame.
Alternatively, as you iterate over each row in either a for loop or generator, before you close the loop, fetch the ID of the record you just created and append each ID to a new column in the dataframe.