Develop Reference

Best way to import daily large amounts of data to a csv with Python

I need to web scrape data with Python using Selenium, because the website where i get the data from creates content dinamycally. Due to the large amount of data (rows) and the specific website, i will need to let the program run daily (maybe for a month). Eventually, al the data will be stored in a CSV file.
I was thinking on insert every row of data for each iteration to PostgreSQL and when I finish the scraping copy the table to a CSV file. My other option was keeping open the CSV file and append row by row with each iteration.
What would be the most efficient way to do this?

From the docs:
"Note that loading a large number of rows using COPY is almost always faster than using INSERT.."

How can I add new rows without deleting the previous ones using openpyxl?

I'm new to python programming and I'm developing code that writes the value of a known variable into an excel sheet. This variable is the result of the python code that I developed. It updates its value every time the code is executed so in the end of the code, I would like to save the variable to the excel file.
My problem is related to updating the excel: when I write a new row, it deletes the previous rows. I want to save all my results in that excel file, but I only manage to save the last result I get.
I have tried many things but none has given me what I wanted as a result.
I'm using python 2.7 with pandas but I was trying to write in the excel file with openpyxl.
Question: How can I achieve saving rows to excel without deleting the previous rows?

Lost data in bigquery

I made a Python 3 script to process some CSV files, but I have a problem with the data.
I send to the stream with the insert_rows function, if I only import one file I have the same rows in the CSV and the BigQuery, but when I import more files, BigQuery lost rows respect CSV file, but insert_rows don't return errors.
errors = connection.client.insert_rows(table_ref, info, selected_fields=schema) # API request
Thanks for the help

Issue was fixed by adding a new unique column into the CSV file, using this Python Standard Library to generate a new column and add in all rows an unique id.

Writing from Excel sheet to a table

I am looking to write certain columns of data from an excel sheet to a HTML table. Not looking to write specific/fixed cells into the table always, need to do this based on conditions. For example, if I have a table with columns Name/Age/Occupation, I would like to make an HTML table using just columns Name and Occupation. Also, within Name, I would only like to write the names starting with 'N' onto the table and corresponding Occupation. The Excel sheet dynamically changes with new data everytime. Essentially, I would not want to write specific cells or range of cells into the table but only the data based on conditions I set. Any suggestions using python/html/jquery or other methods are welcome.

First you should edit the Excel file, export it as a .csv file and then work on the file using a program language of your preference. It would be much much more complicated if you try to work on the .xls or .xlsx files. I recommend using python with its library panda that works on csv files.

For parsing excel files, I've had good success using openpyxl
A Python library to read/write Excel 2010 xlsx/xlsm files

Retain Excel Settings When Adding New CSV

I've written a python/webdriver script that scrapes a table online, dumps it into a list and then exports it to a CSV. It does this daily.
When I open the CSV in Excel, it is unformatted, and there are fifteen (comma-delimited) columns of data in each row of column A.
Of course, I then run 'Text to Columns' and get everything in order. It looks and works great.
But tomorrow, when I run the script and open the CSV, I've got to reformat it.
Here is my question: "How can I open this CSV file with the data already spread across the columns in Excel?"

Try importing it as a csv file, instead of opening it directly on excel.