I have a simple Python code that loops i times to search for a table on a site and exports to an Excel sheet. The problem is that I only see the last iteration of the loop and not the rest. Does anyone know how to get the rest of the loop results on to Excel?
import pandas as pd
filename = r"c:\temp\test.html"
path = open(filename, 'r')
destination = r"c:\users\someone\desktop\test.xlsx"
table = pd.io.html.read_html(filename, attrs= {'class':'table'})
num_tables = (len(table))
for i in range(0, num_tables):
tables = table[i]
print(tables)
writer = pd.ExcelWriter(destination, engine='xlsxwriter')
dfi.to_excel(writer, index=False, sheet_name='Test')
workbook = writer.book
worksheet = writer.sheets['Test']
writer.save()
Try creating the ExcelWriter instance before the loop. Then use to_excel inside loop. Finally, save the writer at the end of the script (as you have).
You should set the sheet_name to a variable that changes on very loop to ensure that you end up with an excel file with many sheets, rather than overwriting the 'test' sheet.
Update
Perhaps you are looking for something like the below. Its difficult to know without seeing an example of the data. This assumes that the dataframes from the website are all in the same format.
list_of_frames = pd.io.html.read_html(filename, attrs= {'class':'table'})
df = pd.concat(list_of_frames)
df.to_excel('test.xlsx')
Related
I am trying to take a workbook, loop through specific worksheets retrieve a dataframe, manipulate it and essentially paste the dataframe back in the same place without changing any of the other data / sheets in the document, this is what I am trying:
path= '<folder location>.xlsx'
wb = pd.ExcelFile(path)
for sht in ['sheet1','sheet2','sheet3']:
df= pd.read_excel(wb,sheet_name = sht, skiprows = 607,nrows = 11, usecols = range(2,15))
# here I manipulate the df, to then save it down in the same place
df.to_excel(wb,sheet_name = sht, startcol=3, startrow=607)
# Save down file
wb.save(path))
wb.close()
My solution so far will just save the first sheet down with ONLY the data that I manipulated, I lose all other sheets and data that was on the sheet that I want to stay, so I end up with just sheet1 with only the data I manipulated.
Would really appreciate any help, thank you
Try using an ExcelWriter instead of an ExcelFile:
path= 'folder location.xlsx'
with pd.ExcelWriter(path) as writer:
for sht in ['sheet1','sheet2','sheet3']:
df= pd.read_excel(wb,sheet_name = sht, skiprows = 607,nrows = 11, usecols = range(2,15))
####here I manipulate the df, to then save it down in the same place###
df.to_excel(writer,sheet_name = sht, startcol=3, startrow=607)
Although I am not sure how it will behave when the file already exists and you overwrite some of them. It might be easier to read everything in first, manipulate the required sheets and save to a new file.
I'm totally new to scripting and have been learning Python. I'm trying to copy an entire row of data from one Excel file to another. More specifically, I have a field called bound in my input excel spreadsheet. When this equals 5002, I'd like to copy that entire row to a sheet called 'bound_5002' in a new spreadsheet created by the Python script. My script works when I hardcode 5002 and bound_5002, but I have a list of about 30 of these unique bound codes that I'd like it to cycle through. I've tried to iterate through a list of the codes (shown below), but it creates an Excel file that is incorrect. Upon opening an error message appears
we found a problem with some content in data_recon_xlsx. Do you want us to try to recover as much as we can...
It has created new tabs with no data and the names ecovered_Sheet1 etc. Is my iterator wrong, missing something or can this function not work when iterating through a list?
Written the script without it iterating and it works when hardcoded in, but on trying to iterate through a list of codes it doesn't. I've tried printing out the fields being iterated, adding in a ' character either side (sheet_ref) or without commas.
Expected - an Excel file called 'data_recon.xlsx' with multiple tabs, containing the data for the corresponding bound field.
Actual - an Excel file with all the tabs created and headers as required, but missing the data that was required to be copied across. New sheets have been added but they are blank and have the names, 'Recovered_Sheet1', 'Recovered_Sheet2', etc.
### Create a list of the domain codes of interest
bounds = ['800', '3001', '3002', '3003', '3101', '3102', '3103', '3105', '3106', '3110', '3111', '3112', '5002', '5003', '5004', '5005', '5006', '5101', '5102', '5104', '5105', '5106', '5107', '5110', '9003', '9004', '9101', '9102', '9103', '9104', '9105', '9106']
### Copy out only the matching domains to the tabs
i = 0
ids = [(bounds[i])]
final_result = {}
while i <= 15:
with open(import_file_path_orig, 'r') as NN:
reader = csv.reader(NN)
next(reader)
for compid, dhid, length, gimp, to, bound, auppm, aucap in reader:
if bound in ids:
final_result.setdefault('compid', []).append(compid)
final_result.setdefault('dhid', []).append(dhid)
final_result.setdefault('length', []).append(length)
final_result.setdefault('gimp', []).append(gimp)
final_result.setdefault('to', []).append(to)
final_result.setdefault('bound', []).append(bound)
final_result.setdefault('auppm', []).append(auppm)
final_result.setdefault('aucap', []).append(aucap)
df = pd.DataFrame.from_dict(final_result)
### Paste the data matching the bound from dataframe to Excel sheet
book = load_workbook('data_recon.xlsx')
sheet_ref = ("'" + 'bound_'+ bounds[i] + "'")
sheet_name = (sheet_ref)
with pd.ExcelWriter('data_recon.xlsx', engine='openpyxl') as writer:
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)
df.to_excel(writer, sheet_name=sheet_name, startrow = 1, startcol=0, header=False, index=False, engine='openpyxl')
writer.save()
print("bound_" + bounds[i] + " Sheet Populated")
### tests
print (sheet_ref)
print (bounds[i])
i += 1
print("DATA RECON FILE COMPLETE")
Below is showing an earlier version, without it iterating and works as required:
### Copy out only the matching domains to the tabs
ids = ['5101']
final_result = {}
with open('inout_file.csv', 'r') as NN:
reader = csv.reader(NN)
next(reader)
for compid, dhid, length, gimp, to, bound, auppm, aucap in reader:
if bound in ids:
final_result.setdefault('compid', []).append(compid)
final_result.setdefault('dhid', []).append(dhid)
final_result.setdefault('length', []).append(length)
final_result.setdefault('gimp', []).append(gimp)
final_result.setdefault('to', []).append(to)
final_result.setdefault('bound', []).append(bound)
final_result.setdefault('auppm', []).append(auppm)
final_result.setdefault('aucap', []).append(aucap)
df = pd.DataFrame.from_dict(final_result)
### Paste the data matching the bound from dataframe to Excel sheet
book = load_workbook('data_recon.xlsx')
sheet_name = 'bound_5101'
with pd.ExcelWriter('data_recon.xlsx', engine='openpyxl') as writer:
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)
df.to_excel(writer, sheet_name=sheet_name, startrow = 1, startcol=0, header=False, index=False, engine='openpyxl')
print(sheet_name + " Sheet Populated")
I've updated this answer to show a much simpler version of what you have outlined above
In order to write multiple dataframes to a file on different sheets you need to do this outside of your loop once you have all the dataframes built.
# Import the csv file into a single datafrome
df = pd.read_csv(import_file_path_orig, columns=['compid', 'dhid', 'length', 'gimp', 'to', 'bound', 'auppm', 'aucap'])
# Creating a new sheet for each dataframe
# Open the proper filehandle
with pd.ExcelWriter('data_recon.xlsx', engine='openpyxl') as writer:
# .... If you have other stuff to do on the main sheet, do it here ....
# Now, we write a single sheet for each set of rows that include the 'bound' value
for b in bounds:
# Filter the dataset for those rows that match the current value of `b`
temp_df = df[df['bound']==b]
# Build the name of the sheet to be written
sheet_name = f'bound_{b}'
# Write the filtered values to a sheet in the current workbook
temp_df.to_excel(writer,sheet_name=sheet_name)
The issue in your current code is that once you write a workbook, it will be rewritten by the next time you try to write the workbook, you can't add sheets. docs
I'm new to Python (and programming in general) and am running into a problem when writing data out to sheets in Excel.
I'm reading in an Excel file, performing a sum calculation on specific columns, and then writing the results out to a new workbook. Then at the end, it creates two charts based on the results.
The code works, except every time I run it, it creates new sheets with numbers appended to the end. I really just want it to overwrite the sheet names I provide, instead of creating new ones.
I'm not familiar enough with all the modules to understand all the options that are available. I've researched openpyxl, and pandas, and similar examples to what I'm trying to do either aren't easy to find, or don't seem to work when I try them.
import pandas as pd
import xlrd
import openpyxl as op
from openpyxl import load_workbook
import matplotlib.pyplot as plt
# declare the input file
input_file = 'TestData.xlsx'
# declare the output_file name to be written to
output_file = 'TestData_Output.xlsx'
book = load_workbook(output_file)
writer = pd.ExcelWriter(output_file, engine='openpyxl')
writer.book = book
# read the source Excel file and calculate sums
excel_file = pd.read_excel(input_file)
num_events_main = excel_file.groupby(['Column1']).sum()
num_events_type = excel_file.groupby(['Column2']).sum()
# create dataframes and write names and sums out to new workbook/sheets
df_1 = pd.DataFrame(num_events_main)
df_2 = pd.DataFrame(num_events_type)
df_1.to_excel(writer, sheet_name = 'TestSheet1')
df_2.to_excel(writer, sheet_name = 'TestSheet2')
# save and close
writer.save()
writer.close()
# dataframe for the first sheet
df = pd.read_excel(output_file, sheet_name='TestSheet1')
values = df[['Column1', 'Column3']]
# dataframe for the second sheet
df = pd.read_excel(output_file, sheet_name='TestSheet2')
values_2 = df[['Column2', 'Column3']]
# create the graphs
events_graph = values.plot.bar(x = 'Column1', y = 'Column3', rot = 60) # rot = rotation
type_graph = values_2.plot.bar(x = 'Column2', y = 'Column3', rot = 60) # rot = rotation
plt.show()
I get the expected results, and the charts work fine. I'd really just like to get the sheets to overwrite with each run.
From the pd.DataFrame.to_excel documentation:
Multiple sheets may be written to by specifying unique sheet_name.
With all data written to the file it is necessary to save the changes.
Note that creating an ExcelWriter object with a file name that already
exists will result in the contents of the existing file being erased.
Try writing to the book like
import pandas as pd
df = pd.DataFrame({'col1':[1,2,3],'col2':[4,5,6]})
writer = pd.ExcelWriter('g.xlsx')
df.to_excel(writer, sheet_name = 'first_df')
df.to_excel(writer, sheet_name = 'second_df')
writer.save()
If you inspect the workbook, you will have two worksheets.
Then lets say you wanted to write new data to the same workbook:
writer = pd.ExcelWriter('g.xlsx')
df.to_excel(writer, sheet_name = 'new_df')
writer.save()
If you inspect the workbook now, you will just have one worksheet named new_df
If there are other worksheets in the excel file that you want to keep and just overwrite the desired worksheets, you would need to use load_workbook.
Before you wrtie any data, you could delete the sheets you want to write to with:
std=book.get_sheet_by_name(<sheee_name>)
book.remove_sheet(std)
That will stop the behavior where a number gets appended to the worksheet name once you attempt to write a workbook with a duplicate sheet name.
I am trying to add an empty excel sheet into an existing Excel File using python xlsxwriter.
Setting the formula up as follows works well.
workbook = xlsxwriter.Workbook(file_name)
worksheet_cover = workbook.add_worksheet("Cover")
Output4 = workbook
Output4.close()
But once I try to add further sheets with dataframes into the Excel it overwrites the previous excel:
with pd.ExcelWriter('Luther_April_Output4.xlsx') as writer:
data_DifferingRates.to_excel(writer, sheet_name='Differing Rates')
data_DifferingMonthorYear.to_excel(writer, sheet_name='Differing Month or Year')
data_DoubleEntries.to_excel(writer, sheet_name='Double Entries')
How should I write the code, so that I can add empty sheets and existing data frames into an existing excel file.
Alternatively it would be helpful to answer how to switch engines, once I have produced the Excel file...
Thanks for any help!
If you're not forced use xlsxwriter try using openpyxl. Simply pass 'openpyxl' as the Engine for the pandas built-in ExcelWriter class. I had asked a question a while back on why this works. It is helpful code. It works well with the syntax of pd.to_excel() and it won't overwrite your already existing sheets.
from openpyxl import load_workbook
import pandas as pd
book = load_workbook(file_name)
writer = pd.ExcelWriter(file_name, engine='openpyxl')
writer.book = book
data_DifferingRates.to_excel(writer, sheet_name='Differing Rates')
data_DifferingMonthorYear.to_excel(writer, sheet_name='Differing Month or Year')
data_DoubleEntries.to_excel(writer, sheet_name='Double Entries')
writer.save()
You could use pandas.ExcelWriter with optional mode='a' argument for appending to existing Excel workbook.
You can also append to an existing Excel file:
>>> with ExcelWriter('path_to_file.xlsx', mode='a') as writer:`
... df.to_excel(writer, sheet_name='Sheet3')`
However unfortunately, this requires using a different engine, since as you observe the ExcelWriter does not support the optional mode='a' (append). If you try to pass this parameter to the constructor, it raises an error.
So you will need to use a different engine to do the append, like openpyxl. You'll need to ensure that the package is installed, otherwise you'll get a "Module Not Found" error. I have tested using openpyxl as the engine, and it is able to append new a worksheet to existing workbook:
with pd.ExcelWriter(engine='openpyxl', path='Luther_April_Output4.xlsx', mode='a') as writer:
data_DifferingRates.to_excel(writer, sheet_name='Differing Rates')
data_DifferingMonthorYear.to_excel(writer, sheet_name='Differing Month or Year')
data_DoubleEntries.to_excel(writer, sheet_name='Double Entries')
I think you need to write the data into a new file. This works for me:
# Write multiple tabs (sheets) into to a new file
import pandas as pd
from openpyxl import load_workbook
Work_PATH = r'C:\PythonTest'+'\\'
ar_source = Work_PATH + 'Test.xlsx'
Output_Wkbk = Work_PATH + 'New_Wkbk.xlsx'
# Need workbook from openpyxl load_workbook to enumerage tabs
# is there another way with only xlsxwriter?
workbook = load_workbook(filename=ar_source)
# Set sheet names in workbook as a series.
# You can also set the series manually tabs = ['sheet1', 'sheet2']
tabs = workbook.sheetnames
print ('\nWorkbook sheets: ',tabs,'\n')
# Replace this function with functions for what you need to do
def default_col_width (df, sheetname, writer):
# Note, this seems to use xlsxwriter as the default engine.
for column in df:
# map col width to col name. Ugh.
column_width = max(df[column].astype(str).map(len).max(), len(column))
# set special column widths
narrower_col = ['OS','URL'] #change to fit your workbook
if column in narrower_col: column_width = 10
if column_width >30: column_width = 30
if column == 'IP Address': column_width = 15 #change for your workbook
col_index = df.columns.get_loc(column)
writer.sheets[sheetname].set_column(col_index,col_index,column_width)
return
# Note nothing is returned. Writer.sheets is global.
with pd.ExcelWriter(Output_Wkbk,engine='xlsxwriter') as writer:
# Iterate throuth he series of sheetnames
for tab in tabs:
df1 = pd.read_excel(ar_source, tab).astype(str)
# I need to trim my input
df1.drop(list(df1)[23:],axis='columns', inplace=True, errors='ignore')
try:
# Set spreadsheet focus
df1.to_excel(writer, sheet_name=tab, index = False, na_rep=' ')
# Do something with the spreadsheet - Calling a function
default_col_width(df1, tab, writer)
except:
# Function call failed so just copy tab with no changes
df1.to_excel(writer, sheet_name=tab, index = False,na_rep=' ')
If I use the input file name as the output file name, it fails and erases the original. No need to save or close if you use With... it closes autmatically.
I'd like for the code to run 12345 thru the loop, input it in a worksheet, then start on 54321 and do the same thing except input the dataframe into a new worksheet but in the same workbook. Below is my code.
workbook = xlsxwriter.Workbook('Renewals.xlsx')
groups = ['12345', '54321']
for x in groups:
(Do a bunch of data manipulation and get pandas df called renewals)
writer = pd.ExcelWriter('Renewals.xlsx', engine='xlsxwriter')
worksheet = workbook.add_worksheet(str(x))
renewals.to_excel(writer, sheet_name=str(x))
When this runs, I am left with a workbook with only 1 worksheet (54321).
try something like this:
import pandas as pd
#initialze the excel writer
writer = pd.ExcelWriter('MyFile.xlsx', engine='xlsxwriter')
#store your dataframes in a dict, where the key is the sheet name you want
frames = {'sheetName_1': dataframe1, 'sheetName_2': dataframe2,
'sheetName_3': dataframe3}
#now loop thru and put each on a specific sheet
for sheet, frame in frames.iteritems(): # .use .items for python 3.X
frame.to_excel(writer, sheet_name = sheet)
#critical last step
writer.save()
import pandas as pd
writer = pd.ExcelWriter('Renewals.xlsx', engine='xlsxwriter')
renewals.to_excel(writer, sheet_name=groups[0])
renewals.to_excel(writer, sheet_name=groups[1])
writer.save()
Building on the accepted answer, you can find situations where the sheet name will cause the save to fail if it has invalid characters or is too long. This could happen if you are using grouped values for the sheet name as an example. A helper function could address this and save you some pain.
def clean_sheet_name(sheet):
"""Clean sheet name so that it is a valid Excel sheet name.
Removes characters in []:*?/\ and limits to 30 characters.
Args:
sheet (str): Name to use for sheet.
Returns:
cleaned_sheet (str): Cleaned sheet name.
"""
if sheet in (None, ''):
return sheet
clean_sheet = sheet.translate({ord(i): None for i in '[]:*?/\\'})
if len(clean_sheet) > 30: # Set value you feel is appropriate
clean_sheet = clean_sheet[:30]
return clean_sheet
Then add a call to the helper function before writing to Excel.
for sheet, frame in groups.items():
# Clean sheet name for length and invalid characters
sheet = clean_sheet_name(sheet)
frame.to_excel(writer, sheet_name = sheet, index=False)
writer.save()