I am new to Python, and I am trying to sort of 'migrate' a excel solver model that I have created to Python, in hopes of more efficient processing time.
I receive a .csv sheet that I use as my input for the model, it is always in the same format.
This model essentially uses 4 different metrics associated with product A, B and C, and I essentially determine how to price A, B, and C accordingly.
I am at the very nascent stage of effectively inputting this data to Python. This is what I have, and I would not be surprised if there is a better approach, so open to trying anything you veterans have to recommend!
import csv
f = open("141881.csv")
for row in csv.reader(f):
price = row[0]
a_metric1 = row[1]
a_metric2 = row[2]
a_metric3 = row[3]
a_metric4 = row[4]
b_metric1 = row[7]
b_metric2 = row[8]
b_metric3 = row[9]
b_metric4 = row[10]
c_metric1 = row[13]
c_metric2 = row[14]
c_metric3 = row[15]
c_metric4 = row[16]
The .csv file comes in the format of price,a_metric1,a_metric2,a_metric3,a_metric4,,price,b_metric1,b_metric2,b_metric3,b_metric4,price,,c_metric1,c_metric2,c_metric3,c_metric4
I skip the second and third price column as they are identical to the first one.
However when I run the python script, I get the following error:
c_metric1 = row[13]
IndexError: list index out of range
And I have no idea why this occurs, when I can see the data is there myself (in excel, this .csv file would go all the way to column Q, or what I understand as row[16].
Your help is appreciated, and any advice on my approach is more than welcomed.
Thanks in advance!
Using print() can be your friend here:
import csv
with open('141881.csv') as file_handle:
file_reader = csv.reader(file_handle)
for row in file_reader:
print(row)
The code above will print out EACH row.
To print out ONLY the first row replace the for loop with: print(file_reader.__next__()) (assuming Python3)
Printing out row(s) will allow you to see what exactly a "row" is.
P.S.
Using with is advisable because it handles the opening and closing of the file for you
Look into pandas.
Read file as:
data = pd.read_csv('141881.csv'))
to read a columns:
col = data.columns['column_name']
to read a row:
row = data.ix[row_number]
CSV Module in Python transforms a spreadsheet into a matrice : a list of list
The python module to read csv transform each line of your input into a list.
For each row, it will split the row into a list of cell.In other words, one array is composed of as many columns you have into your excel spreadsheet.
Try in terminal:
>>> f = open("141881.csv")
>>> print csv.reader(f)
>>>[["id", "name", "company", "email"],[1563, "defoe", "SuperFastCompany",],["def#superfastcie.net"],[1564, "doe", "Awsomestartup", "doe#awesomestartup"], ...]`
So that's why you iterate throught the rows of your spreadsheet assigning the value into a new variable.
I recommend you to read on basics of list manipulation.
But...
What is an IndexError? catching exception:
If one cell is empty or one row has less columns than other: it will thraw an Error. Such as you described. IndexError means Python wasn't able to find a value for this specific cell. In other words if some line of your excel spreadsheet are smaller than the other it will say there is no such value to asign and throw an Index Error. That why knowing how to catch exception could be very useful to see the problem. Try to verify that the list of each has the same lenght if not assign an empty value for example
try:
#if row has always 17 cells with values
#I can just assign it directly using a little trick
price,a_metric1,a_metric2,a_metric3,a_metric4,,price,b_metric1,b_metric2,b_metric3,b_metric4,price,c_metric1,c_metric2,c_metric3,c_metric4 = row'
except IndexError:
# if there is no 17 cells
# tell me how many cells is actually in the list
# you will see there that there less than 17 elements
print len(row)
Now you can just skip the error by assigning None value to those who don't appears in the csv file
You can read more about Catching Exception
Thanks everyone for your input - printing the results made me realize that I was getting the IndexError because of the very first row, which only had headers. Skipping that row got rid of the error.
I will look into pandas, it seems like that will be useful for the type of work I am doing.
Thanks again for all of your help, much appreciated.
Related
I want to make a loop to create a plot for each corresponding column in 2 different csv files such that column 1 in csv A and column 1 in csv B are plotted together with the same timestamp (pulled from another csv). I do not think I will have trouble when I modify my code to create the loop, but I have to get matplotlib to work for the first column before trying to construct a loop.
I have already tried checking to make sure that the correct data is being passed into the function and that is in the correct order. For example, I printed the zipped array as a list (t_array, b_array) and checked my csv files to verify that the data was in the correct order. I have also tried modifying the axes, ticks, and zoom to no avail. I have tried checking the helper functions which I lifted from my other projects and they all work as expected.
def double_plot():
before = read_file(r_before)
after = read_file(r_after)
time = read_file(timestamp)
if len(before) == len(after):
b_array = np.asarray(before[1])
a_array = np.asarray(after[1])
t_array = np.asarray(time[1])
plt.plot(t_array, b_array)
plt.plot(t_array, a_array)
plt.show()
else:
print(len(before))
print(len(after))
print("dimension failure")
read_file() is a helper function that reads csv files and saves the columns to dictionaries with the first column key indexed by key
"1" and so on down the columns. I know I should probably change it to index with 0 first, but this is a problem for later...
Images showing what I want the code to do and what it is doing
What I would like
What my code is actually doing
Thank you for your time. This is my first time posting so I apologize if something I did was incorrect. I did attempt to find the answer before posting.
Edits: data sample; read_file()
screenshot of excel
def read_file(read_file):
data = {}
with open(read_file, 'r') as f:
reader = csv.reader(f)
for row in reader:
col_num = 0
for col in row:
col_num += 1
if col_num in data:
data[col_num].append(col)
else:
ls = col
ls = [ls]
data[col_num] = ls
return data
edit again: ^ its much better to use pandas but I am leaving this here because its funny after seeing it done with dataframes
The arrays I was using with the plot function contained strings rather than floats.
These links explain the problem along with multiple ways to fix it:
Matplotlib y axis values are not ordered
In Python, how do I convert all of the items in a list to floats?
First of all, keep in mind that I'm a complete beginner with Python. I've been trying to figure this out all afternoon with no luck.
This is what I'm trying to do:
Let's say we have two csv files:
file 1:
col1;col2
659039;16,9
659038;27,8
659037:36,4
file 2:
col1;col2
659037:36,4
659039;16,9
659038;30
I want to search col1 of file 2 for all the items in col1 of file 1, and if it is found and there is a difference in col2, return that line. In the above case only the last line of file 2 would be returned, because the other lines are identical(line number doesn't matter). I only want those who are different.
Poorly explained. Hope you understand what I mean. Any help would be hugely appreciated!
Try to do one thing at a time. First, extract all the values you need to check file2 for from file1 and store them in a data structure that is easy to work with. In the example below, I looped through all of the lines in file1 and collected the contents in a dictionary. Specifically, the keys are from column one and the values are from column two.
Now, you can loop through each row in file2 and try to find any row where the content in column one doesn't exist as a key in the dictionary.If the key does exist, make sure that its value doesn't match column two. Only when both of those conditions are satisfied should you return that line.
import csv
fileItems = {}
linesToReturn = []
with open('file1.csv', newline='', encoding='utf-8-sig') as file1:
reader = csv.reader(file1, True)
for row in reader:
fileItems[row[0]] = row[1]
with open('file2.csv', newline='', encoding='utf-8-sig') as file2:
reader = csv.reader(file2, True)
for row in reader:
if fileItems.get(row[0]) != row[1]:
linesToReturn.append(row)
print(linesToReturn)
If you're using csv to search through the files, check out the documentation here.
Break down your problem into sub-problems. You can use Pandas framework to achieve this in following steps-
Read the csv file.
Use pandas framework to compare both the columns. You can refer to - https://www.shanelynn.ie/python-pandas-read_csv-load-data-from-csv-files/
If you find the desired difference, add the line to python list
return the list at the end of the code.
Good morning guys! quick question for Openpyxl:
I am working with Python editing a xlsx document and generating various stats. Part of my script is to generate max values of a cell range :
temp_list=[]
temp_max=[]
for row in sheet.iter_rows(min_row=3, min_col=10, max_row=508, max_col=13):
print(row)
for cell in row:
temp_list.append(cell.value)
print(temp_list)
temp_max.append(max(temp_list))
temp_list=[]
I would also like to be able to print the string of the header of the column that contains the max value for the cell range desired. My data structure looks like this :
Any idea on how to do so?
Thanks!
This seems like a typical INDEX/MATCH Excel problem.
Have you tried retrieving the index for the max value in each temp_list?
You can use a function like numpy.argmax() to get the index of your max value within your "temp_list" array, then use this index to locate the header and append the string to a new list called, say, "max_headers" which contains all the header strings in order of appearance.
It would look something like this
for cell in row:
temp_list.append(cell.value)
i_max = np.argmax(temp_list)
max_headers.append(cell(row = 1, column = i_max).value)
And so on and so forth. Of course, for that to work, your temp_list should be a numpy array instead of a simple python list, and the max_headers list would have to be defined.
First, Thanks Bernardo for the hint. I found a decently working solution but still have a little issue. Perhaps someone can be of assistance.
Let me amend my initial statement : here is the code I am working with now :
temp_list=[]
headers_list=[]
for row in sheet.iter_rows(min_row=3, min_col=27, max_row=508, max_col=32): #Index starts at 1 // Here we set the rows/columns containing the data to be analyzed
for cell in row:
temp_list.append(cell.value)
for cell in row:
if cell.value == max(temp_list):
print(str(cell.column))
print(cell.value)
print(sheet.cell(row=1, column=cell.column).value)
headers_list.append(sheet.cell(row=1,column=cell.column).value)
else:
print('keep going.')
temp_list = []
This formula works, but has a little issue : If, for instance, a row has the same value twice (ie : 25,9,25,8,9), this loop will print 2 headers instead of one. My question is :
how can I get this loop to take in account only the first match of a max value in a row?
You probably want something like this:
headers = [c for c in next(ws.iter_rows(min_col=27, max_col=32, min_row=1, max_row=1, values_only=True))]
for row in ws.iter_rows(min_row=3, min_col=27, max_row=508, max_col=32, values_only=True):
mx = max(row)
idx = row.index(mx)
col = headers[idx]
Let's say I have a cell (9,3). I want to get the values from (9,3) to (9,99). How do I go down the columns to get the values. I am trying to write the values into another excel file that starts from (13, 3) and ends at (13,99). How do I write a loop for that in xlrd?
def write_into_cols_rows(r, c):
for num in range (0,96):
c += 1
return (r,c)
worksheet.row(int) will return you the row, and to get the value of certain columns, you need to run row[int].value to get the value.
For more information, you can read this pdf file (Page 9 Introspecting a sheet).
import xlrd
workbook = xlrd.open_workbook(filename)
# This will get you the very first sheet in the workbook.
worksheet = workbook.sheet_by_name(workbook.sheet_names()[0])
for index in range(worksheet.nrows):
try:
row = worksheet.row(index)
row_value = [col.value for col in row]
# now row_value is a list contains all the column values
print row_value[3:99]
except:
pass
To write data to Excel file, you might want to check out xlwt package.
BTW, seems like you are doing something like reading from excel.. do some work... write to excel...
I would also recommend you take a look at numpy, scipy or R. When I usually do data munging, I use R and it saves me so much time.
I asked a question about two hours ago regarding the reading and writing of data from a website. I've spent the last two hours since then trying to find a way to read the maximum date value from column 'A' of the output, comparing that value to the refreshed website data, and appending any new data to the csv file without overriding the old ones or creating duplicates.
The code that is currently 100% working is this:
import requests
symbol = "mtgoxUSD"
url = 'http://api.bitcoincharts.com/v1/trades.csv?symbol={}'.format(symbol)
data = requests.get(url)
with open("trades_{}.csv".format(symbol), "r+") as f:
f.write(data.text)
I've tried various ways of finding the maximum value of column 'A'. I've tried a bunch of different ways of using "Dict" and other methods of sorting/finding max, and even using pandas and numpy libs. None of which seem to work. Could someone point me in the direction of a decent way to find the maximum of a column from the .csv file? Thanks!
if you have it in a pandas DataFrame, you can get the max of any column like this:
>>> max(data['time'])
'2012-01-18 15:52:26'
where data is the variable name for the DataFrame and time is the name of the column
I'll give you two answers, one that just returns the max value, and one that returns the row from the CSV that includes the max value.
import csv
import operator as op
import requests
symbol = "mtgoxUSD"
url = 'http://api.bitcoincharts.com/v1/trades.csv?symbol={}'.format(symbol)
csv_file = "trades_{}.csv".format(symbol)
data = requests.get(url)
with open(csv_file, "w") as f:
f.write(data.text)
with open(csv_file) as f:
next(f) # discard first row from file -- see notes
max_value = max(row[0] for row in csv.reader(f))
with open(csv_file) as f:
next(f) # discard first row from file -- see notes
max_row = max(csv.reader(f), key=op.itemgetter(0))
Notes:
max() can directly consume an iterator, and csv.reader() gives us an iterator, so we can just pass that in. I'm assuming you might need to throw away a header line so I showed how to do that. If you had multiple header lines to discard, you might want to use islice() from the itertools module.
In the first one, we use a "generator expression" to select a single value from each row, and find the max. This is very similar to a "list comprehension" but it doesn't build a whole list, it just lets us iterate over the resulting values. Then max() consumes the iterable and we get the max value.
max() can use a key= argument where you specify a "key function". It will use the key function to get a value and use that value to figure the max... but the value returned by max() will be the unmodified original value (in this case, a row value from the CSV). In this case, the key function is manufactured for you by operator.itemgetter()... you pass in which column you want, and operator.itemgetter() builds a function for you that gets that column.
The resulting function is the equivalent of:
def get_col_0(row):
return row[0]
max_row = max(csv.reader(f), key=get_col_0)
Or, people will use lambda for this:
max_row = max(csv.reader(f), key=lambda row: row[0])
But I think operator.itemgetter() is convenient and nice to read. And it's fast.
I showed saving the data in a file, then pulling from the file again. If you want to go through the data without saving it anywhere, you just need to iterate over it by lines.
Perhaps something like:
text = data.text
rows = [line.split(',') for line in text.split("\n") if line]
rows.pop(0) # get rid of first row from data
max_value = max(row[0] for row in rows)
max_row = max(rows, key=op.itemgetter(0))
I don't know which column you want... column "A" might be column 0 so I used 0 in the above. Replace the column number as you like.
It seems like something like this should work:
import requests
import csv
symbol = "mtgoxUSD"
url = 'http://api.bitcoincharts.com/v1/trades.csv?symbol={}'.format(symbol)
data = requests.get(url)
with open("trades_{}.csv".format(symbol), "r+") as f:
all_values = list(csv.reader(f))
max_value = max([int(row[2]) for row in all_values[1:]])
(write-out-the-value?)
EDITS: I used "row[2]" because that was the sample column I was taking max of in my csv. Also, I had to strip off the column headers, which were all text, which was why I looked at "all_values[1:]" from the second row to the end of the file.