I have a function, which takes a CSV and processes it. I am trying to count the rows in the CSV file before running through it to ensure I get to the end.
def parse_campaigns_cutsheet(country_code, input_file, DB, DB_hist, zip_id):
filecontent = urllib2.urlopen(input_file)
count = 0
csvFile = csv.DictReader(filecontent)
rowage = len(list(csvFile))
for row in csvFile:
count += 1
if 'MM' not in row['Tier']:
continue
if RB_COUNTRIES_new[country_code]['cut_sheet_country'] != row['Country']:
continue
document = DB.find_one({'rb_account_id': RB_COUNTRIES_new[country_code]['rb_account_id']})
if document is None:
continue
DB.save(document)
report_work(document, DB, DB_hist)
I keep getting the following error UnboundLocalError: local variable 'document' referenced before assignment. If i remove the rowage = len(list(csvFile)) line it works fine?
This happens because the DictReader is a generator.
When you call list on DictReader, it will yield all the values into the list, and won't be iterable any more.
That why when your for-loop tries to iterate over it, it never does, and the document will never be assigned.
If you want to accomplish what you're trying to do, you can keep a reference to the list and then iterate over the list:
...
csvFile = csv.DictReader(filecontent)
filecontent_list = list(csvFile)
rowage = len(filecontent_list)
for row in filecontent_list:
...
Keep in mind - this is mean that all your data will be saved in memory!
When iterating over the generator without forcing it to be a list, only one iterating are being saved in the memory each time.
Related
I'm trying to manipulate text from a word file however when I save it to an array of classes, all the indexes are being overwritten instead of the one particular index I intend to change.
for line in modified:
if line.startswith('Date'):
output.append(line)
list2=line.split(' ')
work.date=list2[1]
# print(work.date)
if line.startswith('PPV'): #list1[2]=l,[3]=t,[4]=v
output.append(line)
list1=line.split(' ')
work.lpv=list1[2]
# print("l is ",list1[2],work.lpv)
work.tpv=list1[3]
# print("t is ",list1[3],work.tpv)
work.vpv=list1[4]
# print("v is ",list1[4],work.vpv)
daylist[count]=work
#print("l2 is ",list1[2],work.lpv)
#print("daylist", count, "saved")
print(count,daylist[count].date) #this displays the correct value at the propper index but all other indexs have also been changed to this value
count+=1
Im trying to save a class which holds a string and a few floats to an array but cannot seem to get it to save to each index properly as it is read from the file. ive tried messing with the scope and list initialization but cant seem to figure it out. Any input would be appreciated, Thanks!
Every index in the daylist array references the same work object. When you change an attribute of work (e.g. work.date) it's reflected in all references to that single object. You want each index to reference a separate, independent object but that's not what the code is doing.
Try something like this where work is a dictionary:
for line in modified:
work = {} # <-- this makes the name "work" refer to a new, empty dict
if line.startswith('Date'):
output.append(line)
list2=line.split(' ')
work["date"] = list2[1]
elif line.startswith('PPV'):
output.append(line)
list1=line.split(' ')
work["lpv"] = list1[2]
# ...
print(count, daylist[count]["date"])
count += 1
Here's a helpful link on how names reference objects: How does Python referencing work?
I have a two lists that are zipped together, i am able to print the list out to view but i when i try to export the list into a csv file, the csv file is created but its empty. not sure why as im using the same method to save the two lists separately and it works.
import csv
import random
import datetime
import calendar
with open("Duty Name List.csv") as CsvNameList:
NameList = CsvNameList.read().split("\n")
date = datetime.datetime.now()
MaxNumofDays = calendar.monthrange(date.year,date.month)
print (NameList)
print(date.year)
print(date.month)
print(MaxNumofDays[1])
x = MaxNumofDays[1] + 1
daylist = list(range(1,x))
print(daylist)
ShuffledList = random.sample(NameList,len(daylist))
print(ShuffledList)
RemainderList = set(NameList) - set(ShuffledList)
print(RemainderList)
with open("remainder.csv","w") as f:
wr = csv.writer(f,delimiter="\n")
wr.writerow(RemainderList)
AssignedDutyList = zip(daylist,ShuffledList)
print(list(AssignedDutyList))
with open("AssignedDutyList.csv","w") as g:
wr = csv.writer(g)
wr.writerow(list(AssignedDutyList))
no error messages are produced.
In Python 3, This line
AssignedDutyList = zip(daylist,ShuffledList)
creates an iterator named AssignedDutyList.
This line
print(list(AssignedDutyList))
exhausts the iterator. When this line is executed
wr.writerow(list(AssignedDutyList))
the iterator has no further output, so nothing is written to the file.
The solution is to store the result of calling list on the iterator in a variable rather than the iterator itself, in cases where the content of an iterator must be reused.
AssignedDutyList = list(zip(daylist,ShuffledList))
print(AssignedDutyList)
with open("AssignedDutyList.csv","w") as g:
wr = csv.writer(g)
wr.writerow(AssignedDutyList)
As a bonus, the name AssignedDutyList now refers to an actual list, and so is less confusing for future readers of the code.
I am writing a script to read a .osu file and convert it to specific objects. This has to be done multiple times for each "hitobject"
The reading part works fine, however appending the object is the problem
When appending the object, it seems to overwrite all existing elements in the list. I can not for the life of me figure out why this is happening.
I have tried creating a "temp" list, which stores the objects in a local list instead of the "self.notes" list, still the same issue.
I believe the error is occurring in this part of the file:
if hitobjline != -1:
hitobjects = self.file_lines[hitobjline+1:]
for i in hitobjects:
ln = i[:i.find(':')].split(',')
new_note = [NoteType.Circle, NoteType.Hold][int(ln[3] == '128')]
add_note = File.Note
add_note.NoteTypeByte = ln[3]
add_note.Note_Number = int(ln[0])
add_note.Time = int(ln[2])
add_note.NoteType = new_note
add_note.Raw = ln
self.notes.append(add_note)
print(ln, ln[3], ln[3] == '128', new_note, add_note.NoteType)
For background, the .osu files have a syntax like: x,y,time,type,hit,end:stuff-i-dont-need-to-worry-about
I expected an out put of self.notes[0].NoteType to be osureader.NoteType.Hold as the first line of the file is 192,192,410,128,0,2974:0:0:0:0: (the 128 indicating the 'Hold'
However, I get, osureader.NoteType.Circle, the last line of the file.
In this function I'm reading from a .txt file, and store the values in a dictionary. I want to be able to pass this dictionary to another function, to do further calculations and sorting.
I can manage to print all rows from the .txt file, but that's it.
Return breaks the loop and only gives the first row.
Global variables and nested functions are bad form.
Have tried to use yield (for the first time), but that only prints "generator object get_all_client_id at 0x03369A20"
file_with_client_info = open("C:\\Users\\clients.txt", "r")
def get_all_client_id():
client_details = {}
for line in file_with_client_info:
element = line.split(",")
while element:
client_details['client_id'] = element[0]
client_details['coordinates'] = {}
client_details['coordinates']['lat'] = element[1]
client_details['coordinates']['long'] = element[2]
break
print(client_details)
There are a few errors in you code.
Use a return statement to output the dictionary.
The while-loop does not loop as you are breaking on the first iteration. Use an if-statement to check if the line is empty instead.
The last entries in the client_details dict are overwritten on each iteration. Create a new entry instead, probably using the client_id as key.
It is recommended you use a with context manager to open your file.
It is preferable to provide the name of your file to your function and let it open it instead of having a globally opened file.
Here is a fixed version of your code.
def get_all_client_id(file):
client_details = {}
with open(file, 'r') as f:
for line in f:
element = line.strip().split(',')
if element:
client_id, lat, long, *more = element
client_details[client_id] = {'lat': lat, 'long': long}
return client_details
clients_dict = get_all_client_id("C:\\Users\\clients.txt")
I'm writing a Python script that reads a CSV file and creates a list of deques. If I print out exactly what gets appended to the list before it gets added, it looks like what I want, but when I print out the list itself I can see that append is overwriting all of the elements in the list with the newest one.
# Window is a list containing many instances
def slideWindow(window, nextInstance, num_attributes):
attribute = nextInstance.pop(0)
window.popleft()
for i in range(num_attributes):
window.pop()
window.extendleft(reversed(nextInstance))
window.appendleft(attribute)
return window
def convertDataFormat(filename, window_size):
with open(filename, 'rU') as f:
reader = csv.reader(f)
window = deque()
alldata = deque()
i = 0
for row in reader:
if i < (window_size-1):
window.extendleft(reversed(row[1:]))
i+=1
else:
window.extendleft(reversed(row))
break
alldata.append(window)
for row in reader:
window = slideWindow(window, row, NUM_ATTRIBUTES)
alldata.append(window)
# print alldata
f.close()
return alldata
This is really difficult to track what you exactly want from this code. I suspect the problem lies in the following:
alldata.append(window)
for row in reader:
window = slideWindow(window, row, NUM_ATTRIBUTES)
alldata.append(window)
Notice that in your slideWindow function, you modify the input deque (window), and then return the modified deque. So, you're putting a deque into the first element of your list, then you modify that object (inside slideWindow) and append another reference to the same object onto your list.
Is that what you intend to do?
The simple fix is to copy the window input in slideWindow and modify/return the copy.
I don't know for sure, but I'm suspicious it might be similar to this problem http://forums.devshed.com/python-programming-11/appending-object-to-list-overwrites-previous-842713.html.