Error:
Traceback (most recent call last):
File "son.py", line 120, in <module>
`main()`
File "son.py", line 101, in main
`temp += item`
TypeError: 'ApplyResult' object is not iterable
Code:
pool = multiprocessing.Pool(processes=int(args.process))
for i in range(int(args.process)):
result_first_round.append(pool.apply_async(Son_Algorithm, (i,)))
pool.close()
pool.join()
first_temp = []
for res in result_first_round:
first_temp.append(res)
#first_temp.append(res.get())
#Second Scan
result_final_round = []
temp = []
for item in first_temp:
temp += item
temp2 = []
for result in temp:
if not result in temp2:
temp2.append(result)
temp_result = temp2
It seems that you want to add the element item to the list temp. In that case you need to use the method append(), like this:
temp = []
for item in first_temp:
temp.append(item)
The operator += for a list only works if the second object is also a list (or at least an iterable).
Related
Trying to replicate this repository: https://github.com/sujiongming/UCF-101_video_classification. I get the following error when I run the CNN_train_UCF101.py file.
Traceback (most recent call last):
File "CNN_train_UCF101.py", line 18, in <module>
data = DataSet()
File "D:\Clones\UCF-101_video_classification-master\UCFdata.py", line 32, in __init__
self.classes = self.get_classes()
File "D:\Clones\UCF-101_video_classification-master\UCFdata.py", line 64, in get_classes
if item[1] not in classes:
IndexError: list index out of range
part of the code referenced is as follows:
def get_data():
"""Load our data from file."""
with open('./data/data_file.csv', 'r') as fin:
reader = csv.reader(fin)
data = list(reader)
def clean_data(self):
"""Limit samples to greater than the sequence length and fewer
than N frames. Also limit it to classes we want to use."""
data_clean = []
for item in self.data:
if int(item[3]) >= self.seq_length and int(item[3]) <= self.max_frames \
and item[1] in self.classes:
data_clean.append(item)
return data_clean
def get_classes(self):
"""Extract the classes from our data. If we want to limit them,
only return the classes we need."""
classes = []
for item in self.data:
if item[1] not in classes:
classes.append(item[1])
# Sort them.
classes = sorted(classes)
# Return.
if self.class_limit is not None:
return classes[:self.class_limit]
else:
return classes
I have updated the question to give clarity on data.
When I do print (self.data) I get something like this:
['train', 'UnevenBars', 'v_UnevenBars_g22_c04', '126'], [] for each image in the dataset.
Can anyone please show me what I'm doing wrong. Thanks in advance.
Window 10
Python 3.7.6
You have a blank line in the CSV file, which is resulting in an empty list at the end of self.data.
You should skip empty items.
for item in self.data:
if len(item) < 2:
continue
if item[1] not in classes:
classes.append(item[1])
I am working on a program that parses through log files and returns the top hits for IP addresses and a couple other things. Currently I am having trouble and I cannot interpret any of the answers to this problem to what I have going on right now. This is all of my code:
import gzip
from collections import Counter
logFileName = open('C:\\Users\\Pawlaczykm\\Desktop\\fileNames.txt', 'r')
ipAdd = []
landingPages = []
ALL_ipAdd = []
ALL_landingPages = []
# everything after this line gets done to all files
for line in logFileName.readlines():
# rstrip removes a blank line from output
# print 'Summary of: ' + line.rstrip()
# use gzip to decompress the file
with gzip.open('C:\\Users\\Pawlaczykm\\Desktop\\logFiles\\' + line.rstrip() + '.gz', 'rb') as f:
# we extract the ip addresses in lines 15-18
for eachLine in f:
parts = eachLine.split('\t')
if len(parts) > 1:
ipAdd.append(parts[2])
ALL_ipAdd.append(ipAdd)
# use gzip to decompress the file
with gzip.open('C:\\Users\\Pawlaczykm\\Desktop\\logFiles\\' + line.rstrip() + '.gz', 'rb') as f:
# we extract the landing pages
for eachLine in f:
parts = eachLine.split('\t')
if len(parts) > 1:
variable = parts[8].split('?')[0]
landingPages.append(variable)
v): (-v, k))[:10]
ALL_landingPages.append(landingPages)
ALL_ipAddDict = dict(Counter(ALL_ipAdd).most_common())
sortedALL_ipAdd = sorted(ALL_ipAddDict.iteritems(), key=lambda (k, v): (-v, k))[:10]
print 'Top IPs of all files'
print(sortedALL_ipAdd)
ALL_LandingPageDict = dict(Counter(ALL_landingPages).most_common())
sortedALL_LandingPage = sorted(ALL_LandingPageDict.iteritems(), key=lambda (k, v): (-v, k))[:10]
print 'Top landing pages of all files'
print (sortedALL_LandingPage)
Now where I am having trouble is in the following line:
ALL_ipAddDict = dict(Counter(ALL_ipAdd).most_common())
The output when I run the whole program is this:
Traceback (most recent call last):
File "C:/Users/Pawlaczykm/PycharmProjects/LogParse/parseText.py", line 35, in <module>
ALL_ipAddDict = dict(Counter(ALL_ipAdd).most_common())
File "C:\Python27\lib\collections.py", line 477, in __init__
self.update(*args, **kwds)
File "C:\Python27\lib\collections.py", line 567, in update
self[elem] = self_get(elem, 0) + 1
TypeError: unhashable type: 'list'
Can somebody help me? This is frustrating.
From your code ALL_ipAdd = [] and ipAdd = [] and ALL_ipAdd.append(ipAdd) we can conclude that ALL_ipAdd is a list of list. Counter is a subtype of dict, which hashes its items before it counts them. Lists cannot be hashed because they are mutable (if the list changed the hash would change) and thus lists can't be counted by Counter objects.
To solve this you can convert the inner lists to tuples before counting them:
ALL_ipAddDict = dict(Counter(map(tuple, ALL_ipAdd)).most_common())
That's normal. ALL_ipAdd is a list of lists. Counter needs a list, a string or any other hashable type :)
I have the following MWE:
def get_files():
file_list = ['first', 'second', 'third', 'fourth']
return file_list
def set_names(orig_flist):
file_list = []
for i in range(len(orig_flist)):
file_list[i] = orig_flist[i]
return file_list
set_names(get_files())
When I run it, I get this error:
Traceback (most recent call last):
File "privpub.py", line 11, in <module>
set_names(get_files())
File "privpub.py", line 8, in set_names
file_list[i] = orig_flist[i]
IndexError: list assignment index out of range
I don't understand what's going on. Can somebody explain me, please?
You are trying to assign a value to an index in the list that does not exist yet:
file_list = []
for i in range(len(orig_flist)):
file_list[i] = orig_flist[i]
You will want to use append in order to lengthen your list like so:
file_list = []
for i in range(len(orig_flist)):
file_list.append(orig_flist[i])
I'm only a beginner python user so my apologies if this is a rather simple question. I have a file containing two lists divided by a tab. I would like to store this in a dictionary, so each entry is associated with the corresponding entry after the tab, such that:
cat hat
mouse bowl
rat nose
monkey uniform
dog whiskers
elephant dance
would be divided into
{'cat'; 'hat', 'mouse' ; 'bowl') etc. etc.
It's a very long list.
This is what I tried:
enhancerTAD = open('TAD_to_enhancer.map', 'r')
list = enhancerTAD.split()
for entry in list:
key, val = entry.split('\t')
ET[key] = val
print ET
Here's my most recent attempt, and the error message that I get below:
enhancerTAD = open('TAD_to_enhancer.map', 'r').read()
ET = {}
lst = enhancerTAD.split("\n")
for entry in lst:
key, val = entry.strip().split(' ',1)
ET[key] = val
enhancergene = open('enhancer_to_gene_map.txt', 'r').read()
GE = {}
lst1 = enhancergene.split("\n")
for entry in lst1:
key, val = entry.strip().split(' ',1)
GE[key] = val
geneTAD = open('TAD_to_gene_map.txt', 'r').read()
GT = {}
lst2 = geneTAD.split("\n")
for entry in lst2:
key, val = entry.strip().split(' ',1)
GT[key] = val
File "enhancertadmaybe.py", line 13, in
key, val = entry.strip().split(' ',1)
ValueError: need more than 1 value to unpack
You can try:
with open('foo.txt', 'r') as f:
print dict(line.strip().split('\t', 1) for line in f)
Result:
{'monkey': 'uniform', 'dog': 'whiskers', 'cat': 'hat', 'rat': 'nose', 'elephant': 'dance', 'mouse': 'bowl'}
Modification to your original method :
enhancerTAD = open('TAD_to_enhancer.map', 'r').read()
ET={}
lst = enhancerTAD.split("\n")
for entry in lst:
key, val = entry.strip().split('\t',1)
ET[key] = val
print ET
Points:
1.Your original method failed because were trying to split on a file object not the file content
i.e)
a=open("amazon1.txt","r")
c=a.split()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'file' object has no attribute 'split'
2.You have to read the content of the file to split it
i.e.)
enhancerTAD =open("amazon1.txt","r").read()
3.Since you have key,value pair in each line you have to split at new line initially
4.Then you can iterate over the list and again split it at \t and form the dictionary
Juniorcomposer method does all of this two line code and is more pythonic
I am trying to parallel process some file by reading chunks and process each chunk by using multiprocessing libraries. Following is my code:
from multiprocessing import Pool
from itertools import islice
import traceback
#Produce key value pairs (Date, Market_Share*Market_Share)
def Map(L):
results = []
for w in L:
temp = w.split(',')
Date = temp[0]
Share = float(temp[1][:-1])
ShareSquare = str(Share*Share)
results.append((Date,ShareSquare))
return results
if __name__=='__main__':
pool = Pool(2)
f = open('C:/Users/Daniel/Desktop/Project/Optiver/atchm_9450.csv','r')
fw = open('C:/Users/Daniel/Desktop/Project/Optiver/marketshare.csv','w')
f.readline()
while True:
next_n_lines = list(islice(f,16))
if not next_n_lines:
break
else:
l = pool.map(Map,next_n_lines)
f.close()
fw.close()
However, it produces index out of range error:
Traceback (most recent call last):
File "trial.py", line 29, in <module>
l = pool.map(Map,next_n_lines)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
IndexError: list index out of range
The list object I passed into the Map function is something like ['6/26/2014,68.90\n', '6/27/2014,68.84\n', '6/30/2014,68.80\n'....]
It works correctly when there is no parallelism involved (pool is not invoked).
What possibly causes this behavior?
At first glance, only those two lines can raise this exception:
Date = temp[0]
Share = float(temp[1][:-1])
Try to check that w have enough data.