Pool Multiprocessing Python - python

Basically the issue is as follows: I have a bunch of workers that have a function prescribed to each (the function is worker(alist) ) and am trying to process 35 workers at the same time. Each worker reads their line from the file (the modulo part) and should process the line using the "worker" function. I've pen-tested and found that the raw manipulation and deletion of the useless indices is working 100% as intended.
The args part of the "pool.apply_async" function isn't passing the list "raw" into it and starting the process. Raw is completely correct and functions normally, worker by itself functions normally, the pool.apply_async function is the only place that there seems to be an issue and I have no idea how to fix it. Any help please?
The relevant code is here:
NUM_WORKERS=35
f=open("test.csv")
pool=multiprocessing.Pool()
open("final.csv",'w')
for workernumber in range(1, NUM_WORKERS):
for i,line in enumerate(f):
if i==0:
print "Skipping first line" #dont do anything
elif i%workernumber==0:
raw = line.split(',')[0][1:-1].split()
uselessindices=[-2,-3,-4,-5,-6]
counter=0
for ui in uselessindices:
del raw[ui+counter]
counter+=1
print raw
pool.apply_async(worker, args=(raw,))
pool.close()
pool.join()


I suggest you put the calculation of raw into a generator function, and then use Pool.imap_unordered() or Pool.map() to run worker() over all of the items in the generator.
Something like this untested code:
def get_raw():
with open("test.csv", 'rU') as f:
for i, line in enumerate(f):
if i == 0:
# skip header
continue
raw = line.split(',')[0][1:-1].split()
uselessindices=[-2,-3,-4,-5,-6]
counter=0
for ui in uselessindices:
del raw[ui+counter]
counter+=1
yield raw
pool=multiprocessing.Pool(processes=NUM_WORKERS)
pool.map(worker, get_raw())
pool.close()
pool.join()


import multiprocessing
def worker(arg):
print 'doing work "%s"' % arg
return
NUM_WORKERS=35
with open('test.csv', 'w') as test:
for i in xrange(100):
if i % 10 == 0:
test.write('\n')
test.write('"%s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23",' % i)
f=open("test.csv")
pool=multiprocessing.Pool(processes=NUM_WORKERS)
open("final.csv",'w')
for i, line in enumerate(f):
if i == 0:
continue
raw = line.split(',')[0][1:-1].split()
uselessindices=[-2,-3,-4,-5,-6]
counter=0
for ui in uselessindices:
del raw[ui+counter]
counter+=1
pool.apply_async(worker, args=(raw,))
pool.close()
pool.join()
print 'last raw len: %s' % len(raw)
print 'last raw value: %s' % raw
Output:
doing work "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['10', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['20', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['30', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['40', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['50', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['60', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['70', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['80', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
doing work "['90', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
last raw len: 19
last raw value: ['90', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']


So I found out it wasn't throwing an error that was occurring inside the worker as a result of a mismatched number of inputs into a child function (AKA worker was calling another function dosomething(a1,a2,...a20) and was only giving it 19 inputs). It seems async won't throw error outputs about issues happening inside the worker which is quite annoying but I now understand. Thanks for all the help!

Related

Lucky Numbers in Python

For who's interested in; lucky numbers are generated by eliminating numbers based on their position in the set. i.e:
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22
First eliminates every second number bc second value is 2:
1,3,5,7,9,11,13,15,17,19,21
First remaining number in the list after 1 is 3; so it eliminates every third number:
1,3,7,9,13,15,17,19,21
Next number is 7; eliminate every 7th number in the list:
1,3,7,9,13,15,21
Next surviving nunmber after 7 is 9 but obviously there are not enough numbers to eliminate.
For further information you can check Lucky Number
So, if my list doesn't contain any negative number and begins with 1 i.e:
numbers = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22']
My code supposed to delete every number unless it's a lucky number, so I tried:
def lucky_numbers(numbers):
del numbers[::-2] # Delete even numbers
while int(numbers[1]) < len(numbers):
x = int(numbers[1])
del numbers[-1::x]
print(numbers)
return
return
lucky_numbers(numbers)
But it returns:
['1', '3', '5', '7', '9', '11', '13', '15', '17', '19']
Where am I wrong? Or is there any efficient way to write it? Thank you.

The negative index is a bit confusing to me (at least), see if this code is easy to interpret-
numbers = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22']
def lucky_numbers(numbers):
index = 1
next_freq = int(numbers[index])
while int(next_freq) < len(numbers):
del numbers[next_freq-1::next_freq]
print(numbers)
if str(next_freq) in numbers:
index += 1
next_freq = int(numbers[index])
else:
next_freq = int(numbers[index])
return
lucky_numbers(numbers)
['1', '3', '5', '7', '9', '11', '13', '15', '17', '19', '21']
['1', '3', '7', '9', '13', '15', '19', '21']
['1', '3', '7', '9', '13', '15', '21']

create a maximum of 8 items per row(python)

i am extremely new to python. i have googled but i haven't found what i need for hours.
i'm very confused on why my program is like this and would very much appreciate pointers. so this is my program:
number = int(input("Enter number: "))
mylist = []
if number > 1:
for num in range(1,number + 1):
if num > 1:
for i in range(2,num):
if (num % i) == 0:
break
else:
mylist.append(str(num))
print(mylist)
else:
print("no prime number")
so e.g if i insert 50 i wanted it to have an output of something like:
['2', '3', '5', '7', '11', '13', '17', '19',
'23', '29', '31', '37', '41', '43', '47']
but instead what i got was this chaos
['2']
['2', '3']
['2', '3', '5']
['2', '3', '5', '7']
['2', '3', '5', '7', '11']
['2', '3', '5', '7', '11', '13']
['2', '3', '5', '7', '11', '13', '17']
['2', '3', '5', '7', '11', '13', '17', '19']
['2', '3', '5', '7', '11', '13', '17', '19', '23']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', '37']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', '37', '41']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', '37', '41', '43']
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', '37', '41', '43', '47']

Your print statement is indented incorrectly:
number = int(input("Enter number: "))
mylist = []
if number > 1:
for num in range(1,number + 1):
if num > 1:
for i in range(2,num):
if (num % i) == 0:
break
else:
mylist.append(str(num))
print(mylist)
else:
print("no prime number")

You can group them into sublists of 8 for the purpose of printing. Just make your mylist as usual by appending the values then after you have processed all the values you can print sublists of 8
number = int(input("Enter number: "))
mylist = []
if number > 1:
for num in range(1,number + 1):
if num > 1:
for i in range(2,num):
if (num % i) == 0:
break
else:
mylist.append(str(num))
print(*[mylist[i:i+8] for i in range(0, len(mylist), 8)], sep="\n")
else:
print("no prime number")
OUTPUT
['2', '3', '5', '7', '11', '13', '17', '19']
['23', '29', '31', '37', '41', '43', '47']

Removing all non digits from a list

After reading from a file I have a list of lists contaning not only digits but also other characters, which I would like to get rid of.
I've tried using re.sub function but this doesn't seem to work
import re
Poly_id= [['0', '[4', '8', '18', '20', '5', '0', '4]'], ['1', '[13', '16',
'6', '11', '13]'], ['2', '[3', '1', '10', '9', '2', '15', '3]'], ['3',
'[13', '12', '16', '13]'], ['4', '[13', '11', '17', '14', '7', '13]']]
for x in Poly_id:
[re.sub(r'\W', '', ch) for ch in x]
This doesn't seem to change a thing in this list.
I would like to have a list with only numbers as elements so that I could convert them into integers

I guess technically [4 is non numeric so you can do something like this:
Poly_id = [[char for char in _list if str.isnumeric(char)] for _list in Poly_id]
Output:
['0', '8', '18', '20', '5', '0']
['1', '16', '6', '11']
['2', '1', '10', '9', '2', '15']
['3', '12', '16']
['4', '11', '17', '14', '7']
If you just want to remove the non numeric values and not the complete entry then you can do this:
Poly_id = [[''.join(char for char in substring if str.isnumeric(char)) for substring in _list] for _list in Poly_id]
Output:
['0', '4', '8', '18', '20', '5', '0', '4']
['1', '13', '16', '6', '11', '13']
['2', '3', '1', '10', '9', '2', '15', '3']
['3', '13', '12', '16', '13']
['4', '13', '11', '17', '14', '7', '13']

Here a solution if you want to get rid of the '[' in '[4' but keep the '4':
res = [[re.sub(r'\W', '', st) for st in inlist] for inlist in Poly_id]
res is:
[
['0', '4', '8', '18', '20', '5', '0', '4'],
['1', '13', '16', '6', '11', '13'],
['2', '3', '1', '10', '9', '2', '15', '3'],
['3', '13', '12', '16', '13'],
['4', '13', '11', '17', '14', '7', '13']
]

You can use a module, "itertools"
import itertools
list_of_lists = [[1, 2], [3, 4]]
print(list(itertools.chain(*list_of_lists)))
>>>[1, 2, 3, 4]

how to sort python list in a way that if 1 should come before and 10 and 2 before 20

['2', '8', '2', '3', '6', '4', '1', '1', '10', '6', '3', '3', '6', '1', '3', '8', '4', '6', '1', '10', '8', '4', '10', '4', '1', '3', '2', '3', '2', '6', '1', '5', '2', '9', '8', '5', '10', '8', '7', '9', '6', '4', '2', '6', '3', '8', '8', '9', '8', '2', '9', '10', '3', '10', '7', '5', '7', '1', '7', '5', '1', '4', '7', '6', '1', '10', '5', '4', '8', '4', '2', '7', '8', '1', '1', '7', '4', '1', '1', '9', '8', '6', '5', '9', '9', '3', '7', '6', '3', '10', '8', '10', '7', '2', '5', '1', '1', '9', '9', '5']
after using lambda function inf following way:
a.sort(key=lambda a: int(a.split()[0]))
a = a[::-1]
I got
['10', '10', '10', '10', '10', '10', '10', '10', '10', '9', '9', '9', '9', '9', '9', '9', '9', '9', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '7', '7', '7', '7', '7', '7', '7', '7', '7', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '5', '5', '5', '5', '5', '5', '5', '5', '4', '4', '4', '4', '4', '4', '4', '4', '4', '3', '3', '3', '3', '3', '3', '3', '3', '3', '3', '2', '2', '2', '2', '2', '2', '2', '2', '2', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1']
But i want
10 in end after 1 , likewise if put 20 and 2 in list than 2 should come before 20 and 20 before 10 etc

The operation:
a.sort()
with no other options sets a to:
['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '10', '10', '10', '10', '10', '10', '10', '10', '10', '2', '2', '2', '2', '2', '2', '2', '2', '2', '3', '3', '3', '3', '3', '3', '3', '3', '3', '3', '4', '4', '4', '4', '4', '4', '4', '4', '4', '5', '5', '5', '5', '5', '5', '5', '5', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '7', '7', '7', '7', '7', '7', '7', '7', '7', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '9', '9', '9', '9', '9', '9', '9', '9', '9']

You want an ordered list of strings. Those strings are representations of numbers. Now you want a grouped sorting. First everything that starts with a '9', then '8' and down to '1'. In each of those groups the values should be sorted in numeric order.
An example list:
a = ['11', '105', '2', '8', '2', '3', '6', '4', '1', '1', '10', '81', '3', '3', '5', '10', '8', '7', '9', '6', '4', '2']
Now let's do a grouped sorting with a.sort(key=lambda v: v[0]):
['11', '105', '1', '1', '10', '10', '2', '2', '2', '3', '3', '3', '4', '4', '5', '6', '6', '7', '8', '81', '8', '9']
We see, that the values are grouped now, but we want the values starting with '9' first. We're going to fix this by reversing the result with a.sort(key=lambda v: v[0], reversed=True)
['9', '8', '81', '8', '7', '6', '6', '5', '4', '4', '3', '3', '3', '2', '2', '2', '11', '105', '1', '1', '10', '10']
The groups are correct, now we have to sort the values in the groups. So after the sorting according to the first character we have to sort the value by number. That's easy, we just have to create a tuple for the key: a.sort(key=lambda v: (v[0], int(v)), reverse=True)
['9', '81', '8', '8', '7', '6', '6', '5', '4', '4', '3', '3', '3', '2', '2', '2', '105', '11', '10', '10', '1', '1']
OK, the values are sorted now, but we have to reverse them in the groups. The easiest way to do that ist to take the negative number: a.sort(key=lambda v: (v[0], -int(v)), reverse=True).
['9', '8', '8', '81', '7', '6', '6', '5', '4', '4', '3', '3', '3', '2', '2', '2', '1', '1', '10', '10', '11', '105']

you can use the following sample using key=str
integers = ['2', '8', '2', '3', '6', '4', '1', '1', '10', '6', '3', '3', '6', '1', '3', '8', '4', '6', '1', '10', '8', '4', '10', '4', '1', '3', '2', '3', '2', '6', '1', '5', '2', '9', '8', '5', '10', '8', '7', '9', '6', '4', '2', '6', '3', '8', '8', '9', '8', '2', '9', '10', '3', '10', '7', '5', '7', '1', '7', '5', '1', '4', '7', '6', '1', '10', '5', '4', '8', '4', '2', '7', '8', '1', '1', '7', '4', '1', '1', '9', '8', '6', '5', '9', '9', '3', '7', '6', '3', '10', '8', '10', '7', '2', '5', '1', '1', '9', '9', '5']
print(sorted(integers, key=str))

you don't need to use lambda method here. Instead of using lambda you can use '.sort()' method to sort the items in the list. like this one:
li=['2', '8', '2', '3', '6', '4', '1', '1', '10', '6', '3', '3', '6', '1', '3', '8', '4', '6', '1', '10', '8', '4', '10', '4', '1', '3', '2', '3', '2', '6', '1', '5', '2', '9', '8', '5', '10', '8', '7', '9', '6', '4', '2', '6', '3', '8', '8', '9', '8', '2', '9', '10', '3', '10', '7', '5', '7', '1', '7', '5', '1', '4', '7', '6', '1', '10', '5', '4', '8', '4', '2', '7', '8', '1', '1', '7', '4', '1', '1', '9', '8', '6', '5', '9', '9', '3', '7', '6', '3', '10', '8', '10', '7', '2', '5', '1', '1', '9', '9', '5','20']
li.sort()
print(li)
output:
['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '10', '10', '10', '10', '10', '10', '10', '10', '10', '2', '2', '2', '2', '2', '2', '2', '2', '2', '20', '3', '3', '3', '3', '3', '3', '3', '3', '3', '3', '4', '4', '4', '4', '4', '4', '4', '4', '4', '5', '5', '5', '5', '5', '5', '5', '5', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '7', '7', '7', '7', '7', '7', '7', '7', '7', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '9', '9', '9', '9', '9', '9', '9', '9', '9']
I hope you get your answere.

This might helps you:
a.sort(key=str)
or
new list = sorted(a, key=str) # if you dont want to change a
this will sort the list as you want even if the items are integer

get max and min value in dictionary

Illinois: ['13', '12', '18', '23', '26', '25', '24', '19', '13', '10', '15', '14', '14', '4', '3']
Indiana: ['7', '6', '7', '8', '11', '11', '13', '12', '7', '7', '7', '7', '9', '2', '2']
Those are in my dictionary as d.
How would I get the largest and smallest value in each key in the dictionary and get the index where's the value is.
For example:
In Illinois, 26 is the largest value which is index 5 and 3 is the smallest value which is index 15.
in Indiana: 13 is largest value which is index 7 and 2 is the smallest value which is index 14
The output:
Illinois: 26 in index 5 and 3 in index 15
Indiana: 13 in index 7 and 2 in index 14
How would I do this?
d = {}
for row in csv_f:
d[row[0]]=row[1:]

You can get the max and mins printed out as your string is like this:
(assuming you only want the first occurrence)
MY_D = {'Illinois': ['13', '12', '18', '23', '26', '25', '24', '19', '13', '10', '15', '14', '14', '4', '3'],
'Indiana': ['7', '6', '7', '8', '11', '11', '13', '12', '7', '7', '7', '7', '9', '2', '2']}
for k,v in MY_D.items():
#This assumes that everything in v is an int, or rather can be converted to one.
my_l = [int(n) for n in v]
#if not
#my_l = [int(n) for n in v if n.isdigit()]
_max, _min = max(my_l), min(my_l)
print("%s: Min - %d in index %d, Max - %d in index %d" % (k, _min, my_l.index(_min), _max, my_l.index(_max)))

Here is a solution returning a dict {country: (maxval, index), (minval, index))}:
d = {
'Illinois': ['13', '12', '18', '23', '26', '25', '24', '19', '13', '10', '15', '14', '14', '4', '3'],
'Indiana': ['7', '6', '7', '8', '11', '11', '13', '12', '7', '7', '7', '7', '9', '2', '2']
}
maxmin = {}
for state, numbers in d.items():
maxmin[state] = (
max(enumerate(numbers), key=lambda x: int(x[1])),
min(enumerate(numbers), key=lambda x: int(x[1]))
)
print(maxmin)

Bit thrown together, but seems to do the job.
d = {"Illinois": ['13', '12', '18', '23', '26', '25', '24', '19', '13', '10', '15', '14', '14', '4', '3'],
"Indiana": ['7', '6', '7', '8', '11', '11', '13', '12', '7', '7', '7', '7', '9', '2', '2']}
if __name__ == "__main__":
print d
for state in d:
# returns the numbers with their index (#, index)
pairs = [(int(d[state][x]), x) for x in xrange(len(d[state]))]
minpair = min(pairs)
maxpair = max(pairs)
print "%s: %d in index %d and %d in index %d"%(state,maxpair[0],maxpair[1],
minpair[0],minpair[1])
Output:
{'Indiana': ['7', '6', '7', '8', '11', '11', '13', '12', '7', '7', '7', '7', '9', '2', '2'], 'Illinois': ['13', '12', '18', '23', '26', '25', '24', '19', '13', '10', '15', '14', '14', '4', '3']}
Indiana: 13 in index 6 and 2 in index 13
Illinois: 26 in index 4 and 3 in index 14
to get around the blank string, you could break up the list comprehension into
pairs = []
for x in xrange(len(d[state])):
try:
pairs.append( (int(d[state][x]), x) )
except ValueError:
pass # not a valid number

Categories

Resources