Develop Reference

Multiprocessing Freezing with Scipy Optimise

EDIT: The error does not lie with multiprocessing but rather with another library. I would close the question but #Lukasz Tracewski's point of using joblib may help someone else.
I've got an issue with a task I'm trying to parallelise on windows.
It seems to work for a while then suddenly halts on a particular instance (I can tell as I get the code to print which iteration it's on). I've noticed in the task manager that the CPU usage, which such be about 50%, is minimal for the Python processes. When I then do a keyboard interrupt on the cmd prompt, this suddenly shoots forward a number of instances and the activity goes back up for a short while, only to go back down again.
I've included the bits of my code which I think are relevant. I know it can work (I don't think it's stuck on a problem) and there seems to be a degree of randomness as to when it does freeze.
I'm using the COBYLA solver with max_iter set. I'm not sure if it is relevant, but when I tried to use BFGS, I got a freezing problem.
def get_optimal_angles(G,p,coeff,objective_function,initial_gamma_range,initial_beta_range,seed):
'''
This performs the classical-quantum interchange, improving the values of beta and gamma by reducing the value of
< beta, gamma | C | beta, gamma >. Returns the best angles found and the objective value this refers to. Bounds on the minimiser are set as the starting points
'''
initial_starting_points = random_intial_values((np.array(initial_gamma_range)),(np.array(initial_beta_range)),p,seed)
optimiser_function = minimize(objective_function, initial_starting_points, method='COBYLA', options={'maxiter':1500})
best_angles = optimiser_function.x
objective_value = optimiser_function.fun
return best_angles,objective_value
def find_best_angles_for_graph_6(graph_seed):
print("6:On graph",graph_seed)
#graph = gp.unweighted_erdos_graph(10,0.4,graph_seed)
graph = gp.unweighted_erdos_graph(6,0.4,graph_seed)
graph_coefficients = quantum_operator_z_coefficients(graph,'Yo')
exact_energy =get_exact_energy(graph)
angle_starting_seed = np.arange(1,number_of_angle_points,1)
objective_function= get_black_box_objective_sv(graph,p,graph_coefficients)
list_of_results = []
for angle_seed in angle_starting_seed:
print("On Angle Seed",angle_seed)
best_angles_for_seed, best_objective_value_for_seed = get_optimal_angles(graph,p,graph_coefficients,objective_function,[0,np.pi],[0,np.pi],angle_seed)
success_prob = calculate_energy_success_prob(graph,p,exact_energy, graph_coefficients,best_angles_for_seed,angle_seed)
angle_seed_data_list = [angle_seed,best_objective_value_for_seed,success_prob,best_angles_for_seed]
list_of_results.append(angle_seed_data_list)
list_of_best = get_best_results(list_of_results)
full_results = [graph_seed,exact_energy,list_of_best,list_of_results]
return full_results
import multiprocessing as mp
def main():
physical_cores = 5
pool = mp.Pool(physical_cores)
list_of_results_every_graph_6 = []
list_of_all_graph_results_6 = pool.map(find_best_angles_for_graph_6,range(1,number_of_graphs+1))
print(list_of_all_graph_results_6)
file_name_6 = 'unweighted_erdos_graph_N_6_p_8.pkl'
pickle_6 = open((file_name_6),'wb')
pickle.dump(list_of_all_graph_results_6, pickle_6)
pickle_6.close()
list_of_results_every_graph_10 = []
list_of_all_graph_results_10 = pool.map(find_best_angles_for_graph_10,range(1,number_of_graphs+1))
print(list_of_all_graph_results_10)
file_name_10 = 'unweighted_erdos_graph_N_9_p_8.pkl'
pickle_10 = open((file_name_10),'wb')
pickle.dump(list_of_all_graph_results_10, pickle_10)
pickle_10.close()
if __name__ == "__main__":
main()
EDIT: Here is the full code as a Jupyter notebook. https://www.dropbox.com/sh/6xb7setjsn1c1o3/AAANfH7mEmhuuf9cxh5QWsRQa?dl=0

Why does my Python script run slower than it should on my HeapSort implementation?

I've got as assignment to implement the heap sort algorithm into either Python or Java (or any other languages). Since I'm not that really "fluent" in Python or Java I decided to do both.
But here I ran into a problem, the running time of the program is way too much hight than it "should" be.
I mean by that, that the heap sort is supposed to run into a O(n * log n) and for current processor running on a clock rate of several GHz I didn't expect for that algorithm to run into over 2000secs for an array of size 320k
So for what I've done, I implemented the algorithm from the pseudo code of this sort in Python and in Java (I also tried the code in Julia from Rosetta Code to see if the running time was similar, why Julia ? Random pick)
So I checked the output for small input size problem, such as an array of size 10, 20 and 30. It appears that the array it correctly sorted in both languages/implementations.
Then I used the heapq library that implement this same algorithm to check once again if the running time was similar. It surprised me when it was actually the case... But after few tries I tried one last thing which is updating Python and then, the program using heapq ran much faster than the previous ones. Actually it was around 2k sec for the 320k array and now it around 1.5 sec or so.
I retried my algorithm and the problem was still there.
So here are the Heapsort class that I implemented:
class MaxHeap:
heap = []
def __init__(self, data=None):
if data is not None:
self.buildMaxHeap(data)
#classmethod
def toString(cls):
return str(cls.heap)
#classmethod
def add(cls, elem):
cls.heap.insert(len(cls.heap), elem)
cls.buildMaxHeap(cls.heap)
#classmethod
def remove(cls, elem):
try:
cls.heap.pop(cls.heap.index(elem))
except ValueError:
print("The value you tried to remove is not in the heap")
#classmethod
def maxHeapify(cls, heap, i):
left = 2 * i + 1
right = 2 * i + 2
largest = i
n = len(heap)
if left < n and heap[left] > heap[largest]:
largest = left
if right < n and heap[right] > heap[largest]:
largest = right
if largest != i:
heap[i], heap[largest] = heap[largest], heap[i]
cls.maxHeapify(heap, largest)
#classmethod
def buildMaxHeap(cls, heap):
for i in range(len(heap) // 2, -1, -1):
cls.maxHeapify(heap, i)
cls.heap = heap
#staticmethod
def heapSort(table):
heap = MaxHeap(table)
output = []
i = len(heap.heap) - 1
while i >= 0:
heap.heap[0], heap.heap[i] = heap.heap[i], heap.heap[0]
output = [heap.heap[i]] + output
heap.remove(heap.heap[i])
heap.maxHeapify(heap.heap, 0)
i -= 1
return output
To log the runtime for each array size (10000 - 320000) I use this loop in the main function :
i = 10000
while i <= 320000:
tab = [0] * i
j = 0
while j < i:
tab[j] = randint(0, i)
j += 1
start = time()
MaxHeap.heapSort(tab)
end = time()
pprint.pprint("Size of the array " + str(i))
pprint.pprint("Total execution time: " + str(end - start) + "s")
i *= 2
If you need the rest of the code to see where the error could be, don't hesitate I'll provide it. Just didn't want to share the whole file for no reasons.
As said earlier the running time I expected is from the worst case running time : O(n * log n)
with modern architecture and a processor of 2.6GHz I would expect something around 1sec or even less (since the running time is asked in nanosecond I suppose that even 1 sec is still too long)
Here are the results :
Python (own) : Java (Own)
Time Size Time Size
593ms. 10k 243ms. 10k
2344ms. 20k 600ms. 20k
9558ms. 40k 1647ms. 40k
38999ms. 80k 6666ms. 80k
233811ms. 160k 62789ms. 160k
1724926ms. 320k 473177ms. 320k
Python (heapq) Julia (Rosetta Code)
Time Size Time Size
6ms. 10k 21ms. 10k
14ms. 20k 21ms. 20k
15ms. 40k 23ms. 40k
34ms. 80k 28ms. 80k
79ms. 160k 39ms. 160k
168ms. 320k 60ms. 320k
And according to the formula the O(n * log n) give me :
40000 10k
86021 20k
184082 40k
392247 80k
832659 160k
1761648 320k
I think that these result could be used to determine how much time it should take depending on the machine (theoretically)
As you can see the high running time result comes from my algorithm, but I can't tell where in the code and that's why I'm asking here for help. (Runs slow both in Java and Python) (Didn't try to use heap sort in java lib is there is one to see the difference with my implementation, my bad)
Thanks a lot.
Edit : I forgot to add that I run this program on a MacBook Pro (last version MacOS, i7 2,6GHz. In case the problem could also comes from anything else than the code.
Edit 2 : Here are the modifications I did on the algorithm, following the answer I received. The program run approximately 200 times faster than previously, and so now it run in barely 2sec for the array of size 320k
class MaxHeap:
def __init__(self, data=None):
self.heap = []
self.size = 0
if data is not None:
self.size = len(data)
self.buildMaxHeap(data)
def toString(self):
return str(self.heap)
def add(self, elem):
self.heap.insert(self.size, elem)
self.size += 1
self.buildMaxHeap(self.heap)
def remove(self, elem):
try:
self.heap.pop(self.heap.index(elem))
except ValueError:
print("The value you tried to remove is not in the heap")
def maxHeapify(self, heap, i):
left = 2 * i + 1
right = 2 * i + 2
largest = i
if left < self.size and heap[left] > heap[largest]:
largest = left
if right < self.size and heap[right] > heap[largest]:
largest = right
if largest != i:
heap[i], heap[largest] = heap[largest], heap[i]
self.maxHeapify(heap, largest)
def buildMaxHeap(self, heap):
for i in range(self.size // 2, -1, -1):
self.maxHeapify(heap, i)
self.heap = heap
#staticmethod
def heapSort(table):
heap = MaxHeap(table)
i = len(heap.heap) - 1
while i >= 0:
heap.heap[0], heap.heap[i] = heap.heap[i], heap.heap[0]
heap.size -= 1
heap.maxHeapify(heap.heap, 0)
i -= 1
return heap.heap
And it runs using the same main as given before

Its interesting that you posted the clock speed of your computer- you COULD calculate the actual number of steps your algorithm requires... but you would need to know an awful lot about the implementation. For example, in python every time an object is created or goes out of scope, the interpreter updates counters on the underlying object, and frees the memory if those ref counts reach 0. Instead, you should look at the relative speed.
The third party examples you posted show the speed as less then doubling when the input array length doubles. That doesn't seem right, does it? Turns out that for those examples the initial work of building the array probably dominates the time spent sorting the array!
In your code, there is already one comment that calls out what I was going to say...
heap.remove(heap.heap[i])
This operation will go through your list (starting at index 0) looking for a value that matches, and then deletes it. This is already bad (if it works as intended, you are doing 320k comparisons on that line if your code worked as you expected!). But it gets worse- deleting an object from an array is not an in-place modification- every object after the deleted object has to be moved forward in the list. Finally, there is no guarantee that you are actually removing the last object there... duplicate values could exist!
Here is a useful website that lists the complexity of various operations in python - https://wiki.python.org/moin/TimeComplexity. In order to implement an algorithm as efficiently as possible, you need as many of your data structure operations to be O(1) as possible. Here is an example... here is some original code, presumably with heap.heap being a list...
output = [heap.heap[i]] + output
heap.remove(heap.heap[i])
doing
output.append(heap.heap.pop())
Would avoid allocating a new list AND use a constant time operation to mutate the old one. (much better to just use the output backwards than use the O(n) time insert(0) method! you could use a dequeue object for output to get appendleft method if you really need the order)
If you posted your whole code there are probably lots of other little things we could help with. Hopefully this helped!

RDSA implementation on sage

First of all I must say my knowledge with using Sage math is really very limited, but I really want to improve an to be able to solve these problems I am having. I have been asked to implement the following:
1 - Read in FIPS 186-4 (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf) the definition of ECDSA and implement using Sage math with:
(a) prime eliptic curves (P-xxx)
(b) binary eliptic curves (B-xxx)
I tried solving (a) by looking around the internet a lot and ended up with the following code:
Exercise 1, a)
class ECDSA_a:
def __init__(self):
#Parameters for Curve p-256 as stated on FIPS 186-4 D1.2.3
p256 = 115792089210356248762697446949407573530086143415290314195533631308867097853951
a256 = p256 - 3
b256 = ZZ("5ac635d8aa3a93e7b3ebbd55769886bc651d06b0cc53b0f63bce3c3e27d2604b", 16)
## base point values
gx = ZZ("6b17d1f2e12c4247f8bce6e563a440f277037d812deb33a0f4a13945d898c296", 16)
gy = ZZ("4fe342e2fe1a7f9b8ee7eb4a7c0f9e162bce33576b315ececbb6406837bf51f5", 16)
self.F = GF(p256)
self.C = EllipticCurve ([self.F(a256), self.F(b256)])
self.G = self.C(self.F(gx), self.F(gy))
self.N = FiniteField (self.C.order()) # how many points are in our curve
self.d = int(self.F.random_element()) # privateKey
self.pd = self.G*self.d # our pubkey
self.e = int(self.N.random_element()) # our message
#sign
def sign(self):
self.k = self.N.random_element()
self.r = (int(self.k)*self.G).xy()[0]
self.s = (1/self.k)*(self.e+self.N(self.r)*self.d)
#verify
def verify(self):
self.w = 1/self.N(self.s)
return self.r == (int(self.w*self.e)*self.G + int(self.N(self.r)*self.w)*self.pd).xy()[0]
#mutate
def mutate(self):
s2 = self.N(self.s)*self.N(-1)
if not (s2 != self.s) : return False
self.w = 1/s2
return self.r == (int(self.w*self.e)*self.G + int(self.N(self.r)*self.w)*self.pd).xy()[0] # sign flip mutant
#TESTING
#Exercise 1 a)
print("Exercise 1 a)\n")
print("Elliptic Curve defined by y^2 = x^3 -3x +b256*(mod p256)\n")
E = ECDSA_a()
E.sign()
print("Verify signature = {}".format(E.verify()))
print("Mutating = {}".format(E.mutate()))
But now I wonder, Is this code really what I have been asked for?
I mean, I got the values for p and all that from the link mentioned above.
But is this eliptic curve I made a prime one? (whatever that really means).
In order words is this code I glued together the answer? And what is the mutate function actually doing? I understand the rest but don't see why it needs to be here...
Also, what could I do about question (b)? I have looked all around the internet but I can't find a single understandable mention about binary eliptic curves in sage...
Could I just reuse the above code and simply change the curve creation to get the answer?

(a.) Is this code really what I have been asked for?
No.
The sign() method has the wrong signature: it does not accept an argument to sign.
It would be very helpful to write unit tests for your code based on published test vectors, perhaps these, cf this Secp256k1 ECDSA test examples question.
You might consider doing the verification methods described in D.5 & D.6 (pp 109 ff).
(b.) binary elliptic curves
The FIPS publication you cited offers some advice on implementing such curves, and yes you could leverage your current code. But there's probably less practical advantage to implementing them, compared to the P-xxx curves, as strength of B-xxx curves is on rockier footing. They have an advantage for hardware implementations such as FPGA, but that's not relevant for your situation.

SimPy Resource of 3 where each has different characteristics

I am trying to simulate a situation where we have 5 machines that occur in a 1 -> 3 -> 1 situation. i.e the 3 in the middle operate in parallel to reduce the effective time they take.
I can easily simulate this by create a SimPy resource with a value of three like this:
simpy.Resource(env, capacity=3)
However in my situation each of the three resources operates slightly differently and sometimes I want to be able to use any of them (when I'm operating) or book a specific one (when i want to clean). Basically the three machines slowly foul up at different rates and operate slower, I want to be able to simulate these and also enable a clean to occur when one gets too dirty.
I have tried a few ways of simulating this but have come up with problems and issues every time.
The first was when it booked the resource it also booked one of the 3 machines (A,B,C) globals flags and a flag itself to tell it which machine it was using. This works but it's not clean and makes it really difficult to understand what is occurring with huge if statements everywhere.
The second was to model it as three separate resources and then try to wait and request one of the 3 machines with something like:
reqA = A.res.request()
reqB = B.res.request()
reqC = C.res.request()
unitnumber = yield reqA | reqB | reqC
yield env.process(batch_op(env, name, machineA, machineB, machineC, unitnumber))
But this doesn't work and I can't work out the best way to look at yielding one of a choice.
What would be the best way to simulate this scenario. For completeness here is what im looking for:
Request any of 3 machines
Request a specific machine
Have each machine track it's history
Have each machines characteristics be different. i.e on fouls up faster but works faster initially
Detect and schedule a clean based on the performance or indicator
This is what I have so far on my latest version of trying to model each as seperate resources
class Machine(object):
def __init__(self, env, cycletime, cleantime, k1foul, k2foul):
self.env = env
self.res = simpy.Resource(env, 1)
self.cycletime = cycletime
self.cleantime = cleantime
self.k1foul = k1foul
self.k2foul = k2foul
self.batchessinceclean = 0
def operate(self):
self.cycletime = self.cycletime + self.k2foul * np.log(self.k1foul * self.batchessinceclean + 1)
self.batchessinceclean += 1
yield self.env.timeout(self.cycletime)
def clean(self):
print('%s begin cleaning at %s' % (self.env.now))
self.batchessinceclean = 0
yield env.timeout(self.cleantime)
print('%s finished cleaning at %s' % (self.env.now))

You should try (Filter)Store:
import simpy
def user(machine):
m = yield machine.get()
print(m)
yield machine.put(m)
m = yield machine.get(lambda m: m['id'] == 1)
print(m)
yield machine.put(m)
m = yield machine.get(lambda m: m['health'] > 98)
print(m)
yield machine.put(m)
env = simpy.Environment()
machine = simpy.FilterStore(env, 3)
machine.put({'id': 0, 'health': 100})
machine.put({'id': 1, 'health': 95})
machine.put({'id': 2, 'health': 97.2})
env.process(user(machine))
env.run()

Operations in Python

My question is fairly basic yet might need a challenging solution.
Essentially, I have an arbitrary function which we will call some_function.
def some_function(n):
for i in range(n):
i+i
r = 1
r = r+1
And I want to count the number of operations took place in an arbitrary call to this function is executed (e.g. some_function(5). there are 7 operations that took place).
How would one count the number of operations that took place within a function call? I cannot modify some_function.

I think you're really after what others already told you - the big O notation.
But if you really want to know the actual number of instructions executed you can use this on linux:
perf stat -e instructions:u python yourscript.py
Which will output:
Performance counter stats for 'python yourscript.py':
22,260,577 instructions:u
0.014450363 seconds time elapsed
Note though that it includes all the instructions for executing python itself. So you'd have to find your own reference.

Using byteplay:
Example:
#!/usr/bin/env python
from byteplay import Code
def some_function(n):
for i in range(n):
i + i
r = 1
r = r + 1
def no_of_bytecode_instructions(f):
code = Code.from_code(f.func_code)
return len(code.code)
print(no_of_bytecode_instructions(some_function))
Output:
$ python -i foo.py
28
>>>
NB:
This still gives you no idea how complex f is here.
"Number of Instructions" != "Algorithm Complexity" (not by itself)
See: Big O
Algorithm complexity is a measure of the no. of instructions executed
relative to the size of your input data set(s).
Some naive examples of "complexity" and Big O:
def func1(just_a_list):
"""O(n)"""
for i in just_a_list:
...
def func2(list_of_lists):
"""O(n^2)"""
for i in list_of_lsits:
for j in i:
...
def func3(a_dict, a_key):
"""O(1)"""
return a_dict[a_key]