I am trying out multiprocessing for my Monty Hall game simulation for improved performance. The game is payed 10mm times and takes ~17 seconds when directly run, however, my multiprocessing implementation is taking significantly longer to run. I am clearly doing something wrong but I can't figure out what.
import multiprocessing
from MontyHall.game import Game
from MontyHall.player import Player
from Timer.timer import Timer
def doWork(input, output):
while True:
try:
f = input.get(timeout=1)
res = f()
output.put(res)
except:
break
def main():
# game setup
player_1 = Player(True) # always switch strategy
game_1 = Game(player_1)
input_queue = multiprocessing.Queue()
output_queue = multiprocessing.Queue()
# total simulations
for i in range(10000000):
input_queue.put(game_1.play_game)
with Timer('timer') as t:
# initialize 5 child processes
processes = []
for i in range(5):
p = multiprocessing.Process(target=doWork, args=(input_queue, output_queue))
processes.append(p)
p.start()
# terminate the processes
for p in processes:
p.join()
results = []
while len(results) != 10000000:
r = output_queue.get()
results.append(r)
win = results.count(True) / len(results)
loss = results.count(False) / len(results)
print(len(results))
print(win)
print(loss)
if __name__ == '__main__':
main()
This is my first post. Advice on posting etiquette is also appreciated. Thank you.
Code for the Classes:
class Player(object):
def __init__(self, switch_door=False):
self._switch_door = switch_door
#property
def switch_door(self):
return self._switch_door
#switch_door.setter
def switch_door(self, iswitch):
self._switch_door = iswitch
def choose_door(self):
return random.randint(0, 2)
class Game(object):
def __init__(self, player):
self.player = player
def non_prize_door(self, door_with_prize, player_choice):
"""Returns a door that doesn't contain the prize and that isn't the players original choice"""
x = 1
while x == door_with_prize or x == player_choice:
x = (x + 1) % 3 # assuming there are only 3 doors. Can be modified for more doors
return x
def switch_function(self, open_door, player_choice):
"""Returns the door that isn't the original player choice and isn't the opened door """
x = 1
while x == open_door or x == player_choice:
x = (x + 1) % 3 # assuming there are only 3 doors. Can be modified for more doors
return x
def play_game(self):
"""Game Logic"""
# randomly places the prize behind one of the three doors
door_with_prize = random.randint(0, 2)
# player chooses a door
player_choice = self.player.choose_door()
# host opens a door that doesn't contain the prize
open_door = self.non_prize_door(door_with_prize, player_choice)
# final player choice
if self.player.switch_door:
player_choice = self.switch_function(open_door, player_choice)
# Result
return player_choice == door_with_prize
Code for running it without multiprocessing:
from MontyHall.game import Game
from MontyHall.player import Player
from Timer.timer import Timer
def main():
# Setting up the game
player_2 = Player(True) # always switch
game_1 = Game(player_2)
# Testing out the hypothesis
with Timer('timer_1') as t:
results = []
for i in range(10000000):
results.append(game_1.play_game())
win = results.count(True) / len(results)
loss = results.count(False) / len(results)
print(
f'When switch strategy is {player_2.switch_door}, the win rate is {win:.2%} and the loss rate is {loss:.2%}')
if __name__ == '__main__':
main()
As you did not give the full code that we can run locally, I can only speculate. My guess is that you are passing an object(a method from your game) to other processes so pickling and unpickling took too much time. Unlike multithreading where you can "share" data, in multiprocessing, you need to pack the data and send to the other process.
However, there's a rule I always follow when I try to optimize my code - profile before optimizing! It would be much better to KNOW what's slow than GUESS.
It's a multiprocessing program so there are not a lot of options in the market. You could try viztracer which supports multiprocessing.
pip install viztracer
viztracer --log_multiprocess your_program.py
It will generate a result.html that you can open with chrome. Or you can just do
vizviewer result.html
I would suggest to reduce the iteration number so you can have a view of the whole picture(because viztracer uses a circular buffer and 10 million iterations will definitely overflow). But, you can still get the last piece of your code executing if you don't, which should be helpful enough for you to figure out what's going on.
I used viztracer as you gave the whole code.
This is one of your iteration in your worker process. As you can tell, the actual working part is very small(the yellow-ish slice in the middle p...). Most of the time has been spent on receiving and putting data, which eliminates the advantage of parallelization.
The correct way to do this is do it in batches. Also as this game does not actually require any data, you should just sent "I want to do it 1000 times" to the process, and let it do it, instead of sending the method one by one.
There's another interesting problem that you can easily find with viztracer:
This is the big picture of your worker process. Notice the large "nothing" in the end? Because your worker needs a timeout to finish, and that's when they are waiting. You should come up with a better idea to elegantly finish your worker process.
Updated my code. I fundamentally misunderstood the multiprocessing method.
def do_work(input, output):
"""Generic function that takes an input function and argument and runs it"""
while True:
try:
f, args = input.get(timeout=1)
results = f(*args)
output.put(results)
except:
output.put('Done')
break
def run_sim(game, num_sim):
"""Runs the game the given number of times"""
res = []
for i in range(num_sim):
res.append(game.play_game())
return res
def main():
input_queue = multiprocessing.Queue()
output_queue = multiprocessing.Queue()
g = Game(Player(False)) # set up game and player
num_sim = 2000000
for i in range(5):
input_queue.put((run_sim, (g, num_sim))) # run sim with game object and number of simulations passed into
# the queue
with Timer('Monty Hall Timer: ') as t:
processes = [] # list to save processes
for i in range(5):
p = multiprocessing.Process(target=do_work, args=(input_queue, output_queue))
processes.append(p)
p.start()
results = []
while True:
r = output_queue.get()
if r != 'Done':
results.append(r)
else:
break
# terminate processes
for p in processes:
p.terminate()
# combining the five returned list
flat_list = [item for sublist in results for item in sublist]
print(len(flat_list))
print(len(results))
Related
import time
import random
def timer():
correct = 1
x = 0
while correct != 2:
time.sleep(0.1)
x = x + 0.1
def round1():
numb = random.randint(1, 100)
print(numb)
timer()
ans = input(">")
if ans == numb:
correct = 2
x = round(x)
print("you did that in", x ,"seconds!")
round1()
I was trying to get both functions to run together (have the game playing and the timer going in the background) but as soon as the timer started it would let me continue the game.
In the given program, I have created two different functions that will work at the same time. I have used threading to create thread of functions and sleep to limit the printing speed. In similar manner you can use game and timer function together.
from threading import Thread
from time import sleep
#sleep is used in functions to delay the print
#the below functions has infinite loop that will run together
#defining functions
def func1():
for x in range(20):
sleep(1)
print("This is function 1.")
def func2():
for x in range(10):
sleep(2)
print("This is function 2.")
#creating thread
thread1=Thread(target=func1)
thread2=Thread(target=func2)
#running thread
thread1.start()
thread2.start()
I want to add a semaphore to where I commented down below the code but i couldnt.. so my code runs 4 threads that have linked list attached, and each linked list retrieves same items.. and they are sorted with a value from the items, then if there are no 2 biggest same numbers, winner is elected. if not nodework is called again for each thread with thr down below. so when there are 2 biggest same values and it rolls again, the linkedlist array from previous thread just mixes with the newly ones created, because the threads edit each other arrays as well. also the threads dont wait for each other to fill their arrays with data under with semaphoreAdding so they get wrong data set from the arrays as they keep executing. so i want to add a semaphore where with semaphoreAddingFinished is located in the code. so that, when each thread finishes .add() call, they can keep going. How can I add semaphore there ? I try everything but i get deadlock
import conset
import random
import threading
from threading import Timer
numberOfNodes = 4 #number of threads, (linked lists)
llList = []
roundCount = 1
roundCountCounter = 1
semaphore = threading.Semaphore(0)
semaphoreAdding = threading.Semaphore(1)
semaphoreAddingFinished = threading.Semaphore(1)
for x in range(0, numberOfNodes): #creating a global list with n number of ConSet instances (linked lists)
ll = conset.LinkedList()
llList.append(ll)
numb = 0 #thread counter
def nodeWork(nodeId, n):
global numb
randomInteger = random.randint(0, 3) #generate random number 0 to n^2
theItemTuple = (nodeId, randomInteger) #create the tuple for mailbox
print("Node", nodeId, "proposes value", randomInteger, "for round", roundCount)
semaphoreAddFinish = threading.Semaphore(0)
with semaphoreAdding:
for m in range(0, len(llList)): #add the tuple to all mailboxes of all nodes
llList[m].add(theItemTuple)
#print(nodeId, llList[nodeId].head.data)
with semaphoreAddingFinished: #i want to add a semaphore here
zzz =1
with semaphore:
for k in range(0, len(llList)):
node = llList[nodeId].head
if node.next:
if node.data[1] != node.next.data[1]:
print("Node", nodeId, "decide", node.data[0], "as the leader")
return
else:
print("Node", nodeId, "could not decide on the leader and moves to the round", roundCount+1)
llList[nodeId] = conset.LinkedList()
thr = threading.Thread(target=nodeWork, args=[nodeId, numberOfNodes])
thr.start()
numb = numb + 1
#print("numb", numb)
if numb == 4:
numb = 0
for l in range(0, numberOfNodes):
semaphoreAdding.release()
return
else:
print("Node", nodeId, "decide", node.data[0], "as the leader")
return
main_thread = threading.currentThread() # getting handle to the current thread
for num in range(0, numberOfNodes):
#threading.Lock.acquire()
t = threading.Thread(target=nodeWork, args=[num, numberOfNodes])
t.start()
if num == numberOfNodes-1:
semaphore.release()
semaphoreAdding.acquire()
#threading.Lock.release()
for t in threading.enumerate(): #waiting for all threads to end
if t is not main_thread:
t.join()
I'm a relatively new python coder and I am having problems attempting to print successful_trades/unsuccessful_trades after a 10 second simulation. The only thing I can think of is making the line of code saying 'print("Successful ender pearl trade")' an operator (which obviously won't work). Any help would be greatly appreciated, code is below. Thank you!
import multiprocessing
import time
import random
successful_trades = 0
unsuccessful_trade = 0
# Your foo function
def foo(n):
for i in range(10000 * n):
print ("Tick")
time.sleep(1)
if __name__ == '__main__':
# Start foo as a process
p = multiprocessing.Process(target=foo, name="Foo", args=(10,))
p.start()
# Wait 10 seconds for foo
time.sleep(10)
# Terminate foo
p.terminate()
# Cleanup
p.join()
user_input = input()
if user_input == "o":
while True:
import random
k = random.randint(1, 109)
number = random.randint(1, 109)
if str(number) == str(k):
print("Successful ender pearl trade")
(str(successful_trades) + str(1))
if str(number) != str(k):
print("Unsuccessful ender pearl trade")
(str(unsuccessful_trade) + str(1))
It works for me
One theory is your infinite while loop is locking up the screen output.
You can test this by changing your while true loop to
for i in range(100):
First off, I am brand new to the multiprocessing and threading world. I have two devices that generate data (gps and mca). the gps simulator is supposed to generate a location every 0.1 seconds. The mca is supposed to generate a random number every randomly generated time interval. When an event is registered by the mca, the count (cnt) is supposed to be sent to the count list.The same goes for the gps. The event handler is supposed to synchronize the count with the latest gps value is registered, and this should be printed to standard output. After 5 seconds, the mca should stop and send 'DONE' over a Queue to stop all of the other functions.I am also very new to Queue. It seems to me that my definitions start but don't do anything.
I would greatly appreciate it if someone could fix my code or let me know what is going wrong in it.
import random
from multiprocessing import Process, Queue
from time import sleep, time, clock
count = []
gps_data = []
def mca(q1):
print 'started'
t = 5
while True:
cnt = random.randint(0,30)
count.append(cnt)
dt = random.randint(0,3)
sleep(dt)
nt = t-dt
if nt <= 0:
break
q1.put('DONE')
def gps(q1):
print 'started2'
while q1.get() != 'DONE':
x = 0
dt = 0.1
sleep(dt)
y = x + 1
gps_data.append(y)
def event_handler(q1):
print 'started3'
size_i = len(count) #initial size of the mca list
while q1.get() != 'DONE':
size_f = len(count)
if size_f > size_i:
local_count = count[-1]
local_location = gps_data[-1]
data = local_count + local_location
print str(data)
size_i = size_f
else:
pass
if __name__ == '__main__':
q1 = Queue()
p_mca = Process(target = mca, args = (q1,))
p_gps = Process(target = gps, args = (q1,))
p_evh = Process(target = event_handler, args = (q1,))
p_evh.start()
p_gps.start()
p_mca.start()
p_evh.join()
p_gps.join()
p_mca.join()
Your variable t in mca() keeps getting set back to 5, thus
if nt <= 0:
is never True.
Like D_rez90 said,
if nt<=0:
is never true. You should change
nt=t-dt
to
t-=dt
if t<=0:
I am trying to make a text based game in which the user is a pilot in space. I want to create a movement system but am unsure how to do it. I want the user to be able to put in the desired grid coordinates, and his vehicle will begin to change its grid coords to get closer and closer to the ones he inputted.
Now, to do this I will probably need multithreading and a time element. But I am unsure how I can use a time element. Any advice is greatly appreciate, i'm just trying to learn here. Thanks guys!
from Gundam2 import Mobilesuits
#Main Variable/Object declarations:
Leo1=Mobilesuits(100,100,"Leo","leo desc","dockpit desc",100,[100,100,100])
Leo2=Mobilesuits(100,100,"Leo","leo desc","dockpit desc",100,[300,100,100])
Leo3=Mobilesuits(100,100,"Leo","leo desc","dockpit desc",100,[100,150,100])
currentmobilesuit=Leo1
#Main Function declarations
def commands(user_input,currentmobilesuit):
if user_input == "radar":
currentmobilesuit.radar()
elif user_input == "commands":
print("Command list:\nradar")
else:
print("Invalid command\nType 'commands' for a list of valid commands")
#Main execution
while True:
commands(raw_input(),currentmobilesuit)
class Mobilesuits:
#class global variables/methods here
instances = [] #grid cords here
def __init__(self,armor,speed,name,description,cockpit_description,\
radar_range, coordinates):
Mobilesuits.instances.append(self)
self.armor=armor
self.speed=speed
self.name=name
self.description=description
self.cockpit_description=cockpit_description
self.radar_range=radar_range
self.coordinates=coordinates
def can_detect(self, other):
for own_coord, other_coord in zip(self.coordinates, other.coordinates):
if abs(own_coord - other_coord) > self.radar_range:
return False
return True
def radar(self):
for other in Mobilesuits.instances:
if other is not self and self.can_detect(other):
print "%s detected at %s" % (other.description, other.coordinates)
Games typically have a "master loop" of some kind; yours does here:
#Main execution
while True:
commands(raw_input(),currentmobilesuit)
The simplest thing to do is to count in the loop:
#Main execution
turn_count = 0
while True:
commands(raw_input(),currentmobilesuit)
turn_count += 1
If you wanted the real time taken to have some impact on the counter, or be the counter, you can get the current time from the time module calling time.time().
#Main execution
import time
time_start = time.time()
time_elapsed = 0
while True:
commands(raw_input(),currentmobilesuit)
time_elapsed = time.time() - time_start
A couple other thoughts:
Make a Game class, and put the turn counter and game loop in that.
Have the commands function return a number that is the number of time units that took place during the command; for example, entering an invalid command might take 0 turns, while repairing a robot might take 5.
#Main execution
turn_count = 0
while True:
turns_taken = commands(raw_input(),currentmobilesuit)
turn_count += turns_taken
You can use non-blocking I/O. This will help you avoid the complications of threading. Here's your sample code implemented with a non-blocking read of stdin:
#!/usr/bin/python
import sys
import select
call_count = 0
#Main Function declarations
def commands(user_input):
global call_count
if len(user_input) > 0:
print('call count: ' + str(call_count) + ' user entered: ' + user_input)
def raw_input_no_block():
global call_count
call_count = call_count + 1
input_avail = select.select([sys.stdin], [], [], 0.1)[0] #wait for 0.1 seconds
if input_avail:
return sys.stdin.readline()
else:
return ''
#Main execution
while True:
commands(raw_input_no_block())