So I have a FCFS and SJF CPU simulator scheduling algorithm, however I'm struggling to implement shortest remaining time first algorithm.
This is what I have so far.
def srtf(submit_times, burst_times):
"""First Come First Serve Algorithm returns the time metrics"""
cpu_clock = 0
job = 0
response_times = []
turn_around_times = []
wait_times = []
total_jobs = []
remaining_burst_times = []
for stuff in range(len(submit_times)):
total_jobs.append(tuple((submit_times[stuff], burst_times[stuff])))
remaining_burst_times.append(burst_times[stuff])
while job < len(submit_times):
if cpu_clock < int(submit_times[job]):
cpu_clock = int(submit_times[job])
ready_queue = []
for the_job in total_jobs:
job_time = int(the_job[0])
if job_time <= cpu_clock:
ready_queue.append(the_job)
short_job = ready_queue_bubble(ready_queue)
submit, burst = short_job[0], short_job[1]
next_event = cpu_clock + int(burst)
response_time = cpu_clock - int(submit)
response_times.append(response_time)
remaining_burst_times[job] = next_event - cpu_clock
# cpu_clock = next_event
if remaining_burst_times[job] == 0:
turn_around_time = next_event - int(submit)
wait_time = turn_around_time - int(burst)
turn_around_times.append(turn_around_time)
wait_times.append(wait_time)
else:
pass
job += 1
total_jobs.remove(short_job)
remaining_burst_times.remove(short_job[1])
return response_times, turn_around_times, wait_times
Basically the function takes in a list of submit times and burst times and returns lists for the response, turn around and wait times. I have been trying to edit remnants from my short job first with a ready queue, to no avail.
Can anyone point me in the right direction?
It's not a very simple simulation due to preemption. Designing simulations is all about representing 1) the state of the world and 2) events that act on the world.
State of the world here is:
Processes. These have their own internal state.
Submit time (immutable)
Burst time (immutable)
Remaining time (mutable)
Completion time (mutable)
Wall clock time.
Next process to be submitted.
Running process.
Run start time (of the currently running process).
Waiting runnable processes (i.e. past submit with remaining > 0).
There are only two kinds of events.
A process's submit time occurs.
The running process completes.
When there are no more processes waiting to be submitted, and no process is running, the simulation is over. You can get the statistics you need from the processes.
The algorithm initializes the state then gets executes a standard event loop:
processes = list of Process built from parameters, sorted by submit time
wall_clock = 0
next_submit = 0 # index in list of processes
running = None # index of running process
run_start = None # start of current run
waiting = []
while True:
event = GetNextEvent()
if event is None:
break
wall_clock = event.time
if event.kind == 'submit':
# Update state for new process submission.
else: # event.kind is 'completion'
# Update state for running process completion.
An important detail is that if completion and submit events happen at the same time, process the completion first. The other way 'round makes update logic complicated; a running process with zero time remaining is a special case.
The "update state" methods adjust all the elements of the state according to the srtf algorithm. Roughly like this...
def UpdateStateForProcessCompletion():
# End the run of the running process
processes[running].remaining = 0
processes[running].completion_time = wall_clock
# Schedule a new one, if any are waiting.
running = PopShortestTimeRemainingProcess(waiting)
run_start_time = clock_time if running else None
A new submit is more complex.
def UpdateStateForProcessCompletion():
new_process = next_submit
next_submit += 1
new_time_remaining = processes[new_process].remaining
# Maybe preempt the running process.
if running:
# Get updated remaining time to run.
running_time_remaining = processes[running].remaining - (wall_clock - run_start)
# We only need to look at new and running processes.
# Waiting ones can't win because they already lost to the running one.
if new_time_remaining < running_time_remaining:
# Preempt.
processes[running].remaining = running_time_remaining
waiting.append(running)
running = new_process
run_start_time = wall_clock
else:
# New process waits. Nothing else changes
waiting.append(new_process)
else:
# Nothing's running. Run the newly submitted process.
running = new_process
run_start_time = wall_clock
The only thing left is getting the next event. You need only inspect processes[next_submit].submit and wall_clock + processes[running].remaining. Choose the smallest. The event has that time and the respective type. Of course you need to deal with the cases where next_submit and/or running are None.
I may not have everything perfect here, but it's pretty close.
Addition
Hope you're done with your homework by this time. This is fun to code up. I ran it on this example, and the trace matches well. Cheers
import heapq as pq
class Process(object):
"""A description of a process in the system."""
def __init__(self, id, submit, burst):
self.id = id
self.submit = submit
self.burst = burst
self.remaining = burst
self.completion = None
self.first_run = None
#property
def response(self):
return None if self.first_run is None else self.first_run - self.submit
#property
def turnaround(self):
return None if self.completion is None else self.completion - self.submit
#property
def wait(self):
return None if self.turnaround is None else self.turnaround - self.burst
def __repr__(self):
return f'P{self.id} # {self.submit} for {self.burst} ({-self.remaining or self.completion})'
def srtf(submits, bursts):
# Make a list of processes in submit time order.
processes = [Process(i + 1, submits[i], bursts[i]) for i in range(len(submits))]
processes_by_submit_asc = sorted(processes, key=lambda x: x.submit)
process_iter = iter(processes_by_submit_asc)
# The state of the simulation:
wall_clock = 0 # Wall clock time.
next_submit = next(process_iter, None) # Next process to be submitted.
running = None # Running process.
run_start = None # Time the running process started running.
waiting = [] # Heap of waiting processes. Pop gets min remaining.
def run(process):
"""Switch the running process to the given one, which may be None."""
nonlocal running, run_start
running = process
if running is None:
run_start = None
return
running.first_run = running.first_run or wall_clock
run_start = wall_clock
while next_submit or running:
print(f'Wall clock: {wall_clock}')
print(f'Running: {running} since {run_start}')
print(f'Waiting: {waiting}')
# Handle completion first, if there is one.
if running and (next_submit is None or run_start + running.remaining <= next_submit.submit):
print('Complete')
wall_clock = run_start + running.remaining
running.remaining = 0
running.completion = wall_clock
run(pq.heappop(waiting)[1] if waiting else None)
continue
# Handle a new submit, if there is one.
if next_submit and (running is None or next_submit.submit < run_start + running.remaining):
print(f'Submit: {next_submit}')
new_process = next_submit
next_submit = next(process_iter, None)
wall_clock = new_process.submit
new_time_remaining = new_process.remaining
if running:
# Maybe preempt the running process. Otherwise new process waits.
running_time_remaining = running.remaining - (wall_clock - run_start)
if new_time_remaining < running_time_remaining:
print('Preempt!')
running.remaining = running_time_remaining
pq.heappush(waiting, (running_time_remaining, running))
run(new_process)
else:
pq.heappush(waiting, (new_time_remaining, new_process))
else:
run(new_process)
for p in processes:
print(f'{p} {p.response} {p.turnaround} {p.wait}')
return ([p.response for p in processes],
[p.turnaround for p in processes],
[p.wait for p in processes])
submits = [6,3,4,1,2,5]
bursts = [1,3,6,5,2,1]
print(srtf(submits, bursts))
Related
Im trying to run the listener function forever, for example I want any time there is new information available in the stream it updates on the list automatically. Any idea in how to do it would be appreciated.
class Notifications(Screen):
notificationslist = ObjectProperty(None)
def listener(self, event = None):
notifications_screen = self.manager.get_screen('notif')
print(event.event_type) # can be 'put' or 'patch'
print(event.path) # relative to the reference, it seems
print(event.data) # new data at /reference/event.path. None if deleted
notifications = event.data
if notifications.items() == None:
return
else:
for key, value in notifications.items():
thevalue = value
notifications_screen.notificationslist.adapter.data.extend([value[0:17] + '\n' + value[18:]])
print(thevalue)
id = (thevalue[thevalue.index("(") + 1:thevalue.rindex(")")])
print(id)
If you want a function to run forever but that does not block you from doing other functions, then you can use threads.
Here is an example with example_of_process that runs forever, and then the main program with time.sleep(3)
import threading
import time
def example_of_process():
count = 0
while True:
print("count", count)
count += 1
time.sleep(1)
thread_1 = threading.Thread(target=example_of_process)
thread_1.daemon = True # without the daemon parameter, the function in parallel will continue even if your main program ends
thread_1.start()
# Now you can do anything else. I made a time sleep of 3s, otherwise the program ends instantly
time.sleep(3)
I'm trying to do a multiprocessing computation in python but from some reason i can't control the number of new created processes. after few seconds the IDE is getting crushed under hundreds of new processes.
here is the problematic code (start with expectiEvaluation at the bottom):
def serialExpecti(state, level):
allPieces = [L(), T(), O(), Z(), I(), J(), S()]
if level == 0:
return expectiHelper(state) / len(allPieces)
queueStates = util.PriorityQueueWithFunction(expectiHelper)
evaluation = 0;
for i in range(len(allPieces)):
succsecors = findAllPossiblePositions(state, allPieces[i])
for curState in succsecors:
queueStates.push(curState)
bestState = queueStates.pop()
evaluation += serialExpecti(bestState, level + 1)
queueStates = util.PriorityQueueWithFunction(expectiHelper) # clear queue
# print evaluation
return evaluation
def parallelExpecti(state, queue):
print os.getpid()
queue.put(serialExpecti(state, 0))
def expectiEvaluation(state):
allPieces = [L(), T(), O(), Z(), I(), J(), S()]
queue = multiprocessing.JoinableQueue()
queueStates = util.PriorityQueueWithFunction(expectiHelper)
for i in range(len(allPieces)):
succsecors = findAllPossiblePositions(state, allPieces[i])
for curState in succsecors:
queueStates.push(curState)
bestState = queueStates.pop()
p = multiprocessing.Process(target=parallelExpecti, args=(bestState, queue,))
p.start()
queue.join()
evaluation = 0;
while not queue.empty():
evaluation += queue.get()
return evaluation
the computation in serialExpecti() is very heavy so i wanted to make it parallel.
the function expectiEvaluation() is being called from the main thread many times during the run of the program, and i think that from some reason the queue.join() doesn't block the execution so that the main thread keeps ruining and calling expectiEvaluation() causing it to create more and more processes.
what could be the problem?
UPDATE:
I tried to get rid of the queue.join() by just keeping all the processes in a list and joining them at the end of the the function and it worked...sort of. now the processes are under control and not being created like crazy, but the programme is running very very slow. much much slower then the pure serial version. what am i doing wrong?
maybe it will help if i put here the serial version:
def expectiHelper(state):
return 10*holesEvaluation(state) + fullRowEvaluation(state) + heightEvaluation(state) + maxLengthEvaluation(state) + averageEvaluation(state)+ medianEvalutaion(state)
def expectiEvaluation(state, level=0):
global problem
allPieces=[L(), T(), O(), Z(), I(), J(), S()]
if level==3:
return expectiHelper(state)/len(allPieces)
queueStates = util.PriorityQueueWithFunction(expectiHelper)
evaluation=0;
for i in range(len(allPieces)):
succsecors = problem.findAllPossiblePositions(state,allPieces[i])
for curState in succsecors:
queueStates.push(curState)
bestState = queueStates.pop()
evaluation+= expectiEvaluation(bestState, level+1) #add best evaluation for this piece
queueStates = util.PriorityQueueWithFunction(expectiHelper) #clear queue
# print evaluation
return evaluation
as you can see it is a recursive function and every iteration in this loop is independent and very very heavy, so i it could be doen in parallel:
for i in range(len(allPieces)):
I am currently learning datastructures and algorithms.
I found this code on Interactive python
from pythonds.basic.queue import Queue
import random
class Printer:
def __init__(self, ppm):
self.pagerate = ppm
self.currentTask = None
self.timeRemaining = 0
def tick(self):
if self.currentTask != None:
self.timeRemaining = self.timeRemaining - 1
if self.timeRemaining <= 0:
self.currentTask = None
def busy(self):
if self.currentTask != None:
return True
else:
return False
def startNext(self,newtask):
self.currentTask = newtask
self.timeRemaining = newtask.getPages() * 60/self.pagerate
class Task:
def __init__(self,time):
self.timestamp = time
self.pages = random.randrange(1,21)
def getStamp(self):
return self.timestamp
def getPages(self):
return self.pages
def waitTime(self, currenttime):
return currenttime - self.timestamp
def simulation(numSeconds, pagesPerMinute):
labprinter = Printer(pagesPerMinute)
printQueue = Queue()
waitingtimes = []
for currentSecond in range(numSeconds):
if newPrintTask():
task = Task(currentSecond)
printQueue.enqueue(task)
if (not labprinter.busy()) and (not printQueue.isEmpty()):
nexttask = printQueue.dequeue()
waitingtimes.append( nexttask.waitTime(currentSecond))
labprinter.startNext(nexttask)
labprinter.tick()
averageWait=sum(waitingtimes)/len(waitingtimes)
print("Average Wait %6.2f secs %3d tasks remaining."%(averageWait,printQueue.size()))
def newPrintTask():
num = random.randrange(1,181)
if num == 180:
return True
else:
return False
for i in range(10):
simulation(3600,5)
Please can someone explain how does the waitingtimes.append( nexttask.waitTime(currentSecond)) compute waitime for the currentsecond.
Won't it be zero for that particular currentsecond.
Also as per the simulation, every 180 seconds there is a newtask , but it is enqueued and dequeued in the same currentsecond.
So the printqueue is always empty at any particular time or is it ?
Please Help...
Every second, there is a random chance a task is added to the queue. Only if the printer is available (not labprinter.busy() is true) is then a task taken from the queue to be sent to the printer.
Once a task is been added to the printer, it'll take that printer a a certain number of ticks ('seconds') to handle the random number of pages assigned to each task. No new task can then be sent to it! Each loop iteration labprinter.tick() is called, which decrements self.timeRemaining (calculated based on the task size and printer page rate). Only when that number reaches 0 is the task cleared and is the printer no longer busy (ready to take another task).
So the queue could be filling up while the printer is busy. Tasks that spend several rounds of the loop in the queue will have had a waiting time accumulate.
You could write down the ticks; lets say it can handle 20 pages per minute, so it'll take 3 seconds per page:
0. Nothing happens
1. A task of size 10 is created. The printer is free, so it'll take the task. 10 pages take 30 seconds.
2 - 5. No new tasks are created, the printer prints 1 page.
6 - 9. One new task is created at tick 8, added to the queue. The printer prints a 2nd page.
9 - 30. More tasks could be created, the printer prints the rest of the pages.
31. The printer is free, the task created at tick 8 can now be handled. That task waited 31 - 8 == 23 seconds.
I have a large list L to operate over. Let f() be the function which operates on L. f() takes another variable, which expires every 15 minutes and needs to be renewed. Here is an example, in serial:
def main():
L = openList()
# START THE CLOCK
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
a = getRenewed()
for item in L:
f(item, a) # operate on item given a
# CHECK TIME REMAINING
clockCur = dt.datetime.now()
clockRem = (clockExp - clockCur).total_seconds()
# RENEW a IF NEEDED
if clockRem < 5: # renew with 5 seconds left
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
a = getRenewed()
Since f() takes a few seconds (or longer sometimes), I would like to parallelize the code. Any tips for how to do that given the timer? I envision sharing clockExp and "a", and when a process satisfies clockRem < 5, it calls getRenewed() and shares the new "a" and clockExp, and repeat.
If getRenewed is idempotent (that is, you can call it multiple times without side effects), you can simply move your existing timer code to your worker processes, and let them each call it once when they notice their own timer has run down. This only requires synchronization for the items from the list that you pass in, and multiprocessing.Pool can handle that easily enough:
def setup_worker():
global clockExp, a
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
a = getRenewed()
def worker(item):
global clockExp, a
clockCur = dt.datetime.now()
clockRem = (clockExp - clockCur).total_seconds()
if clockRem < 5: # renew with 5 seconds left
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
a = getRenewed()
f(item, a)
def main(L):
pool = multiprocessing.Pool(initializer=setup_worker)
pool.map(worker, L)
If getRenewed is not idempotent, things will need to be a little more complicated. You won't be able to call it in each worker process, so you'll need to have a set up some kind of communication method between your processes so they can each get the latest version when it is available.
I'd suggest using a multiprocessing.queue to pass the a value from the main process to the workers. You can still use a Pool for the list items, you just need to make sure you use it asynchronously from the main process. Like this, perhaps:
def setup_worker2(queue):
global x
x = random.random()
global a_queue, a, clockExp
a_queue = queue
a = a_queue.get() # wait for the first `a` value
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
def worker2(item):
global a, clockExp
clockCur = dt.datetime.now()
clockRem = (clockExp - clockCur).total_seconds()
if clockRem < 60: # start checking for a new `a` value 60 seconds before its needed
try:
a = a_queue.get_nowait()
clockStart = dt.datetime.now()
clockExp = clockStart + dt.timedelta(seconds=900)
except queue.Empty:
pass
return f(item, a)
def main2(L):
queue = multiprocessing.Queue() # setup the queue for the a values
pool = multiprocessing.Pool(initializer=setup_worker2, initargs=(queue,))
result = pool.map_async(worker2, L) # send the items to the pool asynchronously
while True: # loop for sending a values through the queue
a = getRenewed() # get a new item
for _ in range(os.cpu_count()):
queue.put(a) # send one copy per worker process
try:
result.wait(900-5) # sleep for ~15 minutes, or until the result is ready
except multiprocessing.TimeoutError:
pass # if we got a timeout, keep looping!
else:
break # if not, we are done, so break out of the loop!
The workers still need to have to have some timing code, because otherwise you'd face a race condition where one worker might consume two of the a values sent down the queue in a single batch from the main process. That could happen if some of the calls to f are significantly slower than others (which is probably likely if they involve downloading things from the web).
I am running pool.map on big data array and i want to print report in console every minute.
Is it possible? As i understand, python is synchronous language, it can't do this like nodejs.
Perhaps it can be done by threading.. or how?
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
# I want to call this function every minute
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(make_job, data)
pool.close()
pool.join()
You can use a permanent threaded timer, like those from this question: Python threading.timer - repeat function every 'n' seconds
from threading import Timer,Event
class perpetualTimer(object):
# give it a cycle time (t) and a callback (hFunction)
def __init__(self,t,hFunction):
self.t=t
self.stop = Event()
self.hFunction = hFunction
self.thread = Timer(self.t,self.handle_function)
def handle_function(self):
self.hFunction()
self.thread = Timer(self.t,self.handle_function)
if not self.stop.is_set():
self.thread.start()
def start(self):
self.stop.clear()
self.thread.start()
def cancel(self):
self.stop.set()
self.thread.cancel()
Basically this is just a wrapper for a Timer object that creates a new Timer object every time your desired function is called. Don't expect millisecond accuracy (or even close) from this, but for your purposes it should be ideal.
Using this your example would become:
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
# set up the monitor to make run the function every minute
monitor = PerpetualTimer(60,display_status)
monitor.start()
results = pool.map(make_job, data)
pool.close()
pool.join()
monitor.cancel()
EDIT:
A cleaner solution may be (thanks to comments below):
from threading import Event,Thread
class RepeatTimer(Thread):
def __init__(self, t, callback, event):
Thread.__init__(self)
self.stop = event
self.wait_time = t
self.callback = callback
self.daemon = True
def run(self):
while not self.stop.wait(self.wait_time):
self.callback()
Then in your code:
def main():
data = [...]
pool = ThreadPool(45)
stop_flag = Event()
RepeatTimer(60,display_status,stop_flag).start()
results = pool.map(make_job, data)
pool.close()
pool.join()
stop_flag.set()
One way to do this, is to use main thread as the monitoring one. Something like below should work:
def main():
data = [...]
results = []
step = 0
pool = ThreadPool(16)
pool.map_async(make_job, data, callback=results.extend)
pool.close()
while True:
if results:
break
step += 1
sleep(1)
if step % 60 == 0:
print "status update" + ...
I've used .map() instead of .map_async() as the former is synchronous one. Also you probably will need to replace results.extend with something more efficient. And finally, due to GIL, speed improvement may be much smaller than expected.
BTW, it is little bit funny that you wrote that Python is synchronous in a question that asks about ThreadPool ;).
Consider using the time module. The time.time() function returns the current UNIX time.
For example, calling time.time() right now returns 1410384038.967499. One second later, it will return 1410384039.967499.
The way I would do this would be to use a while loop in the place of results = pool(...), and on every iteration to run a check like this:
last_time = time.time()
while (...):
new_time = time.time()
if new_time > last_time+60:
print "status update" + ...
last_time = new_time
(your computation here)
So that will check if (at least) a minute has elapsed since your last status update. It should print a status update approximately every sixty seconds.
Sorry that this is an incomplete answer, but I hope this helps or gives you some useful ideas.