Develop Reference

How do you share data between functions in threads with python

I Have a counting function that I would like to start and restart while getting the live variables to use in another function my problem is while using threading it seams like even global variables don't seem to work to pass variables around. What I want the code to do is have a counter be triggered as needed or maybe free running I'm not sure yet. To be able to reset the counter and get the value of the counter.
Right now the counter will start and run fine but the print_stuff function keeps telling me that there is no attribute countval.
The count thread gets started at startup but I don't necessarily want it to start up immediately, I would like to trigger it as needed but I cant put count_thread.start() twice or it will through a error so I'm calling the thread at startup and then calling the function again to restart it as needed. Maybe there is a more elegant way of doing that.?
import threading
import time
def count():
global countval
for countval in range(3):
print('looping')
time.sleep(1)
def print_stuff():
global countval
e = input("press enter to start")
count()
while True:
if countval == 3:
print("time out")
count_thread = threading.Thread(target=count)
print_thread = threading.Thread(target=print_stuff)
print_thread.start()
count_thread.start()

print_stuff is getting to the if statement before the count function is able to create the variable. Just do them in the opposite order. Either that, or create a global countval = 0 to start things off.

To solve the no attribute problem you can use Queue,
and if you want to stop your counting thread you can set a global variable or you can pass a function (using lambda or inner function or ...) to do that.
Here is one way to do that:
import threading
import time
from queue import Queue
from typing import Callable
def count(q, stop_counting):
# type: (Queue, Callable[[], bool]) -> None
for countval in range(3):
if stop_counting():
print('stopped')
break
print(f'looping {countval}')
q.put(countval)
time.sleep(1)
def print_stuff(q):
# type: (Queue) -> None
while True:
countval = q.get()
print(f'countval gotten: {countval}')
if countval == 3:
print("time out")
def main():
flag_stop_counting = False
q = Queue()
def stop_counting():
return flag_stop_counting
count_thread = threading.Thread(target=count, args=(q, stop_counting,))
print_thread = threading.Thread(target=print_stuff, args=(q,))
print_thread.start()
count_thread.start()
time.sleep(1.25)
flag_stop_counting = True
if __name__ == '__main__':
main()
In this code:
counter checks if it should stop counting or not
counter puts the value that it made to q
print_stuff get the value from q (Note: he waits until counter puts his value in q)
To check that program works:
after 1.25 seconds we change the value of flag_stop_counting
But if you want your counter to only have a for, i guess it's better to don't make it as a thread and run it whenever you want.
Hope it was helpful.

Python CPU Scheduler Simulator

So I have a FCFS and SJF CPU simulator scheduling algorithm, however I'm struggling to implement shortest remaining time first algorithm.
This is what I have so far.
def srtf(submit_times, burst_times):
"""First Come First Serve Algorithm returns the time metrics"""
cpu_clock = 0
job = 0
response_times = []
turn_around_times = []
wait_times = []
total_jobs = []
remaining_burst_times = []
for stuff in range(len(submit_times)):
total_jobs.append(tuple((submit_times[stuff], burst_times[stuff])))
remaining_burst_times.append(burst_times[stuff])
while job < len(submit_times):
if cpu_clock < int(submit_times[job]):
cpu_clock = int(submit_times[job])
ready_queue = []
for the_job in total_jobs:
job_time = int(the_job[0])
if job_time <= cpu_clock:
ready_queue.append(the_job)
short_job = ready_queue_bubble(ready_queue)
submit, burst = short_job[0], short_job[1]
next_event = cpu_clock + int(burst)
response_time = cpu_clock - int(submit)
response_times.append(response_time)
remaining_burst_times[job] = next_event - cpu_clock
# cpu_clock = next_event
if remaining_burst_times[job] == 0:
turn_around_time = next_event - int(submit)
wait_time = turn_around_time - int(burst)
turn_around_times.append(turn_around_time)
wait_times.append(wait_time)
else:
pass
job += 1
total_jobs.remove(short_job)
remaining_burst_times.remove(short_job[1])
return response_times, turn_around_times, wait_times
Basically the function takes in a list of submit times and burst times and returns lists for the response, turn around and wait times. I have been trying to edit remnants from my short job first with a ready queue, to no avail.
Can anyone point me in the right direction?

It's not a very simple simulation due to preemption. Designing simulations is all about representing 1) the state of the world and 2) events that act on the world.
State of the world here is:
Processes. These have their own internal state.
Submit time (immutable)
Burst time (immutable)
Remaining time (mutable)
Completion time (mutable)
Wall clock time.
Next process to be submitted.
Running process.
Run start time (of the currently running process).
Waiting runnable processes (i.e. past submit with remaining > 0).
There are only two kinds of events.
A process's submit time occurs.
The running process completes.
When there are no more processes waiting to be submitted, and no process is running, the simulation is over. You can get the statistics you need from the processes.
The algorithm initializes the state then gets executes a standard event loop:
processes = list of Process built from parameters, sorted by submit time
wall_clock = 0
next_submit = 0 # index in list of processes
running = None # index of running process
run_start = None # start of current run
waiting = []
while True:
event = GetNextEvent()
if event is None:
break
wall_clock = event.time
if event.kind == 'submit':
# Update state for new process submission.
else: # event.kind is 'completion'
# Update state for running process completion.
An important detail is that if completion and submit events happen at the same time, process the completion first. The other way 'round makes update logic complicated; a running process with zero time remaining is a special case.
The "update state" methods adjust all the elements of the state according to the srtf algorithm. Roughly like this...
def UpdateStateForProcessCompletion():
# End the run of the running process
processes[running].remaining = 0
processes[running].completion_time = wall_clock
# Schedule a new one, if any are waiting.
running = PopShortestTimeRemainingProcess(waiting)
run_start_time = clock_time if running else None
A new submit is more complex.
def UpdateStateForProcessCompletion():
new_process = next_submit
next_submit += 1
new_time_remaining = processes[new_process].remaining
# Maybe preempt the running process.
if running:
# Get updated remaining time to run.
running_time_remaining = processes[running].remaining - (wall_clock - run_start)
# We only need to look at new and running processes.
# Waiting ones can't win because they already lost to the running one.
if new_time_remaining < running_time_remaining:
# Preempt.
processes[running].remaining = running_time_remaining
waiting.append(running)
running = new_process
run_start_time = wall_clock
else:
# New process waits. Nothing else changes
waiting.append(new_process)
else:
# Nothing's running. Run the newly submitted process.
running = new_process
run_start_time = wall_clock
The only thing left is getting the next event. You need only inspect processes[next_submit].submit and wall_clock + processes[running].remaining. Choose the smallest. The event has that time and the respective type. Of course you need to deal with the cases where next_submit and/or running are None.
I may not have everything perfect here, but it's pretty close.
Addition
Hope you're done with your homework by this time. This is fun to code up. I ran it on this example, and the trace matches well. Cheers
import heapq as pq
class Process(object):
"""A description of a process in the system."""
def __init__(self, id, submit, burst):
self.id = id
self.submit = submit
self.burst = burst
self.remaining = burst
self.completion = None
self.first_run = None
#property
def response(self):
return None if self.first_run is None else self.first_run - self.submit
#property
def turnaround(self):
return None if self.completion is None else self.completion - self.submit
#property
def wait(self):
return None if self.turnaround is None else self.turnaround - self.burst
def __repr__(self):
return f'P{self.id} # {self.submit} for {self.burst} ({-self.remaining or self.completion})'
def srtf(submits, bursts):
# Make a list of processes in submit time order.
processes = [Process(i + 1, submits[i], bursts[i]) for i in range(len(submits))]
processes_by_submit_asc = sorted(processes, key=lambda x: x.submit)
process_iter = iter(processes_by_submit_asc)
# The state of the simulation:
wall_clock = 0 # Wall clock time.
next_submit = next(process_iter, None) # Next process to be submitted.
running = None # Running process.
run_start = None # Time the running process started running.
waiting = [] # Heap of waiting processes. Pop gets min remaining.
def run(process):
"""Switch the running process to the given one, which may be None."""
nonlocal running, run_start
running = process
if running is None:
run_start = None
return
running.first_run = running.first_run or wall_clock
run_start = wall_clock
while next_submit or running:
print(f'Wall clock: {wall_clock}')
print(f'Running: {running} since {run_start}')
print(f'Waiting: {waiting}')
# Handle completion first, if there is one.
if running and (next_submit is None or run_start + running.remaining <= next_submit.submit):
print('Complete')
wall_clock = run_start + running.remaining
running.remaining = 0
running.completion = wall_clock
run(pq.heappop(waiting)[1] if waiting else None)
continue
# Handle a new submit, if there is one.
if next_submit and (running is None or next_submit.submit < run_start + running.remaining):
print(f'Submit: {next_submit}')
new_process = next_submit
next_submit = next(process_iter, None)
wall_clock = new_process.submit
new_time_remaining = new_process.remaining
if running:
# Maybe preempt the running process. Otherwise new process waits.
running_time_remaining = running.remaining - (wall_clock - run_start)
if new_time_remaining < running_time_remaining:
print('Preempt!')
running.remaining = running_time_remaining
pq.heappush(waiting, (running_time_remaining, running))
run(new_process)
else:
pq.heappush(waiting, (new_time_remaining, new_process))
else:
run(new_process)
for p in processes:
print(f'{p} {p.response} {p.turnaround} {p.wait}')
return ([p.response for p in processes],
[p.turnaround for p in processes],
[p.wait for p in processes])
submits = [6,3,4,1,2,5]
bursts = [1,3,6,5,2,1]
print(srtf(submits, bursts))

Executing a while loop in main thread until we get the return value from a function called in secondary thread

I want to write code that does something like-
In a thread call a function that will return two links and in the main thread keep printing
"processing..." until that function, called in secondary thread returns those values
and when we get those return values the while loop of the main thread terminates and print
those
values.
Now I have tried writing few codes in python but couldn't manage to do it!
I have just started python programming so I'm not familiar with it.
BTW the above-mentioned scenario is just a prototype.
The real case looks something like that:-
def search_button(self): #main thread
mname = self.root.ids.name.text
quality = Quality
#print(mname)
#print(quality)
global link4, link5
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = executor.submit(movie_bot, mname, quality) #movie_bot is function
link4, link5 = futures.result() #returning two links
while(link4==None and link5==None):
self.root.ids.status.text = 'Searching, please wait...'
self.root.ids.status.text = ''
print(link4)
print(link5)
In case further details are required then please just let me know.
Thank you for the help.

Tried a lot of things and now finally it got resolved.
Basically what I wanted to do was take two inputs from GUI then call a function (from another python program) with those two parameters and then once the processing is completed then from GUI user and either watch or download that content by pressing the watch or download buttons respectively.
So to do that earlier I was returning the watch and download link from that thread called function and even on calling that function on another thread, as it was returnig values after execution so the GUI freezes and shows not responding
Then after trying a lot of thing I came across daemon thing so I just made that thread daemon and that solved the main problem of freezing but now I wasn't able to take return values (when I tried to take the return values it again started to freeze the GUI)
So then I found an alternative to access those links from the main thread.
Here the point is if the function doesn't return anything that is being called in the thread then just make it daemon thread_name.daemon() = True and now it won't freeze the GUI
Now in case you wanna do something exactly after the thread is finished then this can be used thread_name.is_alive()
MY CODE LOOKS SOMETHING LIKE THAT:-
from selenium_l_headless import movie_bot
from kivy.lang import Builder
from kivymd.app import MDApp
from kivy.properties import ObjectProperty
from selenium_l_headless import *
import threading
import time
from kivy.clock import Clock
class MovieBot(MDApp):
mname = ObjectProperty(None)
quality = ObjectProperty(None)
Quality = ObjectProperty(None)
link4 = ObjectProperty(None)
link5 = ObjectProperty(None)
checks = []
def build(self):
self.theme_cls.theme_style= "Dark"
self.theme_cls.primary_palette = "Teal"
return Builder.load_file('kivy_bot_md.kv')
def checkBox_click(self, instance, value, Q):
global Quality
if value==True:
MovieBot.checks.append(Q)
Quality=''
for x in MovieBot.checks:
Quality = f'{Quality}{x}'
else:
MovieBot.checks.remove(Q)
Quality = ''
for x in MovieBot.checks:
Quality = f'{Quality} {x}'
def complete(self):
self.root.ids.status.text = 'Searching completed, Now you can
download or watch the movie!'
global flag
flag=0
def status_sleep(self, *args):
try:
self.root.ids.status.text = 'Searching, please wait...'
if(t1.is_alive() is False):
self.complete()
except:
pass
def search_button(self):
try:
mname = self.root.ids.name.text
quality = Quality
global link4,link5
global t1, flag
flag=1
t1 = threading.Thread(target = movie_bot, args= [mname, quality])
t1.daemon = True
t1.start()
if(t1.is_alive()):
Clock.schedule_interval(self.status_sleep,1)
except:
pass
def watch(self):
try:
if(flag is 1):
pass
else:
self.root.ids.status.text = ''
t2 = threading.Thread(target=watch_now)
t2.daemon = True
t2.start()
except:
pass
def download(self):
try:
if(flag is 1):
pass
else:
self.root.ids.status.text = ''
t3 = threading.Thread(target=download_movie)
t3.daemon = True
t3.start()
except:
pass
def close(self):
exit(0)
MovieBot().run()

Is it possible to execute function every x seconds in python, when it is performing pool.map?

I am running pool.map on big data array and i want to print report in console every minute.
Is it possible? As i understand, python is synchronous language, it can't do this like nodejs.
Perhaps it can be done by threading.. or how?
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
# I want to call this function every minute
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(make_job, data)
pool.close()
pool.join()

You can use a permanent threaded timer, like those from this question: Python threading.timer - repeat function every 'n' seconds
from threading import Timer,Event
class perpetualTimer(object):
# give it a cycle time (t) and a callback (hFunction)
def __init__(self,t,hFunction):
self.t=t
self.stop = Event()
self.hFunction = hFunction
self.thread = Timer(self.t,self.handle_function)
def handle_function(self):
self.hFunction()
self.thread = Timer(self.t,self.handle_function)
if not self.stop.is_set():
self.thread.start()
def start(self):
self.stop.clear()
self.thread.start()
def cancel(self):
self.stop.set()
self.thread.cancel()
Basically this is just a wrapper for a Timer object that creates a new Timer object every time your desired function is called. Don't expect millisecond accuracy (or even close) from this, but for your purposes it should be ideal.
Using this your example would become:
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
# set up the monitor to make run the function every minute
monitor = PerpetualTimer(60,display_status)
monitor.start()
results = pool.map(make_job, data)
pool.close()
pool.join()
monitor.cancel()
EDIT:
A cleaner solution may be (thanks to comments below):
from threading import Event,Thread
class RepeatTimer(Thread):
def __init__(self, t, callback, event):
Thread.__init__(self)
self.stop = event
self.wait_time = t
self.callback = callback
self.daemon = True
def run(self):
while not self.stop.wait(self.wait_time):
self.callback()
Then in your code:
def main():
data = [...]
pool = ThreadPool(45)
stop_flag = Event()
RepeatTimer(60,display_status,stop_flag).start()
results = pool.map(make_job, data)
pool.close()
pool.join()
stop_flag.set()

One way to do this, is to use main thread as the monitoring one. Something like below should work:
def main():
data = [...]
results = []
step = 0
pool = ThreadPool(16)
pool.map_async(make_job, data, callback=results.extend)
pool.close()
while True:
if results:
break
step += 1
sleep(1)
if step % 60 == 0:
print "status update" + ...
I've used .map() instead of .map_async() as the former is synchronous one. Also you probably will need to replace results.extend with something more efficient. And finally, due to GIL, speed improvement may be much smaller than expected.
BTW, it is little bit funny that you wrote that Python is synchronous in a question that asks about ThreadPool ;).

Consider using the time module. The time.time() function returns the current UNIX time.
For example, calling time.time() right now returns 1410384038.967499. One second later, it will return 1410384039.967499.
The way I would do this would be to use a while loop in the place of results = pool(...), and on every iteration to run a check like this:
last_time = time.time()
while (...):
new_time = time.time()
if new_time > last_time+60:
print "status update" + ...
last_time = new_time
(your computation here)
So that will check if (at least) a minute has elapsed since your last status update. It should print a status update approximately every sixty seconds.
Sorry that this is an incomplete answer, but I hope this helps or gives you some useful ideas.

python add time to a countdown already running

I want to have an app where if I click a button I add X amount of time to my running countdown timer.
I'm guessing I have to use threads for this but am not sure how to implement it..
Here is the code I have so far:
def countdown_controller(add_time):
end_it = False
def timer(time_this):
start = time.time()
lastprinted = 0
finish = start + time_this
while time.time() < finish:
now = int(time.time())
if now != lastprinted:
time_left = int(finish - now)
print time_left
lastprinted = now
if end_it == True:
now = finish
time.sleep(0.1)
# Check if the counter is running otherwise just add time.
try:
time_left
except NameError:
timer(add_time)
else:
if time_left == 0:
timer(add_time)
else:
add_this = time_left
end_it = True
while now != finish:
time.sleep(0.1)
timer(add_time + add_this)
Obviously this will not work, because every time I call countdown_controller(15) fx, it will start counting down for 15 seconds and if I click my button nothing happens until the timer is ended.
Help would be greatly appreciated.

I would say that there is a flaw in the design of the code, because your screen output blocks down the entire program doing nothing (time.sleep(0.1)).
Typically what you want to to do in these cases is having a main loop in your program that cycles through the various operations that make your program run. This guarantees a sensible distribution of system resources between the various tasks.
In your specific case, what you would like to have in your main loop is:
Check user input (has extra time been added?)
Update output of the countdown
Example implementation:
import time
import curses
# The timer class
class Timer():
def __init__(self):
self.target = time.time() + 5
def add_five(self):
self.target += 5
def get_left(self):
return int(self.target-time.time())
# The main program
t = Timer()
stdscr = curses.initscr()
stdscr.nodelay(True)
curses.noecho()
# This is the main loop done in curses, but you can implement it with
# a GUI toolkit or any other method you wish.
while True:
left = t.get_left()
if left <= 0:
break
stdscr.addstr(0, 0, 'Seconds left: %s ' % str(left).zfill(3))
c = stdscr.getch()
if c == ord('x') :
t.add_five()
# Final operations start here
stdscr.keypad(0)
curses.echo()
curses.endwin()
print '\nTime is up!\n'
The above program will increase the counter of 5 seconds if you press the x key (lowercase). Most of the code is boilerplate to use the curses module, but of course if you use PyGTK, PySide or any other graphical toolkit, it will be different.
EDIT: As a rule of thumb, in python you want to avoid threading as much as you can, both because it often (but not always) slows down programs (see "Global Interpreter Lock") and because it makes software harder to debug/maintain.
HTH!

I would probably have a Timer object with a finish attribute that I could simply add an int to. Have that timer running in another thread that you can then query for the current time remaining from your GUI.
class Timer(object):
def __init__(self, length):
self.finish = time.time() + length
def get_time(self):
return time.time() >= self.finish