Is it possible to set up a loop with live variable changes? I'm using threading, and the variables can change very often in between lines.
I'm looking for something like this:
length = len(some_list)
while length == len(some_list):
if check_something(some_list):
# The variable could change right here for
# example, and the next line would still be called.
do_something(some_list)
So far I've had no luck, is this something that's possible in python?
EDIT: More what I'm looking for is something so that the loop restarts if some_list changes.
If its just a single changing list, you can make a local copy.
def my_worker():
my_list = some_list[:]
if check_something(my_list):
do_something(my_list)
UPDATE
A queue may work for you. The thing that modifies needs to post to the queue, so its not an automatic thing. There is also the risk that the background thread falls behind and processes old stuff or ends up crashing everything if memory is exhausted by the queue.
import threading
import queue
import time
def worker(work_q):
while True:
some_list = work_q.get()
if some_list is None:
print('exiting')
return
print(some_list)
work_q = queue.Queue()
work_thread = threading.Thread(target=worker, args=(work_q,))
work_thread.start()
for i in range(10):
some_list.append(i)
work_q.put(some_list[:])
time.sleep(.2)
work_q.put(None)
work_thread.join()
Related
i wrote a simple script. something like this:
import multiprocessing
my_list = []
def run_a_process():
proc = multiprocessing.Process(target=worker)
proc.start()
my_list.append(proc)
def worker():
# do something here
...
def close_done_processes():
global my_list
for idx, proc in enumerate(my_list):
if not proc.is_alive():
del my_list[idx]
def main():
while True:
if len(my_list) <= 10:
for _ in range(10 - len(my_list)):
run_a_process()
if len(my_list):
close_done_processes()
main()
i dont know about this example but the real program works just fine and no problem.
but after some days it will freeze without any error or anything. the program is running the interpreter is working on it but there will be no more logs and no more functionality. even ctrl+c wont stop the program. i think the problem is with the del my_list[id] part. i think its not removing the index from memory and not garbage collect it. so it will pile up and cause freeze of memory limitation??
i want to know how can i solve this issue??
i want to add items to the list and remove the ones that are already done processing from memory while keeping the other unprocessed items in the list without getting this freeze thing.
You've got a few problems here:
As written, on Windows this code should fill your machine with a nigh infinite number of processes in seconds (at least according to the documentation; you may be avoiding this by luck). You need the if __name__ == '__main__': guard around your invocation of main to prevent it:
if __name__ == '__main__':
main()
Your code for cleaning up the list is broken (mutating a collection while you iterate over it is a bad idea), and will delete the wrong elements of the list when there are two or more elements to remove (so some might never get deleted if there's always an element before them that's deleted first)
You're not actually joining the dead processes, which delays cleanup of Process resources for a potentially significant (indefinite?) period of time.
To fix #2 and #3, the easiest solution is to just build a new list of alive processes to replace the existing list, and join the ones that aren't alive:
def close_done_processes():
global my_list
new_list = []
for proc in my_list:
if proc.is_alive():
new_list.append(proc)
else:
proc.join()
my_list = new_list
This might have asked already here, but I couldn't come up with right keywords to search.
I have a array that I would like to split them into chunks, and hand them to threads to do some work on each slice and dump the result.
However, I need to reassemble the results from each thread in order.
I tried passing the lock for each thread to lock and dump the result into another array, but the order is not correct. I assume because each thread completes in different time.
What would be the best way to do this in Python 3?
import threading
import numpy as np
from queue import Queue
def add(lock, work):
value = 0
for v in work:
#Do some work!
lock.acquire()
result.append(value)
lock.release()
a = np. arange(0,100)
result = []
lock = threading.Lock()
q = Queue()
for i in range(0,a.shape[0],10):
work = a[i:i+10]
t = threading.Thread(target=add, args=(lock,work))
t.start()
q.put(t)
while q.empty() == False:
q.get().join()
value = 0
for v in result:
#Assemble
print(value)
You're getting your results in a mixed up order because append puts each result at the end of the list when it comes in, which may not be in the same order the threads were started. A better approach might be to pass each worker an index into a properly-sized list, and let it assign its results there whenever it finishes. Lists are sufficiently thread-safe that you shouldn't need a lock for this (your Queue is also completely unnecessary since only the main thread interacts with it).
def add(work, result_index):
value = 0
for v in work:
#Do some work!
result[result_index] = value
a = np.arange(0,100)
results = []
threads = []
for i in range(0,a.shape[0],10):
work = a[i:i+10]
results.append(None) # enlarge the results list, so we have room for this thread's result
t = threading.Thread(target=add, args=(work, i//10))
t.start()
threads.append(t)
for t in threads:
t.join()
I would warn you that if your #Do some work! code is CPU limited, you're unlikely to get much benefit from using multiple threads. The CPython interpreter has a Global Interpreter Lock that prevents more than one thread from running Python code at the same time (so that interpreter state like reference counts on objects can remain consistent without each one needing its own lock). Threading is only really useful for IO-limited jobs (like fetching lots of documents from the internet.
For CPU limited work, you usually want to use multiprocessing instead. If that's what you need, look at multiprocessing.map, which can handle passing objects between processes and reassembling the results into an ordered list automatically.
So I have been figuring how I should make a thread-safe, the reason for it was that whenever I ran the program that I created just for fun. I realized the console got so much spammed that it doesn't happen to be fast enough to print it one by one.
Basically what I did is that I use a list of list that is no special than just a list of different fruits lets say
list = ['apple','banana','kiwi'....]
and then I have something called data that basically prints out using logger.
logger.log(data)
The full program would look like something like
def sendData(list, data):
logger.log(data)
def main():
...
...
...
data_list.append((list[i], data))
for index, data in data_list:
threading.Thread(target=sendData, args=(list, data)).start()
So basically as we can see this would probably be a lot of threads running at the same time which would cause a interact that would make the console to print out alot of mistake so now the question is:
How can I turn this into a sort of thread-safe? Would sort of sleep for each thread start be the magic?
You might want to look into threading.Lock(), it can be used to prevent multiples threads from doing output tasks at the same time and thus mixing the words in the console :
def sendData(list, data):
with lock:
logger.log(data)
lock = threading.Lock()
def main():
...
...
...
data_list.append((list[i], data))
for index, data in data_list:
threading.Thread(target=sendData, args=(list, data)).start()
This will prevent multiple threads from running the code in the "with" at the same time.
When a thread X enter in the "with" block, it will claim the lock. If another thread try to claim it (enter the "with" block), it will have to wait until the lock is released by the thread X.
I have a situation, where at some point in my code I want to trigger a number of timers, the code will keep running, but at some point these functions will trigger and remove an item from a given list. Similar though not exactly like the code below. The problem is, I want these functions to wait a certain amount of time, the only way I know how is to use sleep, but that stops all of the code, when I need the first function to keep running. So how can I set a function aside with out making everything wait for it? If the answer involves threading, please know that I have very little experience with it and like explanations with pictures and small words.
from time import sleep
from datetime import datetime
def func():
x = 1
for i in range(20):
if i % 4 == 0:
func2()
print("START", datetime.now())
x += 1
else:
print("continue")
def func2():
print("go")
sleep(10)
print("func 2--------------------------------------", datetime.now())
func()
You need to use threading. http://docs.python.org/2/library/threading.html
You can start functions in their own threads.
I used background function. It will run in the background, even if going to another page.
You need to import threading, also time to use time.sleep():
import threading
import time
I had a function where I wanted to sleep code in the background, here is an example:
# This is the one that will sleep, but since you used args on the Thread, it will not make the mainFunction to sleep.
def backgroundFunction(obj):
theObj = obj
time.sleep(120)
# updates the Food to 5 in 2 minutes
obj["Food"] = 5
return
def mainFunction():
obj = {"Food": 4, "Water": 3}
# Make sure there are a comma in the args().
t1 = threading.Thread(target=backgroundFunction, args=(obj,))
t1.start()
return
If you used t1 = threading.Thread(target=backgroundFunction(obj)) it will not be in the background so don't use this, unless you want mainFunction to sleep also.
Depending on the situation, another option might be an event queue based system. That avoids threads, so it can be simpler.
The idea is that instead of using sleep(20), you calculate when the event should fire, using datetime.now() + timedelta(seconds=20). You then put that in a sorted list.
Regularly, perhaps each time through the main loop of your program, you check the first element in the list; if the time has passed, you remove it and call the relevant function.
To add an event:
pending_events.append((datetime.now() + timedelta(seconds=20), e))
pending_events.sort()
Then, as part of your main loop:
for ... # your main loop
# handle timed events:
while pending_events[0][0] < datetime.now():
the_time, e = pending_events.pop(0)
handle_event(e, the_time)
... # rest of your main loop
This relies on your main loop regularly calling the event-handling code, and on the event-handling code not taking much time to handle the event. Depending on what the main loop and the events are doing, this may come naturally or it may be some effort or it may rule out this method...
Notes:
You only need to check the first element in the list, because the list is sorted in time order; checking the first element checks the earliest one and you don't need to check the others until that one has passed.
Instead of a sorted list, you can use a heapq, which is more complicated but faster; in practice, you'd need a lot of pending events to notice any difference.
If the event is to be "every 20s" rather than "after 20s", use the_time + timedelta(seconds=20) to schedule each subsequent event; that way, the delay in getting to and processing the event won't be added.
I was wondering if it would be possible to start a new thread and update its argument when this argument gets a new value in the main of the program, so something like this:
i = 0
def foo(i):
print i
time.sleep(5)
thread.start_new_thread(foo,(i,))
while True:
i = i+1
Thanks a lot for any help!
An argument is just a value, like anything else. Passing the value just makes a new reference to the same value, and if you mutate that value, every reference will see it.
The fact that both the global variable and the function parameter have the same name isn't relevant here, and is a little confusing, so I'm going to rename one of them. Also, your foo function only does that print once (possibly before you even increment the value), then sleeps for 5 seconds, then finishes. You probably wanted a loop there; otherwise, you can't actually tell whether things are working or not.
So, here's an example:
i = []
def foo(j):
while True:
print j
time.sleep(5)
thread.start_new_thread(foo,(i,))
while True:
i.append(1)
So, why doesn't your code work? Well, i = i+1 isn't mutating the value 0, it's assigning a new value, 0 + 1, to i. The foo function still has a reference to the old value, 0, which is unchanged.
Since integers are immutable, you can't directly solve this problem. But you can indirectly solve it very easily: replace the integer with some kind of wrapper that is mutable.
For example, you can write an IntegerHolder class with set and get methods; when you i.set(i.get() + 1), and the other reference does i.get(), it will see the new value.
Or you can just use a list as a holder. Lists are mutable, and hold zero or more elements. When you do i[0] = i[0] + 1, that replaces i[0] with a new integer value, but i is still the same list value, and that's what the other reference is pointing at. So:
i = [0]
def foo(j):
print j[0]
time.sleep(5)
thread.start_new_thread(foo,(i,))
while True:
i[0] = i[0]+1
This may seem a little hacky, but it's actually a pretty common Python idiom.
Meanwhile, the fact that foo is running in another thread creates another problem.
In theory, threads run simultaneously, and there's no ordering of any data accesses between them. Your main thread could be running on core 0, and working on a copy of i that's in core 0's cache, while your foo thread is running on core 1, and working on a different copy of i that's in core 1's cache, and there is nothing in your code to force the caches to get synchronized.
In practice, you will often get away with this, especially in CPython. But to actually know when you can get away with it, you have to learn how the Global Interpreter Lock works, and how the interpreter handles variables, and (in some cases) even how your platform's cache coherency and your C implementation's memory model and so on work. So, you shouldn't rely on it. The right thing to do is to use some kind of synchronization mechanism to guard access to i.
As a side note, you should also almost never use thread instead of threading, so I'm going to switch that as well.
i = []
lock = threading.Lock()
def foo(j):
while True:
with lock:
print j[0]
time.sleep(5)
t = threading.Thread(target=foo, args=(i,))
t.start()
while True:
with lock:
i[0] = i[0]+1
One last thing: If you create a thread, you need to join it later, or you can't quit cleanly. But your foo thread never exits, so if you try to join it, you'll just block forever.
For simple cases like this, there's a simple solution. Before calling t.start(), do t.daemon = True. This means when your main thread quits, the background thread will be automatically killed at some arbitrary point. That's obviously a bad thing if it's, say, writing to a file or a database. But in your case, it's not doing anything persistent or dangerous.
For more realistic cases, you generally want to create some way to signal between the two threads. Often you've already got something for the thread to wait on—a Queue, a file object or collection of them (via select), etc. If not, just create a flag variable protected by a lock (or condition or whatever is appropriate).
Try globals.
i = 0
def foo():
print i
time.sleep(5)
thread.start_new_thread(foo,())
while True:
i = i+1
You could also pass a hash holding the variables you need.
args = {'i' : 0}
def foo(args):
print args['i']
time.sleep(5)
thread.start_new_thread(foo,(args,))
while True:
args['i'] = arg['i'] + 1
You might also want to use a thread lock.
import thread
lock = thread.allocate_lock()
args = {'i' : 0}
def foo(args):
with lock:
print args['i']
time.sleep(5)
thread.start_new_thread(foo,(args,))
while True:
with lock:
args['i'] = arg['i'] + 1
Hoped this helped.