So I have a pyhthon application which im attempting to emulate a queue line up system. It import a library and calls to that library, specificaly psycopg2. An example code is as below
import threading,queue,psycopg2
q = queue.Queue()
def workerChecker():
while True:#Keeps the thread always checking for new things in queue
item = q.get()
addItemToDb(item) <------This part
q.task_done()
threading.Thread(target=workerChecker, daemon=True).start()
def addItemToDb(item):
#Do something and use psycopg2 to insert item to db here
So I can't seem to find a clear answer online on which thread will the codes in addItemToDb run in.
More specifically, will all the codes in the function addItemToDb be restricted to execution within the same thread as workerChecker ? Given that it uses an imported library.
Any assistance or help will be really appreciated...
The code inside addItemToDb will be restricted to the thread that runs workerChecker as long as it is called only by that function, but it can be used anywhere outside that thread in the main thread or any other thread that you create.
If you want to make sure that only workerChecker has access to that function you could define addItemToDb inside workerChecker
def workerChecker():
def addItemToDb(item):
# addItemToDb definition
pass
while True:
item = q.get()
addItemToDb(item)
q.task_done()
Related
I have two python programs which one of them connects to a bluetooth device(socket package), it receives and saves data from device, and another one read the stored data and draw a real time plot. I should make one application from these two programs.
I tried to mix these two python programs, but since bluetooth should wait to receive data (through a while loop), the other parts of program does not work. I tried to solve this problem using Clock.schedule_interval, but the program will hang after a period of time. So I decided to run these two programs simultaneously. I read, we can run some python programs at a same time using a python script. Is there any trick to join these two programs and build one application?
Any help would be greatly appreciated.
Install threaded:
pip install threaded
Create a new python file:
from threading import Thread
def runFile1(): import file1
def runFile2(): import file2
Thread(target=runFile1).start()
runFile2()
Run the new python file.
It can be done with threading. To do communication between the threaded function and your main function, use objects such as queue.Queue and threading.Event.
the bluetooth functions can be placed into a function that is the target of the thread
import time
from threading import Thread
from queue import Queue
class BlueToothFunctions(Thread):
def __init__(self, my_queue):
super().__init__()
self.my_queue = my_queue
# optional: causes this thread to end immediately if the main program is terminated
self.daemon = True
def run(self) -> None:
while True:
# do all the bluetooth stuff foreverer
g = self.my_queue.get()
if g == (None, None):
break
print(g)
time.sleep(1.0)
print("bluetooth closed")
if __name__ == '__main__':
_queue = Queue() # just one way to communicate to a thread
# pass an object reference so both main and thread have a way to communicate on this common queue
my_bluetooth = BlueToothFunctions(_queue)
my_bluetooth.start() # creates the thread and executes run() method
for i in range(5):
# communicate to the threaded functions
_queue.put(i)
_queue.put((None, None)) # optional, a way to cause the thread to end
my_bluetooth.join(timeout=5.0) # optional, pause here until thread ends
print('program complete')
My basic question is: how do I detect whether the current thread is a dummy thread? I am new to threading and I recently was debugging some code in my Apache2/Flask app and thought it might be useful. I was getting a flip flopping error where a request was processed successfully on the main thread, unsuccessfully on a dummy thread and then successfully on the main thread again, etc.
Like I said I am using Apache2 and Flask which seems the combination of which creates these dummy threads. I would also be interested in knowing more about that if anyone can teach me.
My code is meant to print information about the threads running on the service and looks something like this:
def allthr_info(self):
"""Returns info in JSON form of all threads."""
all_thread_infos = Queue()
for thread_x in threading.enumerate():
if thread_x is threading.current_thread() or thread_x is threading.main_thread():
continue
info = self._thr_info(thread_x)
all_thread_infos.put(info)
return list(all_thread_infos.queue)
def _thr_info(self, thr):
"""Consolidation of the thread info that can be obtained from threading module."""
thread_info = {}
try:
thread_info = {
'name': thr.getName(),
'ident': thr.ident,
'daemon': thr.daemon,
'is_alive': thr.is_alive(),
}
except Exception as e:
LOGGER.error(e)
return thread_info
You can check if the current thread is an instance of threading._DummyThread.
isinstance(threading.current_thread(), threading._DummyThread)
threading.py itself can teach you what dummy-threads are about:
Dummy thread class to represent threads not started here.
These aren't garbage collected when they die, nor can they be waited for.
If they invoke anything in threading.py that calls current_thread(), they
leave an entry in the _active dict forever after.
Their purpose is to return something from current_thread().
They are marked as daemon threads so we won't wait for them
when we exit (conform previous semantics).
def current_thread():
"""Return the current Thread object, corresponding to the caller's thread of control.
If the caller's thread of control was not created through the threading
module, a dummy thread object with limited functionality is returned.
"""
try:
return _active[get_ident()]
except KeyError:
return _DummyThread()
I want to create multi process app. Here is sample:
import threading
import time
from logs import LOG
def start_first():
LOG.log("First thread has started")
time.sleep(1000)
def start_second():
LOG.log("second thread has started")
if __name__ == '__main__':
### call birhtday daemon
first_thread = threading.Thread(target=start_first())
### call billing daemon
second_thread = threading.Thread(target=start_second())
### starting all daemons
first_thread.start()
second_thread.start()
In this code second thread does not work. I guess, after calling sleep function inside first_thread main process is slept. I found this post. But here sleep was used with class. I got that(Process finished with exit code 0
) as a result when I run answer. Could anybody explain me where I made a mistake ?
I am using python 3.* on windows
When creating your thread you are actually invoking the functions when trying to set the target for the Thread instead of passing a function to it. This means when you try to create the first_thread you are actually calling start_first which includes the very long sleep. I imagine you then get frustrated that you don't see the output from the second thread and kill it, right?
Remove the parens from your target= statements and you will get what you want
first_thread = threading.Thread(target=start_first)
second_thread = threading.Thread(target=start_second)
first_thread.start()
second_thread.start()
will do what you are trying
I'm trying to use a new thread or multiprocessing to run a function.
The function is called like this:
Actualize_files_and_folders(self)
i've read alot about multiprocessing and threads, and looking the questions at StackOverflow but i can't make it work... It would be great to have some help >.<
im calling the function with a button.
def on_button_act_clicked(self, menuitem, data=None):
self.window_waiting.show()
Actualize_files_and_folders(self)
self.window_waiting.hide()
In the waiting_window i have a button called 'cancel', it would be great if i can have a command/function that kills the thread.
i've tryed a lot of stuff, for exemple:
self.window_waiting.show()
from multiprocessing import Process
a=Process(Target=Actualize_files_and_folders(self))
a.start()
a.join()
self.window_waiting.hide()
But the window still freezing, and window_waiting is displayed at the end of Actualize_files_and_folders(self), like if i had called a normal function.
Thanks so much for any help!!
It looks like the worker function is being called rather than used as a callback for the process target:
process = Process(target=actualize_files_and_folders(self))
This is essentially equivalent to:
tmp = actualize_files_and_folders(self)
process = Process(target=tmp)
So the worker function is called in the main thread blocking it. The result of this function is passed into the Process as the target which will do nothing if it is None. You need to pass the function itself as a callback, not its result:
process = Process(target=actualize_files_and_folders, args=[self])
process.start()
See: https://docs.python.org/2/library/multiprocessing.html
I'm building a web scraper of a kind. Basically, what the soft would do is:
User (me) inputs some data (IDs) - IDs are complex, so not just numbers
Based on those IDs, the script visits http://localhost/ID
What is the best way to accomplish this? So I'm looking upwards of 20-30 concurrent connections to do it.
I was thinking, would a simple loop be the solution? This loop would start QThreads (it's a Qt app), so they would run concurrently.
The problem I am seeing with the loop however is how to instruct it to use only those IDs not used before i.e. in the iteration/thread that had been executed just before it was? Would I need some sort of a "delegator" function which will keep track of what IDs had been used and delegate the unused ones to the QThreads?
Now I've written some code but I am not sure if it is correct:
class GUI(QObject):
def __init__(self):
print "GUI CLASS INITIALIZED!!!"
self.worker = Worker()
for i in xrange(300):
QThreadPool().globalInstance().start(self.worker)
class Worker(QRunnable):
def run(self):
print "Hello world from thread", QThread.currentThread()
Now I'm not sure if these achieve really what I want. Is this actually running in separate threads? I'm asking because currentThread() is the same every time this is executed, so it doesn't look that way.
Basically, my question comes down to how do I execute several same QThreads concurrently?
Thanks in advance for the answer!
As Dikei says, Qt is red herring here. Focus on just using Python threads as it will keep your code much simpler.
In the code below we have a set, job_queue, containing the jobs to be executed. We also have a function, worker_thread which takes a job from the passed in queue and executes. Here it just sleeps for a random period of time. The key thing here is that set.pop is thread safe.
We create an array of thread objects, workers, and call start on each as we create it. From the Python documentation threading.Thread.start runs the given callable in a separate thread of control. Lastly we go through each worker thread and block until it has exited.
import threading
import random
import time
pool_size = 5
job_queue = set(range(100))
def worker_thread(queue):
while True:
try:
job = queue.pop()
except KeyError:
break
print "Processing %i..." % (job, )
time.sleep(random.random())
print "Thread exiting."
workers = []
for thread in range(pool_size):
workers.append(threading.Thread(target=worker_thread, args=(job_queue, )))
workers[-1].start()
for worker in workers:
worker.join()
print "All threads exited"