I know that Queue.get() method in python is a blocking function. I need to know if I implemented this function inside the main, waiting for an object set by a thread, does this means that all the main will be blocked.
For instance, if the main contains functions for transmitter and receiver, will the two work together or not?
Yes -- if you call some_queue.get() within either the thread or the main function, the program will block there until some object as passed through the queue.
However, it is possible to use queues so that they don't block, or so that they have a timeout of some kind:
import Queue
while True:
try:
data = some_queue.get(False)
# If `False`, the program is not blocked. `Queue.Empty` is thrown if
# the queue is empty
except Queue.Empty:
data = None
try:
data2 = some_queue.get(True, 3)
# Waits for 3 seconds, otherwise throws `Queue.Empty`
except Queue.Empty:
data = None
You can do the same for some_queue.put -- either do some_queue.put(item, False) for non-blocking queues, or some_queue.put(item, True, 3) for timeouts. If your queue has a size limit, it will throw a Queue.Full exception if there is no more room left to append a new item.
Yes it will block main/thread. if you want to get all messages without blocking try this
def get_messages(q):
messages = []
while q.qsize():
messages.append(q.get())
# or process message here
return messages
If messages are like stream above code might get caught in loop.
to avoid that use "for loop" and get all the messages sent so far
def get_messages(q):
messages = []
for _ in range(q.qsize()):
messages.append(q.get())
# or process message here
return messages
Related
I'm having trouble with a thread hanging when I call join() on it. What I am trying to do is use the Go-back N protocol for sending/receiving packets over a network, and I created a separate thread for handling the ACK's that come back from the server.
I have a single thread run on this method that checks for incoming packets and retrieves the ACK number, then stores that number in a variable set-up in the init called self.lastAck. Simplified version of the method:
#Anything not explicitly defined here is global variable
def ack_check(self):
ack_num = 0
pktHdrData = '!BBBBHHLLQQLL'
# Listening for ack number from server and store it in self.lastAck.
while True:
# variable also inside the __init__ method
if (self.finish == 1):
break
data,address = sock.recvfrom(4096)
clientAck = struct.unpack(pktHdrData,data)
ackNumRecv = clientAck[9]
self.lastAck = ackNumRecv
A simplified version of the function that creates the thread and handles the sending of the client packets:
def send(self,buffer):
# Assume packet header and all relevant data is set up correctly here
# ...
t1 = threading.Thread(target = self.ack_check, args=())
t1.setDaemon = True
t1.start()
# All of this works perfectly and breaks as expected
while True:
# Packets/data get sent here and break when self.lastAck reaches a specific number. Assume this works properly and breaks
self.finish = 1
print("About to hang here")
t1.join()
return bytessent
I end up hanging right after printing the About to end here and I can't figure out why. I can get it to work if I break out of the while True loop in the else section, but then I end up closing the thread before I receive all the ACK numbers from the receiver. So instead of the full 32 ACK's I'll end up with anywhere from 1 ACK to the full 32.
I think the problem lies in the def ack_check(self) method where it doesn't break out of the loop even though it should be after I call self.finish = 1 but it just ends up hanging every time.
Additionally there is nothing else outside of these two methods that are calling self.finish and self.lastAck. I know about deadlocking but I couldn't see how that would be possible in this situation.
Sidenote: I realize the Go-Back N protocol is not properly implemented at all here, but this was the first step I took in creating it.
As per the comments, the recvfrom call in ack_check left the thread hanging. Fixed code:
def ack_check(self):
ack_num = 0
pktHdrData = '!BBBBHHLLQQLL'
# Listening for ack number from server and store it in self.lastAck.
while True:
# variable also inside the __init__ method
if (self.finish == 1):
break
sock.timeout(0.2)
try:
data,address = sock.recvfrom(4096)
except socket.timeout:
break
clientAck = struct.unpack(pktHdrData,data)
ackNumRecv = clientAck[9]
self.lastAck = ackNumRecv
I'm trying to write a chat application in Python as a school project.
It should be receiving messages from a server and at the same time, it also should be able to send messages to that server. In order to do that, I created two threads: one waits for the incoming messages and the other one is takes inputs from me to send over. The problem is that it can't print the messages which are coming from the server because the other thread is always asking for input. Is there any way to make the message-receiving thread print the incoming messages while the input function asks for input?
Here is the troublesome part of the code:
def sendmsg(conn):
while True:
msg=input("Your message: ")
conn.send(bytes(msg,"utf-8"))
def getmsg(conn):
while True:
data=conn.recv(1024)
print(data.decode("utf-8"))
def server():
soket=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
soket.bind((HOST,PORT))
soket.listen()
print("Listining")
conn, addr = soket.accept()
print("Connection established!")
send = threading.Thread(target = sendmsg(conn))
get = threading.Thread(target = getmsg(conn))
get.start()
send.start()
The basic problem is with the following two lines:
send = threading.Thread(target = sendmsg(conn))
get = threading.Thread(target = getmsg(conn))
There is a big difference in Python between the function object sendmsg and the result of calling that object sendmsg(conn).
The Thread object send never gets created, much less started, because the parameter you are trying to pass in is the result of the call sendmsg(conn), but the function never returns. You have effectively entered an infinite loop at that point, always asking for user input in the main thread.
Instead, you should be passing in the sendmsg function object, and using the args parameter to Thread to let it know that you want to pass in an extra parameter when it does get called. The same applies to getmsg:
send = threading.Thread(target=sendmsg, args=(conn,))
get = threading.Thread(target=getmsg, args=(conn,))
Be careful to include the comma in args=(conn,), or the argument won't be interpreted as a tuple. You can use a list instead if you prefer: args=[conn].
I am using Python3 modules:
requests for HTTP GET calls to a few Particle Photons which are set up as simple HTTP Servers
As a client I am using the Raspberry Pi (which is also an Access Point) as a HTTP Client which uses multiprocessing.dummy.Pool for making HTTP GET resquests to the above mentioned photons
The polling routine is as follows:
def pollURL(url_of_photon):
"""
pollURL: Obtain the IP Address and create a URL for HTTP GET Request
#param: url_of_photon: IP address of the Photon connected to A.P.
"""
create_request = 'http://' + url_of_photon + ':80'
while True:
try:
time.sleep(0.1) # poll every 100ms
response = requests.get(create_request)
if response.status_code == 200:
# if success then dump the data into a temp dump file
with open('temp_data_dump', 'a+') as jFile:
json.dump(response.json(), jFile)
else:
# Currently just break
break
except KeyboardInterrupt as e:
print('KeyboardInterrupt detected ', e)
break
The url_of_photon values are simple IPv4 Addresses obtained from the dnsmasq.leases file available on the Pi.
the main() function:
def main():
# obtain the IP and MAC addresses from the Lease file
IP_addresses = []
MAC_addresses = []
with open('/var/lib/misc/dnsmasq.leases', 'r') as leases_file:
# split lines and words to obtain the useful stuff.
for lines in leases_file:
fields = lines.strip().split()
# use logging in future
print('Photon with MAC: %s has IP address: %s' %(fields[1],fields[2]))
IP_addresses.append(fields[2])
MAC_addresses.append(fields[1])
# Create Thread Pool
pool = ThreadPool(len(IP_addresses))
results = pool.map(pollURL, IP_addresses)
pool.close()
pool.join()
if __name__ == '__main__':
main()
Problem
The program runs well however when I press CTRL + C the program does not terminate. Upon digging I found that the way to do so is using CTRL + \
How do I use this in my pollURL function for a safe way to exit the program, i.e. perform poll.join() so no leftover processes are left?
notes:
the KeyboardInterrupt is never recognized with the function. Hence I am facing trouble trying to detect CTRL + \.
The pollURL is executed in another thread. In Python, signals are handled only in the main thread. Therefore, SIGINT will raise the KeyboardInterrupt only in the main thread.
From the signal documentation:
Signals and threads
Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.
Besides, only the main thread is allowed to set a new signal handler.
You can implement your solution in the following way (pseudocode).
event = threading.Event()
def looping_function( ... ):
while event.is_set():
do_your_stuff()
def main():
try:
event.set()
pool = ThreadPool()
pool.map( ... )
except KeyboardInterrupt:
event.clear()
finally:
pool.close()
pool.join()
I use multiprocessing.connection.Listener for communication between processes, and it works as a charm for me. Now i would really love my mainloop to do something else between commands from client. Unfortunately listener.accept() blocks execution until connection from client process is established.
Is there a simple way of managing non blocking check for multiprocessing.connection? Timeout? Or shall i use a dedicated thread?
# Simplified code:
from multiprocessing.connection import Listener
def mainloop():
listener = Listener(address=(localhost, 6000), authkey=b'secret')
while True:
conn = listener.accept() # <--- This blocks!
msg = conn.recv()
print ('got message: %r' % msg)
conn.close()
One solution that I found (although it might not be the most "elegant" solution is using conn.poll. (documentation) Poll returns True if the Listener has new data, and (most importantly) is nonblocking if no argument is passed to it. I'm not 100% sure that this is the best way to do this, but I've had success with only running listener.accept() once, and then using the following syntax to repeatedly get input (if there is any available)
from multiprocessing.connection import Listener
def mainloop():
running = True
listener = Listener(address=(localhost, 6000), authkey=b'secret')
conn = listener.accept()
msg = ""
while running:
while conn.poll():
msg = conn.recv()
print (f"got message: {msg}")
if msg == "EXIT":
running = False
# Other code can go here
print(f"I can run too! Last msg received was {msg}")
conn.close()
The 'while' in the conditional statement can be replaced with 'if,' if you only want to get a maximum of one message at a time. Use with caution, as it seems sort of 'hacky,' and I haven't found references to using conn.poll for this purpose elsewhere.
You can run the blocking function in a thread:
conn = await loop.run_in_executor(None, listener.accept)
I've not used the Listener object myself- for this task I normally use multiprocessing.Queue; doco at the following link:
https://docs.python.org/2/library/queue.html#Queue.Queue
That object can be used to send and receive any pickle-able object between Python processes with a nice API; I think you'll be most interested in:
in process A
.put('some message')
in process B
.get_nowait() # will raise Queue.Empty if nothing is available- handle that to move on with your execution
The only limitation with this is you'll need to have control of both Process objects at some point in order to be able to allocate the queue to them- something like this:
import time
from Queue import Empty
from multiprocessing import Queue, Process
def receiver(q):
while 1:
try:
message = q.get_nowait()
print 'receiver got', message
except Empty:
print 'nothing to receive, sleeping'
time.sleep(1)
def sender(q):
while 1:
message = 'some message'
q.put('some message')
print 'sender sent', message
time.sleep(1)
some_queue = Queue()
process_a = Process(
target=receiver,
args=(some_queue,)
)
process_b = Process(
target=sender,
args=(some_queue,)
)
process_a.start()
process_b.start()
print 'ctrl + c to exit'
try:
while 1:
time.sleep(1)
except KeyboardInterrupt:
pass
process_a.terminate()
process_b.terminate()
process_a.join()
process_b.join()
Queues are nice because you can actually have as many consumers and as many producers for that exact same Queue object as you like (handy for distributing tasks).
I should point out that just calling .terminate() on a Process is bad form- you should use your shiny new messaging system to pass a shutdown message or something of that nature.
The multiprocessing module comes with a nice feature called Pipe(). It is a nice way to share resources between two processes(never tried more than two before). With the dawn of python 3.80 came the shared memory function in the multiprocessing module but i have not really tested that so i cannot vouch for it
You will use the pipe function something like
from multiprocessing import Pipe
.....
def sending(conn):
message = 'some message'
#perform some code
conn.send(message)
conn.close()
receiver, sender = Pipe()
p = Process(target=sending, args=(sender,))
p.start()
print receiver.recv() # prints "some message"
p.join()
with this you should be able to have separate processes running independently and when you get to the point which you need the input from one process. If there is somehow an error due to the unrelieved data of the other process you can put it on a kind of sleep or halt or use a while loop to constantly check pending when the other process finishes with that task and sends it over
while not parent_conn.recv():
time.sleep(5)
this should keep it in an infinite loop until the other process is done running and sends the result. This is also about 2-3 times faster than Queue. Although queue is also a good option personally I do not use it.
I am trying to create a two player game in pygame using sockets, the thing is, when I try to receive data on on this line:
message = self.conn.recv(1024)
python hangs until it gets some data. The problem with this is that is pauses the game loop when the client is not sending anything through the socket and causes a black screen. How can I stop recv from doing this?
Thanks in advance
Use nonblocking mode. (See socket.setblocking.)
Or check if there is data available before call recv.
For example, using select.select:
r, _, _ = select.select([self.conn], [], [])
if r:
# ready to receive
message = self.conn.recv(1024)
you can use signal module to stop an hangs recv thread.
in recv thread:
try:
data = sock.recv(1024)
except KeyboardInterrupt:
pass
in interpret thread:
signal.pthread_kill(your_recving_thread.ident, signal.SIGINT)
I know that this is an old post, but since I worked on a similar project lately, I wanted to add something that hasn't already been stated yet for anybody having the same issue.
You can use threading to create a new thread, which will receive data. After this, run your game loop normally in your main thread, and check for received data in each iteration. Received data should be placed inside a queue by the data receiver thread and read from that queue by the main thread.
#other imports
import queue
import threading
class MainGame:
def __init__(self):
#any code here
self.data_queue = queue.Queue()
data_receiver = threading.Thread(target=self.data_receiver)
data_receiver.start()
self.gameLoop()
def gameLoop(self):
while True:
try:
data = self.data_queue.get_nowait()
except queue.Empty:
pass
self.gameIteration(data)
def data_receiver(self):
#Assuming self.sock exists
data = self.sock.recv(1024).decode("utf-8")
#edit the data in any way necessary here
self.data_queue.put(data)
def gameIteration(self, data):
#Assume this method handles updating, drawing, etc
pass
Note that this code is in Python 3.