Develop Reference

multithreading to run UDP client and UDP server in the same program

I am trying to write a code in python, using multithreading, that runs UDP server side and UDP client side in a single program file. I need to make sure that the threads are synchronized.
The problem (as far as i have understood) with my code is that when thread1 runs, it acquires the lock since thread1's run() method runs the serverSide() method which contains a forever while loop, thread1 does not release the lock and therefore, the program gets stuck.
Can anyone please help me synchronize the threads while making sure the server and client run properly
import threading
import time
import sys
from datetime import datetime
from socket import *
class myThread (threading.Thread):
def __init__(self, threadID, name, counter):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
def run(self):
print "Starting " + self.name
# Get lock to synchronize threads
threadLock.acquire()
serverSide()
# Free lock to release next thread
threadLock.release()
class myThread1 (threading.Thread):
def __init__(self, threadID, name, counter):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
def run(self):
print "Starting " + self.name
# Get lock to synchronize threads
threadLock.acquire()
clientSide()
# Free lock to release next thread
threadLock.release()
def serverSide():
serverPort = 44000
serverIP = '192.168.0.0'
serverSocket = socket(AF_INET, SOCK_DGRAM)
serverSocket.bind((serverIP,serverPort))
print ("SERVER HERE!\nThe server is ready to receive")
while 1:
message, clientAddress = serverSocket.recvfrom(2048)
modifiedMessage = message.upper()
serverSocket.sendto(modifiedMessage, clientAddress)
def clientSide():
serverIP = "192.168.0.0"
serverPort = 44000
clientSocket = socket(AF_INET, SOCK_DGRAM)
message = raw_input("CLIENT HERE!\nInput lowercase sentence:")
clientSocket.sendto(message.encode(),(serverIP, serverPort))
modifiedMessage, serverAddress = clientSocket.recvfrom(2048)
print(modifiedMessage) # print the received message
clientSocket.close() # Close the socket
threadLock = threading.Lock()
threads = []
# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread1(2, "Thread-2", 2)
# Start new Threads
thread1.start()
thread2.start()
# Add threads to thread list
threads.append(thread1)
threads.append(thread2)
# Wait for all threads to complete
for t in threads:
t.join()
print "Exiting Main Thread"

Your code works fine except the synchronisation part.
So the issue is this, There is one threadLock = threading.Lock() which once acquired by either of the two threads the other thread wont be able to acquire.
Once a thread has acquired it, It wont release it till its job is done. It's job cant be done unless the other thread is up and running. The other thread is waiting for the 1st thread to release lock.
You have artificially manage to induce a race condition in your code which is not at all required. I simply removed the entire part with thread lock and its working fine. Except that i had to interrupt the program to end it.
import threading
import time
import sys
from datetime import datetime
from socket import *
class myThread (threading.Thread):
def __init__(self, threadID, name, counter):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
def run(self):
print ("Starting " + self.name)
# Get lock to synchronize threads
# threadLock.acquire()
serverSide()
# Free lock to release next thread
# threadLock.release()
class myThread1 (threading.Thread):
def __init__(self, threadID, name, counter):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
def run(self):
print ("Starting " + self.name)
# Get lock to synchronize threads
# threadLock.acquire()
clientSide()
# Free lock to release next thread
# threadLock.release()
def serverSide():
serverPort = 44000
serverIP = '127.0.0.1'
serverSocket = socket(AF_INET, SOCK_DGRAM)
serverSocket.bind((serverIP,serverPort))
print ("SERVER HERE!\nThe server is ready to receive")
while 1:
message, clientAddress = serverSocket.recvfrom(2048)
modifiedMessage = message.upper()
serverSocket.sendto(modifiedMessage, clientAddress)
def clientSide():
serverIP = "127.0.0.1"
serverPort = 44000
clientSocket = socket(AF_INET, SOCK_DGRAM)
message = input("CLIENT HERE!\nInput lowercase sentence:")
clientSocket.sendto(message.encode(),(serverIP, serverPort))
modifiedMessage, serverAddress = clientSocket.recvfrom(2048)
print("received", modifiedMessage) # print the received message
clientSocket.close() # Close the socket
# threadLock = threading.Lock()
threads = []
# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread1(2, "Thread-2", 2)
# Start new Threads
thread1.start()
thread2.start()
# Add threads to thread list
threads.append(thread1)
threads.append(thread2)
# Wait for all threads to complete
for t in threads:
t.join()
print ("Exiting Main Thread")
output:
Starting Thread-1
Starting Thread-2
SERVER HERE!
The server is ready to receive
CLIENT HERE!
Input lowercase sentence:viki
received b'VIKI'
NOTE:
i am trying to write a code in python, using multithreading, that runs UDP server side and UDP client side in a single program file. i need to make sure that the threads are synchronized.
A client server architecture is in most cases not supposed to be synchronized. Google server and my browser are not synchronized. And they are not supposed to be same applies for your code. Reason for that being that a server should run independent of wether a client is running or not.
A client should run independent of wether a server is up or not. Client request will fail if the server is down. But still it should run.

Tkinter Threading causing UI freeze

I've searched this site (and others) up and down but I can't seem to find the right solution.
I have a client program that connects to a server and automatically sends a message every few seconds, as well as on user command. I'm using multiple threads for this.
Enter Tkinter: Once I hit the 'Connect' button, my UI freezes, either until the connection attempt times out or until the end of time, should the client connect to the server.
I've tried calling the thread from the button's command parameter, from inside the main loop, and outside the main loop. I've tried putting the main loop in a thread and then creating a new thread for the connection from there.
Nothing seems to be working...the UI continues to hang.
class EventSim(Frame):
def __init__(self, parent):
self.queue = Queue
Frame.__init__(self, parent)
self.parent = parent
def initUI(self,IP_Address,Port_Number,Events_Directory):
#...
self.Con_Button = Button(frame7,text='Connect', command = lambda:
self.connect(IP_Text.get(),Port_Text.get(),))
def connect(self,IP,Port):
ConnectionThread = Thread(eventsim.con_thread(IP,Port))
ConnectionThread.start()
def main():
root = Tk()
root.geometry("300x310+750+300")
Sim = EventSim(root)
eventsim.readconfig()
Sim.initUI(eventsim.ipa,eventsim.portnum,eventsim.event_dir)
root.mainloop()

You pass the result of eventsim.con_thread(IP,Port) to Thread(...) function, so it will wait until the execution of eventsim.con_thread(...) completes. Try changing:
def connect(self, IP, Port):
ConnectionThread = Thread(eventsim.con_thread(IP,Port))
ConnectionThread.start()
to:
def connect(self, IP, Port):
ConnectionThread = Thread(target=lambda ip=IP, port=Port: eventsim.con_thread(ip,port))
ConnectionThread.start()

What is correct way of creating threaded zeromq socket?

I'm wondering how to correctly create background thread that would be listenning some random port and pushing received object to Queue?
I want my socket wrapper to launch new thread, select some random port and start listenning on in. I have to be able to get this port number from socket wrapper.
I've come up with simple class:
class SocketWrapper(Thread):
def __init__(self, socket_type, *args, **kwargs):
super(Thread, self).__init__(*args, **kwargs)
self._ctx = zmq.Context()
self._socket = self._ctx._socket(socket_type)
self.port = self._socket.bind_to_random_port('tcp://*')
self._queue = Queue()
def run(self):
while not self.stop_requested:
try:
item = socket.recv_pyobj(flags=zmq.NOBLOCK)
self._queue.put(item)
except ZMQError:
time.sleep(0.01) # Wait a little for next item to arrive
However, zmq sockets can't be shared between threads, they are not thread-safe (http://api.zeromq.org/2-1:zmq). So socket creation and binding should be moved to run() method:
class SocketWrapper2(Thread):
def __init__(self, socket_type, *args, **kwargs):
super(Thread, self).__init__(*args, **kwargs)
self._socket_type = socket_type
self._ctx = zmq.Context()
self._queue = Queue()
self._event = Event()
def run(self):
socket = self._ctx._socket(self._socket_type)
self.port = self._socket.bind_to_random_port('tcp://*')
self._event.set()
while not self.stop_requested:
try:
item = socket.recv_pyobj(flags=zmq.NOBLOCK)
self._queue.put(item)
except ZMQError:
time.sleep(0.01) # Wait a little for next item to arrive
def get_port(self):
self._event.wait()
return self.port
I had to add event to be sure that port is already binded before I can read it but it introduces risk of deadlock, when SocketWrapper2.get_port() is called before start(). This can be avoided by using Thread's _started Event:
def get_port(self):
if not self._started.is_set():
raise RuntimeError("You can't call run_port before thread start.")
self._event.wait()
return self.port
Is this is at last thread-safe? Is there anything else to take care of?
Problem I still see here is that I want to get port right after SocketWrapper is created. Can I safely call Thread's start() in __init__?

I ended up modifying this solution a little to avoid deadlocking main thread:
def get_port(self):
if not self._started.is_set():
raise RuntimeError("You can't call run_port before thread start.")
if not self._event.wait(1):
raise RuntimeError("Couldn't get port after a while.")
return self.port
This is not perfect. Since we delay get_port but it's simple and do the job. Any suggestions how to improve it?

How to have two programs communicate with one another

I am dipping my toes into networking and multithreading in python. I have gone through the docs on concurrent.futures and SocketServer and am attempting to use these in the example I am working on. The examples in the docs seem strait forward enough but am struggling to apply them to the example I am working on.
The example is as follows. 2 GUI applications, one that sends information to the other via user interaction and the other displays this information.
So far I have the 2 applications up and running. Application A has the server running in a separate thread. Application B is able to connect to the server and send the desired information.
At this point I cannot seem to find a nice way to get the information displayed in the GUI of application A. I can think of several hacky ways of doing it but am interested in the nicest / most pythonic way. This seems a common problem so there must be a common pattern to use here. So my questions are.
While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
Example code is as follows
Server Window
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
t = QtGui.QTextEdit()
l.addWidget(t)
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()
Client Window
import socket
import sys
import functools
from PyQt4 import QtGui
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
class ClientWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 650, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
for i in range(5):
name = 'test %d' % i
b = QtGui.QPushButton(name)
l.addWidget(b)
b.pressed.connect(functools.partial(self.onButtonClick, name))
self.show()
def onButtonClick(self, buttonName):
print('connecting to server')
tcpCliSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpCliSock.connect(ADDR)
print('Sending name %s' % buttonName)
tcpCliSock.send(buttonName)
tcpCliSock.close()
def launch():
app = QtGui.QApplication(sys.argv)
ex = ClientWindow()
sys.exit(app.exec_())
if __name__ == '__main__':
launch()

So one of the few ways I know of, that transfers data from a thread to the main GUI thread of an application, is to place the data in a python Queue. This Queue is read by a QThread (a Qt threading implementation that supports Qt signals and slots). The QThread makes a blocking call to queue.get(). When data is placed in the Queue by your handle() method, the QThread unblocks, reads the data out of the queue, and emits a thread-safe signal to the GUI thread. As a demonstration, I added the data into the QTextEdit.
So basically you need an intermediary between a python thread and the Qt GUI which generally interacts via signals/slots. The QThread performs this task, linking the thread-safe Queue object, with the thread-safe qt signal emission.
This effectively follows a similar answer I gave here, but here you have a socket instead of a custom thread putting the data in the queue.
You might also be interested in this SO post, which explains why some of the lines of code I have used to make the QThread and connect the signals, etc, are written in the order they are!
P.S. I added a line to shutdown the socket server when the app window is closed (the socket server used to keep running in the background)
Server Window code
import SocketServer
import concurrent.futures
import sys
from PyQt4 import QtGui
from PyQt4 import QtCore
from Queue import Queue
HOST = 'localhost'
PORT = 21567
BUFSIZ = 1024
ADDR = (HOST, PORT)
# create a global queue object that both the handle() method and the QThread (see later in the code) can access
queue = Queue()
class MyRequestHandler(SocketServer.StreamRequestHandler):
def handle(self):
print('...connected from:', self.client_address)
data = self.rfile.readline().strip()
print('Data from client %s' % data)
queue.put(data)
class ServerWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.setGeometry(1500, 100, 500, 500)
self._control = QtGui.QWidget()
self.setCentralWidget(self._control)
l = QtGui.QVBoxLayout(self._control)
self.t = QtGui.QTextEdit()
l.addWidget(self.t)
self.executor = futures.ThreadPoolExecutor(max_workers=1)
self.startServerThread()
self.show()
#QtCore.pyqtSlot(str)
def receive_data(self, data):
self.t.moveCursor(QtGui.QTextCursor.End)
self.t.insertPlainText( data )
def startServerThread(self):
self.executor.submit(self.startServer)
# How to get information from the thread while it is still running?
def startServer(self):
print('starting server')
self.tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
print('waiting for connection...')
self.tcpServ.serve_forever()
# How to get information from the client (custom request handler)
# back to the GUI in a thread safe manner?
# This class runs in a QThread and listens on the end of a queue and emits a signal to the GUI
class MyReceiver(QtCore.QObject):
mysignal = QtCore.pyqtSignal(str)
def __init__(self,queue,*args,**kwargs):
QtCore.QObject.__init__(self,*args,**kwargs)
self.queue = queue
#QtCore.pyqtSlot()
def run(self):
while True:
text = self.queue.get()
self.mysignal.emit(text)
def launch():
app = QtGui.QApplication(sys.argv)
ex = ServerWindow()
# Create thread that will listen on the other end of the queue, and send the text to the textedit in our application
thread = QtCore.QThread()
my_receiver = MyReceiver(queue)
my_receiver.mysignal.connect(ex.receive_data)
my_receiver.moveToThread(thread)
thread.started.connect(my_receiver.run)
thread.start()
ret_code = app.exec_()
ex.tcpServ.shutdown()
sys.exit(ret_code)
if __name__ == '__main__':
launch()

While the server is running, how to get the information from the custom request handler to the server.
how to get information from the thread to the main application while it is running?
These are my thoughts of how it should work.
class MyRequestHandler(SocketServer.StreamRequestHandler):
#property
def application_window(self):
# you would override setup() for this, usually.
return self.server.application_window
def handle(self):
print(self.application_window)
# ...
def startServer(self):
print('starting server')
tcpServ = SocketServer.TCPServer(ADDR, MyRequestHandler)
tcpServ.application_window = self # !!!!!!!!!!!!!!!!!!!!!!!!! added
print('waiting for connection...', self)
tcpServ.serve_forever()
Maybe you need to adjust something. By the way there are techniques for sharing information between server and client.

Thread doesn't close on exiting tk window

I have an application written in Python, using Tkinter. One of the features allows a serial port to be opened, after which any messages received over the serial port are displayed in a text window. This works fine. The problem comes when I close the window, which doesn't kill the thread monitoring the serial port. It then has to be killed manually (alternatively, unplugging the USB-serial cable causes an exception which kills the process).
I assume I'm missing something simple here, but I would have thought closing the application would close all associated threads. I can't seem to find anything about this in the documentation, but I'm probably looking in the wrong place?
Code for the serial thread in case it's relevant:
class SerialThread(threading.Thread):
def __init__(self, queue, sp):
threading.Thread.__init__(self)
self.queue = queue
self.ser_handle = sp;
def run(self):
while True:
if self.ser_handle.inWaiting():
text = self.ser_handle.readline(self.ser_handle.inWaiting())
self.queue.put(text)
time.sleep(0.2)

You must have a way to ask the thread for stop, you can accomplish that using: threading.Event.
class SerialThread(threading.Thread):
def __init__(self, queue, sp):
threading.Thread.__init__(self)
self.queue = queue
self.event = threading.Event() # An event object.
self.ser_handle = sp;
def stop(self):
self.event.set()
def run(self):
while not self.event.isSet():
if self.ser_handle.inWaiting():
text = self.ser_handle.readline(self.ser_handle.inWaiting())
self.queue.put(text)
time.sleep(0.2)
Then in the on close event of your window, call your_thread.stop()