I would like to use XSUB/XPUB to enable multiple ZMQ publishers and subscribers. Everything works when I use zmq.proxy(xpub_socket, xsub_socket), but I need something custom because I need to write code between XSUB and XPUB that examines the messages.
Here's where I'm at:
import time
import zmq
context = zmq.Context()
address = '127.0.0.1'
pub_port = '3000'
sub_port = '3001'
# XSUB socket
xsub_socket = context.socket(zmq.XSUB)
xsub_socket.bind(f'tcp://{address}:{pub_port}')
# XPUB socket
xpub_socket = context.socket(zmq.XPUB)
xpub_socket.bind(f'tcp://{address}:{sub_port}')
time.sleep(1)
# PUB socket
pub_socket = context.socket(zmq.PUB)
pub_socket.connect(f'tcp://{address}:{pub_port}')
# SUB socket
sub_socket = context.socket(zmq.SUB)
sub_socket.subscribe('')
sub_socket.connect(f'tcp://{address}:{sub_port}')
time.sleep(1)
pub_socket.send_string('test')
time.sleep(1)
print(poller.poll(0))
The values sent from the PUB socket do not reach the XSUB socket.
I read here that the first byte needs to be 1. Both of these also don't work:
pub_socket.send(b'\x01')
pub_socket.send_multipart([b'\x01', 'test'.encode('utf-8')])
What am I doing wrong here?
A PUB socket won't send any messages to an XSUB socket unless it has received a subscription request, which you get by calling subscribe on a SUB socket.
The only way those subscription messages get passed through is if you set up your XSUB/XPUB proxy.
Here's a simple proxy that connects an XPUB and XSUB socket, printing out messages it receives in either direction:
import zmq
ctx = zmq.Context()
xpub_sock = ctx.socket(zmq.XPUB)
xpub_sock.bind("tcp://127.0.0.1:3000")
xsub_sock = ctx.socket(zmq.XSUB)
xsub_sock.bind("tcp://127.0.0.1:3001")
poller = zmq.Poller()
poller.register(xpub_sock, zmq.POLLIN)
poller.register(xsub_sock, zmq.POLLIN)
while True:
socks = dict(poller.poll())
if xpub_sock in socks and socks[xpub_sock] == zmq.POLLIN:
msg = xpub_sock.recv_multipart()
print("(sub)", msg)
xsub_sock.send_multipart(msg)
elif xsub_sock in socks and socks[xsub_sock] == zmq.POLLIN:
msg = xsub_sock.recv_multipart()
print("(pub)", msg)
xpub_sock.send_multipart(msg)
If I connect to this with an PUB socket, like this...
import zmq
import time
ctx = zmq.Context()
pub_sock = ctx.socket(zmq.PUB)
pub_sock.connect("tcp://localhost:3001")
while True:
pub_sock.send_string("test")
time.sleep(1)
...I won't see any messages arriving at the XSUB socket, because
there are no active subscriptions. However, if I connect a SUB
socket to the XPUB socket and set a subscription...
import zmq
ctx = zmq.Context()
sub_sock = ctx.socket(zmq.SUB)
sub_sock.connect("tcp://localhost:3000")
sub_sock.subscribe("")
while True:
msg = sub_sock.recv()
print(msg)
...then I will start to see messages passing from the PUB socket to
the XSUB socket, and then from the XPUB socket to the SUB
socket.
Related
I am following the Bidirectional pair socket example from the pyzmq documentation. I created two files,
socket1.py with this code
import zmq
import random
import sys
import time
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.connect("tcp://localhost:{}".format(port))
print("Socket created")
i = 0
while True:
msg = socket.recv()
print("socket: msg recved")
print(msg)
socket.send_string("hello from 1")
time.sleep(1)
print(i)
and socket2.py with this code:
import zmq
import random
import sys
import time
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.connect("tcp://localhost:{}".format(port))
print("Socket created")
i = 0
while True:
socket.send_string("hello from 2")
print("socket2: msg sent")
msg = socket.recv()
print(msg)
time.sleep(1)
print(i)
I run both the files in 2 separate terminals but it prints the following messages in terminal1 and terminal2 respectively:
Socket created
Socket created
socket2: msg sent
I am not able to understand why socket1 is not receiving the msg sent by socket2 and is stuck at socket.recv(). I would really appreciate any help. TIA.
You are close. The first example, socket1.py needs to bind to the port. zeromq will turn that into a listen for the client to connect.
import zmq
import random
import sys
import time
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.bind("tcp://localhost:{}".format(port))
print("Socket created")
i = 0
while True:
msg = socket.recv()
print("socket: msg recved")
print(msg)
socket.send_string("hello from 1")
time.sleep(1)
print(i)
When I use DEALER-DEALER model to communicate, I find: server send messages in different time, but client receive messages in the same. For example (Python code):
# server
context = zmq.Context()
sock_conn = self.context.socket(zmq.DEALER)
sock_conn.bind("tcp://*:%d" % port)
sock_conn.send("msg1")
time.sleep(0.05)
sock_conn.send("msg2")
# client
context_send = zmq.Context()
sock_send = self.context.socket(zmq.DEALER)
sock_send.connect("tcp://%s:%d" % (ip, port1))
while True:
msg = sock_send.recv()
print(msg)
The server sends the message in different time, but sometimes the client receives the messages at the same time, who can explain this problem, thanks.
I have run into a strange behaviour with ZeroMQ that I have been trying to debug the whole day now.
Here is a minimal example script which reproduces the problem. It can be run with Python3.
One server with a REP socket is started and five clients with REP sockets connect to it basically simultaneously. The result is that the server starts to block for some reason after the first few messages. It seems like the poller.poll(1000) is what blocks indefinitely.
This behavior also seems to be timing-dependant. Insert a sleep(0.1) in the loop that starts the clients and it works as expected.
I would have expected the REP socket to queue all incoming messages and release them one after the other via sock.recv_multipart().
What is happening here?
import logging
from threading import Thread
from time import sleep
import zmq
logging.basicConfig(level=logging.INFO)
PORT = "3446"
stop_flag = False
def server():
logging.info("started server")
context = zmq.Context()
sock = context.socket(zmq.REP)
sock.bind("tcp://*:" + PORT)
logging.info("bound server")
poller = zmq.Poller()
poller.register(sock, zmq.POLLIN)
while not stop_flag:
socks = dict(poller.poll(1000))
if socks.get(sock) == zmq.POLLIN:
request = sock.recv_multipart()
logging.info("received %s", request)
# sleep(0.5)
sock.send_multipart(["reply".encode()] + request)
sock.close()
def client(name:str):
context = zmq.Context()
sock = context.socket(zmq.REQ)
sock.connect("tcp://localhost:" + PORT)
sock.send_multipart([name.encode()])
logging.info(sock.recv_multipart())
sock.close()
logging.info("starting server")
server_thread = Thread(target=server)
server_thread.start()
sleep(1)
nr_of_clients = 5
for i in range(nr_of_clients):
Thread(target=client, args=[str(i)]).start()
stop_flag = True
For me the problem seems to be that you are "shutting down" the server before all clients have received their reply. So I guess its not the server who's blocking but clients are.
You can solve this by either waiting some time before you set the stop_flag:
sleep(5)
stop_flag = True
or, better, you explicitely join the client threads like:
nr_of_clients = 5
threads = []
for i in range(nr_of_clients):
thread = Thread(target=client, args=[str(i)])
thread.start()
threads.append(thread)
for thread in threads:
thread.join()
stop_flag = True
I'm trying to implement a simple ZeroMQ socket server as a twisted protocol, my code is currently:
# Setup
context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.bind("tcp://127.0.0.1:5558")
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
# Respond to incoming messages.
while True:
if poller.poll(1000):
msg = self.socket.recv()
print "Received a message ..."
else:
time.sleep(0.05)
and I found:
http://twistedmatrix.com/documents/11.0.0/api/twisted.internet.interfaces.IProtocol.html
but honestly can't make sense of it. Where should I be putting the code for setup, and code for receiving data?
I want to test a complex class, which wraps some methods of the socket module: connect, sendall and recv. Especially, I want to test the recv method of this class.
The working example code below shows how I could do that (in a basic, underlying form to keep it simple, testsocket would correspond to the complex wrapper class):
import socket
# This is just a socket for testing purposes, binds to the loopback device
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("127.0.0.1", 1234))
sock.listen(5)
# This is the socket later part of the complex socket wrapper.
# It just contains calls to connect, sendall and recv
testsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
testsocket.connect(("127.0.0.1", 1234))
testsocket.sendall("test_send")
# The testing socket connects to a client
(client, adr) = sock.accept()
print client.recv(1024)
# Now I can do the actual test: Test the receive method of the socket
# wrapped in the complex class
client.sendall("test_recv")
print testsocket.recv(1024) # <-- This is what I want to test !!
# close everything
testsocket.close()
client.close()
sock.close()
But in order to test testsocket.recv I need to use testsocket.sendall before.
Is it possible to modify this code in a simple way (without forks or threads) in order to test testsocket.recv without using the method testsocket.sendall?
How about using socket.socketpair? :
import socket
client, testsocket = socket.socketpair()
client.sendall("test_recv")
print testsocket.recv(1024)
testsocket.close()
client.close()
NOTE only available in Unix.
Using mock
import mock
testsocket = mock.Mock()
testsocket.configure_mock(**{'recv.return_value': 'test_recv'})
print testsocket.recv(1024)
You can't run the client and the server socket in the same process/thread since the server.recv() is a blocking call
my routine :
import socket, threading
# Protocols supported
TCP = (0x00)
UDP = (0x01)
UDP_Multicast = (0x02)
# Client/ Server mode
CLIENT = (0x00)
SERVER = (0x01)
# Data to be sent
data = 'Test. Please Ignore'
# Server processing
def simple_processing(data):
print "messsage : ", data
def start_socket( protocol, client, processing_callback):
# switch on protocol
if protocol == TCP:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
else:
return
# switch on client
if client == SERVER:
# Server mode = listening to incoming datas
sock.bind(("127.0.0.1", 1234))
sock.listen(5)
(sock, adr) = sock.accept()
processing_callback( sock.recv(1024) ) # processing data
elif client == CLIENT:
# Client mode : connecting and sending data
sock.connect(("127.0.0.1", 1234))
sock.sendall(data)
else:
return
sock.close()
def test():
# Thread creations
server = threading.Thread( target = start_socket,
args=( TCP,
SERVER,
simple_processing, )
)
client = threading.Thread( target= start_socket,
args=( TCP,
CLIENT,
None)
)
server.start()
client.start()
# Join : wait on every thread to finish
client.join()
server.join()
if __name__ == '__main__':
# Launch the test
test()