I am trying to use broadcasting packets to subnet address, I successfully tried to do that using socket's broadcast option, But I recently started learning ZeroMQ so I would like to use it to broadcast the packets to the subnet. I used zmq.PUB, zmq.SUB but at the subscriber side, the packets are undelivered because I use subnet address. If I use IP address of the machine then it works, but that's not what I want.
Is there any option for broadcasting using ZMQ?
Here is the code I tried so far:
Publisher:
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://192.X.X.255:9999") # Note.
while True:
socket.send_string('hello')
time.sleep(1)
Subscriber:
context = zmq.Context()
sub=context.socket(zmq.SUB) # Note.
sub.setsockopt(zmq.SUBSCRIBE, b"") # Note.
sub.connect('tcp://192.x.x.255:9999') -> publishing only to subnet
while True:
print(sub.recv())
We can do the broadcasting of packets using a regular socket, for example, using:
sock.setsockopt(socket.SOL_SOCKET,socket.SO_BROADCAST,1)
But I want to replace this in a way I do it with ZMQ. Does the ZMQ really have a broadcast discovery in a different way or we should use the same code above as we do for regular broadcasting?
Suppose you have three Machines (M1, M2, M3) with three different IP addresses with the same subnet and a defined port. We want to publish a message (from M1) to each subscribers (M1, M2), therefore we would have the following code snippet:
Publisher (Machine1):
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:9999") # Note.
while True:
socket.send_string('hello-all')
time.sleep(1)
Subscriber (Machine2):
context = zmq.Context()
sub=context.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, b"")
sub.connect('tcp://Machine1_IP:9999') # Note
while True:
print(sub.recv())
Subscriber (Machine3):
context = zmq.Context()
sub=context.socket(zmq.SUB)
sub.setsockopt(zmq.SUBSCRIBE, b"")
sub.connect('tcp://Machine1_IP:9999') # Note
while True:
print(sub.recv())
Related
I have a process A that publishes a message constantly and processes B and C subscribe to the topic and get the latest message published by the publisher in process A.
So, I set zmq.CONFLATE to both publisher and subscriber. However, I found that one subscriber was not able to receive messages.
def publisher(sleep_time=1.0, port="5556"):
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.bind("tcp://*:%s" % port)
print ("Running publisher on port: ", port)
while True:
localtime = time.asctime( time.localtime(time.time()))
string = "Message published time: {}".format(localtime)
socket.send_string("{}".format(string))
time.sleep(sleep_time)
def subscriber(name="sub", sleep_time=1, ports="5556"):
print ("Subscriber Name: {}, Sleep Time: {}, Port: {}".format(name, sleep_time, ports))
context = zmq.Context()
print ("Connecting to publisher with ports %s" % ports)
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.setsockopt_string(zmq.SUBSCRIBE, "")
socket.connect ("tcp://localhost:%s" % ports)
while True:
message = socket.recv()
localtime = time.asctime( time.localtime(time.time()))
print ("\nSubscriber [{}]\n[RECV]: {} at [TIME]: {}".format(name, message, localtime))
time.sleep(sleep_time)
if __name__ == "__main__":
Process(target=publisher).start()
Process(target=subscriber, args=("SUB1", 1.2, )).start()
Process(target=subscriber, args=("SUB2", 1.1, )).start()
I tried to unset the socket.setsockopt(zmq.CONFLATE, 1) in the publisher, and that seemed to solve the problem. Both subscribers in processes B and C could receive messages and the messages seemed to be the latest ones.
I'm trying to find out why setting the publisher with CONFLATE caused the problem I had. I could not find information about it. Does anyone know what causes this behavior?
Also, I want to know, in the situation of one publisher to multiple subscribers, what is the correct code setup, so that subscriber can always get the latest messages?
It's most likely a timing issue, the ZMQ_CONFLATE socket option limits the inbound and outbound queue to 1 message.
The way PUB/SUB works is the subscriber sends a subscription message to the publisher when you set the ZMQ_SUBSCRIBE option. If you start both subscribers at the same time then its possible that one of the subscription messages that arrived on the publisher queue will be discarded.
Try adding a sleep between the starting each subscriber.
From the zeromq docs
If set, a socket shall keep only one message in its inbound/outbound
queue, this message being the last message received/the last message
to be sent. Ignores ZMQ_RCVHWM and ZMQ_SNDHWM options. Does not
support multi-part messages, in particular, only one part of it is
kept in the socket internal queue.
I am not saying this is the solution to you problem, but if that is the case we may need to post a change to libzmq to make the conflate options more granular so you can choose if conflate should be applied to inbound or outbound queues.
There is a manner to get "Last message only" option in ZMQ Subscribe socket (using CONFLATE option).
You need it on the subscriber side.
Here is an example:
import zmq
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, '')
socket.setsockopt(zmq.CONFLATE, 1) # last msg only.
socket.connect("tcp://localhost:%s" % port) # must be placed after above options.
while True:
data = socket.recv()
print data
On the other word, I removed any buffered queue in subscriber code.
[In Additional]:
With the zmq.SNDBUF and zmq.RCVBUF options we could set a limit on ZMQ buffer size. (More complete and an example)
For a project I need to communicate between C++ and Python via ZMQ over the WLAN network.
If I use my C++ implementation, everything works fine. I just type in the IP+Port number at the client.bind("tcp:// ...) and I can send messages via WLAN.
If I try the same with the Python Code, it does not work.
So I just tested the python examples (so no C++ anymore): http://zguide.zeromq.org/py:durapub
http://zguide.zeromq.org/py:durasub
I replaced the >localhost< in the client with the IP of my host computer. I do not receive any messages. I am using exactly the code from the example, except the replacement.
Here is the Code:
PUBLISHER:
import zmq
import time
context = zmq.Context()
# Subscriber tells us when it's ready here
sync = context.socket(zmq.PULL)
sync.bind("tcp://*:5564")
# We send updates via this socket
publisher = context.socket(zmq.PUB)
publisher.bind("tcp://*:5565")
# Wait for synchronization request
sync_request = sync.recv()
# Now broadcast exactly 10 updates with pause
for n in xrange(10):
msg = "Update %d" % n
publisher.send(msg)
time.sleep(1)
publisher.send("END")
time.sleep(1) # Give 0MQ/2.0.x time to flush output
SUBSCRIBER
import zmq
import time
context = zmq.Context()
# Connect our subscriber socket
subscriber = context.socket(zmq.SUB)
subscriber.setsockopt(zmq.IDENTITY, "Hello")
subscriber.setsockopt(zmq.SUBSCRIBE, "")
subscriber.connect("tcp://192.168.2.119:5565")
# Syncronize with the publisher
sync = context.socket(zmq.PUSH)
sync.connect("tcp://192.168.2.119:5564")
sync.send("")
# Get updates, expect random Ctrl-C death
while True:
data = subscriber.recv()
print data
if data == "END":
break
Its exactly the example code, except that I changed localhost to the IP Adress of my publisher in the Subscriber-Code. Btw, I did the same in the C++ example Code and it works.
I am trying to publish a message(it's like broadcast when using raw sockets) to my subnet with a known port but at subscriber end, the message is not received. The idea is the IP address of the first machine should not be known to the second machine that's why I am using broadcast IP. With UDP or TCP raw socket, it works but I am trying to learn pub-sub pattern not sure how to incorporate that idea.
This is my codes:
Publisher:
import zmq
import sys
import time
context=zmq.Context()
socket=context.socket(zmq.PUB)
socket.bind("tcp://192.168.1.255:5677")
while True:
data='hello'.encode()
socket.send(data)
#time.sleep(1)
Subscriber:
context=zmq.Context()
sub=context.socket(zmq.PUB)
sub.setsocketopt(zmq.SUBSCRIBE, "".encode())
sub.connect('tcp://192.168.1.255:5677')
sub.recv()
print(sub.recv())
In terms of raw UDP, I wrote a code which works perfectly.
broadcast:
def broadcast(Host,port):
#send bd
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
msg=get_ip_data("wlp3s0")
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
time.sleep(1.5)
# print("yes sending", client)
sock.sendto(msg.encode(), (Host,port))
recv:
def broadcast_recv():
#listen bd
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind((get_bd_address("wlp1s0"),12345))
# receive broadcast
msg, client = sock.recvfrom(1024)
a=(msg.decode())
print(a)
It seems you forgot the zmq.SUB in the subscriber side. Also you used sub.setsocketopt() instead of sub.setsockopt().
Try it:
Publisher:
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5677") # Note.
while True:
socket.send_string('hello')
time.sleep(1)
Subscriber:
context = zmq.Context()
sub=context.socket(zmq.SUB) # Note.
sub.setsockopt(zmq.SUBSCRIBE, b"") # Note.
sub.connect('tcp://192.168.1.255:5677')
while True:
print(sub.recv())
[NOTE]:
You can also change the .bind() and .connect() in subscriber and publisher with your policy. (This post is relevant).
Make sure that 5677 is open in the firewall.
socket.bind("tcp://*:5677") or socket.bind("tcp://0.0.0.0:5677") is broadcasting trick.
I think the problem is that the SUB socket cannot register itself with the PUB socket. Even though in-concept the data only goes from PUB to SUB, in reality, there are also control messages (e.g. subscription topics), being sent back to the PUB.
If your netmask is 255.255.255.0, this will probably not work as expected.
I'm using ZeroMQ to establish a publisher/subscriber communication model.
The publisher creates a zmq context and then opens a socket with the PUB
communication pattern. It then binds to a port, as the TCP transport protocol is used. For the synchronization a separate socket is opened with the REP communication pattern that binds in a different path. Unless a synchronization request in msg = syncservice.recv() is received, the program cannot continue. It then does some rudimentary work and starts over again. Here's the code for the publisher:
import pickle, zmq, random, string
# Wait for 1 subscriber
SUBSCRIBERS_EXPECTED = 1
def randomword(length):
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(length))
while True:
try:
arguments = {}
data = {}
context = zmq.Context()
# Socket to talk to clients
publisher = context.socket(zmq.PUB)
# set SNDHWM, in case of slow subscribers
publisher.sndhwm = 1100000
publisher.bind('tcp://*:5561')
# Socket to receive signals
syncservice = context.socket(zmq.REP)
syncservice.bind('tcp://*:5562')
# Get synchronization from subscribers
subscribers = 0
while subscribers < SUBSCRIBERS_EXPECTED:
# wait for synchronization request
msg = syncservice.recv()
# send synchronization reply
syncservice.send(b'')
subscribers += 1
for n in range(1000):
for i in range(random.randrange(1, 6)):
arguments[i] = randomword(random.randrange(2, 10))
data['func_name_' + str(n)] = randomword(8)
data['arguments_' + str(n)] = arguments
data_string = pickle.dumps(data)
publisher.send(data_string)
except KeyboardInterrupt:
print("Interrupt received, stopping...")
break
The subscriber functions pretty much the same way the publisher does, albeit
from a subscriber perspective. Here's the code for the subscriber:
import pickle, zmq, pprint, time
context = zmq.Context()
# Connect the subscriber socket
subscriber = context.socket(zmq.SUB)
subscriber.connect('tcp://localhost:5561')
subscriber.setsockopt(zmq.SUBSCRIBE, b'')
time.sleep(1)
# Synchronize with publisher
syncclient = context.socket(zmq.REQ)
syncclient.connect('tcp://localhost:5562')
# Initialize poll set
poller = zmq.Poller()
poller.register(syncclient, zmq.POLLIN)
poller.register(subscriber, zmq.POLLIN)
# send a synchronization request
syncclient.send(b'')
while True:
try:
socks = dict(poller.poll())
except KeyboardInterrupt:
print("Interrupt received, stopping...")
break
# wait for synchronization reply
if syncclient in socks:
syncclient.recv()
print('Sync')
if subscriber in socks:
msg = subscriber.recv()
data = pickle.loads(msg)
pprint.pprint(data)
syncclient.send(b'')
The desired result would be for the publisher to publish endlessly, while the
subscriber continually receives and prints everything. If I remove the
synchronization part, everything runs as expected. If I keep the synchronization
part the subscriber hangs after a number of transmissions. The interesting thing
is that if I send a keyboard interrupt (Ctrl-C) and then restart the subscriber,
it will receive a couple of transmissions again and hang again and so on and so
forth.
I have tried different high-watermark settings, but it didn't make any difference. I have tried closing the sockets and terminating the context after every loop. I've tested if the overhead from printing or pickling (serializing) was too much, but it wasn't it either. I have also modified the suicidal snail example to work in this case, but the subscriber didn't die. What am I missing? (Python 3 is used for every example)
Here are the "sender" and "receiver" I am using.
sender.py
import time
import zmq
context = zmq.Context()
sender = context.socket(zmq.PUSH)
sender.connect("tcp://127.0.0.1:5557")
c = 0
while True:
sender.send_json(c)
print "Sent {}".format(c)
c += 1
time.sleep(1)
reciever.py
import zmq
context = zmq.Context()
receiver = context.socket(zmq.PULL)
receiver.connect("tcp://127.0.0.1:5557")
while True:
m = receiver.recv_json()
print m
When I start the receiver then start the sender nothing is written to stdout on the receiver's cmd (I am trying to get this to work on windows 7). I've added inbound and outbound exceptions to the firewall for port 5557 and running netstat -an doesn't show anything running on port 5557. Lastly I am using zmq 2.2.0.
In your sender.py you should write sender.bind, not sender.connect.