For a project I need to communicate between C++ and Python via ZMQ over the WLAN network.
If I use my C++ implementation, everything works fine. I just type in the IP+Port number at the client.bind("tcp:// ...) and I can send messages via WLAN.
If I try the same with the Python Code, it does not work.
So I just tested the python examples (so no C++ anymore): http://zguide.zeromq.org/py:durapub
http://zguide.zeromq.org/py:durasub
I replaced the >localhost< in the client with the IP of my host computer. I do not receive any messages. I am using exactly the code from the example, except the replacement.
Here is the Code:
PUBLISHER:
import zmq
import time
context = zmq.Context()
# Subscriber tells us when it's ready here
sync = context.socket(zmq.PULL)
sync.bind("tcp://*:5564")
# We send updates via this socket
publisher = context.socket(zmq.PUB)
publisher.bind("tcp://*:5565")
# Wait for synchronization request
sync_request = sync.recv()
# Now broadcast exactly 10 updates with pause
for n in xrange(10):
msg = "Update %d" % n
publisher.send(msg)
time.sleep(1)
publisher.send("END")
time.sleep(1) # Give 0MQ/2.0.x time to flush output
SUBSCRIBER
import zmq
import time
context = zmq.Context()
# Connect our subscriber socket
subscriber = context.socket(zmq.SUB)
subscriber.setsockopt(zmq.IDENTITY, "Hello")
subscriber.setsockopt(zmq.SUBSCRIBE, "")
subscriber.connect("tcp://192.168.2.119:5565")
# Syncronize with the publisher
sync = context.socket(zmq.PUSH)
sync.connect("tcp://192.168.2.119:5564")
sync.send("")
# Get updates, expect random Ctrl-C death
while True:
data = subscriber.recv()
print data
if data == "END":
break
Its exactly the example code, except that I changed localhost to the IP Adress of my publisher in the Subscriber-Code. Btw, I did the same in the C++ example Code and it works.
Related
I have a process A that publishes a message constantly and processes B and C subscribe to the topic and get the latest message published by the publisher in process A.
So, I set zmq.CONFLATE to both publisher and subscriber. However, I found that one subscriber was not able to receive messages.
def publisher(sleep_time=1.0, port="5556"):
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.bind("tcp://*:%s" % port)
print ("Running publisher on port: ", port)
while True:
localtime = time.asctime( time.localtime(time.time()))
string = "Message published time: {}".format(localtime)
socket.send_string("{}".format(string))
time.sleep(sleep_time)
def subscriber(name="sub", sleep_time=1, ports="5556"):
print ("Subscriber Name: {}, Sleep Time: {}, Port: {}".format(name, sleep_time, ports))
context = zmq.Context()
print ("Connecting to publisher with ports %s" % ports)
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.setsockopt_string(zmq.SUBSCRIBE, "")
socket.connect ("tcp://localhost:%s" % ports)
while True:
message = socket.recv()
localtime = time.asctime( time.localtime(time.time()))
print ("\nSubscriber [{}]\n[RECV]: {} at [TIME]: {}".format(name, message, localtime))
time.sleep(sleep_time)
if __name__ == "__main__":
Process(target=publisher).start()
Process(target=subscriber, args=("SUB1", 1.2, )).start()
Process(target=subscriber, args=("SUB2", 1.1, )).start()
I tried to unset the socket.setsockopt(zmq.CONFLATE, 1) in the publisher, and that seemed to solve the problem. Both subscribers in processes B and C could receive messages and the messages seemed to be the latest ones.
I'm trying to find out why setting the publisher with CONFLATE caused the problem I had. I could not find information about it. Does anyone know what causes this behavior?
Also, I want to know, in the situation of one publisher to multiple subscribers, what is the correct code setup, so that subscriber can always get the latest messages?
It's most likely a timing issue, the ZMQ_CONFLATE socket option limits the inbound and outbound queue to 1 message.
The way PUB/SUB works is the subscriber sends a subscription message to the publisher when you set the ZMQ_SUBSCRIBE option. If you start both subscribers at the same time then its possible that one of the subscription messages that arrived on the publisher queue will be discarded.
Try adding a sleep between the starting each subscriber.
From the zeromq docs
If set, a socket shall keep only one message in its inbound/outbound
queue, this message being the last message received/the last message
to be sent. Ignores ZMQ_RCVHWM and ZMQ_SNDHWM options. Does not
support multi-part messages, in particular, only one part of it is
kept in the socket internal queue.
I am not saying this is the solution to you problem, but if that is the case we may need to post a change to libzmq to make the conflate options more granular so you can choose if conflate should be applied to inbound or outbound queues.
There is a manner to get "Last message only" option in ZMQ Subscribe socket (using CONFLATE option).
You need it on the subscriber side.
Here is an example:
import zmq
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, '')
socket.setsockopt(zmq.CONFLATE, 1) # last msg only.
socket.connect("tcp://localhost:%s" % port) # must be placed after above options.
while True:
data = socket.recv()
print data
On the other word, I removed any buffered queue in subscriber code.
[In Additional]:
With the zmq.SNDBUF and zmq.RCVBUF options we could set a limit on ZMQ buffer size. (More complete and an example)
I am trying to publish a message(it's like broadcast when using raw sockets) to my subnet with a known port but at subscriber end, the message is not received. The idea is the IP address of the first machine should not be known to the second machine that's why I am using broadcast IP. With UDP or TCP raw socket, it works but I am trying to learn pub-sub pattern not sure how to incorporate that idea.
This is my codes:
Publisher:
import zmq
import sys
import time
context=zmq.Context()
socket=context.socket(zmq.PUB)
socket.bind("tcp://192.168.1.255:5677")
while True:
data='hello'.encode()
socket.send(data)
#time.sleep(1)
Subscriber:
context=zmq.Context()
sub=context.socket(zmq.PUB)
sub.setsocketopt(zmq.SUBSCRIBE, "".encode())
sub.connect('tcp://192.168.1.255:5677')
sub.recv()
print(sub.recv())
In terms of raw UDP, I wrote a code which works perfectly.
broadcast:
def broadcast(Host,port):
#send bd
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
msg=get_ip_data("wlp3s0")
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
time.sleep(1.5)
# print("yes sending", client)
sock.sendto(msg.encode(), (Host,port))
recv:
def broadcast_recv():
#listen bd
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind((get_bd_address("wlp1s0"),12345))
# receive broadcast
msg, client = sock.recvfrom(1024)
a=(msg.decode())
print(a)
It seems you forgot the zmq.SUB in the subscriber side. Also you used sub.setsocketopt() instead of sub.setsockopt().
Try it:
Publisher:
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5677") # Note.
while True:
socket.send_string('hello')
time.sleep(1)
Subscriber:
context = zmq.Context()
sub=context.socket(zmq.SUB) # Note.
sub.setsockopt(zmq.SUBSCRIBE, b"") # Note.
sub.connect('tcp://192.168.1.255:5677')
while True:
print(sub.recv())
[NOTE]:
You can also change the .bind() and .connect() in subscriber and publisher with your policy. (This post is relevant).
Make sure that 5677 is open in the firewall.
socket.bind("tcp://*:5677") or socket.bind("tcp://0.0.0.0:5677") is broadcasting trick.
I think the problem is that the SUB socket cannot register itself with the PUB socket. Even though in-concept the data only goes from PUB to SUB, in reality, there are also control messages (e.g. subscription topics), being sent back to the PUB.
If your netmask is 255.255.255.0, this will probably not work as expected.
I'm using ZeroMQ to establish a publisher/subscriber communication model.
The publisher creates a zmq context and then opens a socket with the PUB
communication pattern. It then binds to a port, as the TCP transport protocol is used. For the synchronization a separate socket is opened with the REP communication pattern that binds in a different path. Unless a synchronization request in msg = syncservice.recv() is received, the program cannot continue. It then does some rudimentary work and starts over again. Here's the code for the publisher:
import pickle, zmq, random, string
# Wait for 1 subscriber
SUBSCRIBERS_EXPECTED = 1
def randomword(length):
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(length))
while True:
try:
arguments = {}
data = {}
context = zmq.Context()
# Socket to talk to clients
publisher = context.socket(zmq.PUB)
# set SNDHWM, in case of slow subscribers
publisher.sndhwm = 1100000
publisher.bind('tcp://*:5561')
# Socket to receive signals
syncservice = context.socket(zmq.REP)
syncservice.bind('tcp://*:5562')
# Get synchronization from subscribers
subscribers = 0
while subscribers < SUBSCRIBERS_EXPECTED:
# wait for synchronization request
msg = syncservice.recv()
# send synchronization reply
syncservice.send(b'')
subscribers += 1
for n in range(1000):
for i in range(random.randrange(1, 6)):
arguments[i] = randomword(random.randrange(2, 10))
data['func_name_' + str(n)] = randomword(8)
data['arguments_' + str(n)] = arguments
data_string = pickle.dumps(data)
publisher.send(data_string)
except KeyboardInterrupt:
print("Interrupt received, stopping...")
break
The subscriber functions pretty much the same way the publisher does, albeit
from a subscriber perspective. Here's the code for the subscriber:
import pickle, zmq, pprint, time
context = zmq.Context()
# Connect the subscriber socket
subscriber = context.socket(zmq.SUB)
subscriber.connect('tcp://localhost:5561')
subscriber.setsockopt(zmq.SUBSCRIBE, b'')
time.sleep(1)
# Synchronize with publisher
syncclient = context.socket(zmq.REQ)
syncclient.connect('tcp://localhost:5562')
# Initialize poll set
poller = zmq.Poller()
poller.register(syncclient, zmq.POLLIN)
poller.register(subscriber, zmq.POLLIN)
# send a synchronization request
syncclient.send(b'')
while True:
try:
socks = dict(poller.poll())
except KeyboardInterrupt:
print("Interrupt received, stopping...")
break
# wait for synchronization reply
if syncclient in socks:
syncclient.recv()
print('Sync')
if subscriber in socks:
msg = subscriber.recv()
data = pickle.loads(msg)
pprint.pprint(data)
syncclient.send(b'')
The desired result would be for the publisher to publish endlessly, while the
subscriber continually receives and prints everything. If I remove the
synchronization part, everything runs as expected. If I keep the synchronization
part the subscriber hangs after a number of transmissions. The interesting thing
is that if I send a keyboard interrupt (Ctrl-C) and then restart the subscriber,
it will receive a couple of transmissions again and hang again and so on and so
forth.
I have tried different high-watermark settings, but it didn't make any difference. I have tried closing the sockets and terminating the context after every loop. I've tested if the overhead from printing or pickling (serializing) was too much, but it wasn't it either. I have also modified the suicidal snail example to work in this case, but the subscriber didn't die. What am I missing? (Python 3 is used for every example)
I'm trying to subscribe to a pub-sub server.
When doing that using Python there is no problem, it works like expected.
But when I'm trying to subscribe to the exact same server with NodeJS ZMQ nothing happens.
I can't figure out where something goes wrong, probably at the subscribe-part?
Python:
from gzip import GzipFile
from cStringIO import StringIO
import zmq
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:6701")
socket.setsockopt(zmq.SUBSCRIBE, '')
while True:
multipart = socket.recv_multipart()
address = multipart[0]
contents = ''.join(multipart[1:])
contents = GzipFile('','r',0,StringIO(contents)).read()
print("[%s] %s\n" % (address, contents))
socket.close()
context.term()
NodeJS:
var zmq = require('zmq')
, sock = zmq.socket('sub');
sock.connect('tcp://127.0.0.1:6701');
sock.subscribe('');
console.log('Subscriber connected to port 6701');
sock.on('message', function(topic, message) {
console.log('received a message related to:', topic, 'containing message:', message);
});
The on-message part in the NodeJS example never gets fired.
When I run a simple NodeJS publisher the subscriber works like expected.
Note that the address I connect to is a local IP due to the fact that I run some local distribution tool for the ZeroMQ messages.
What can be the difference between the two scripts that the NodeJS script does not behave like the Python script on the same publishing source?
Both are using ZeroMQ 4.
Edit:
As suggested by Jason I would post some the code of the publisher. But because this is a 3rd party I don't have the code of that. But when I fire up a simple Python publisher the subscribers act the same like they do on the 3rd party publisher.
Simple Python publisher:
import zmq
import random
import sys
import time
port = "6701"
if len(sys.argv) > 1:
port = sys.argv[1]
int(port)
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:%s" % port)
while True:
topic = random.randrange(9999,10005)
messagedata = random.randrange(1,215) - 80
print "%d %d" % (topic, messagedata)
socket.send("%d %d" % (topic, messagedata))
time.sleep(1)
NodeJS publisher:
var zmq = require('zmq')
, sock = zmq.socket('pub');
sock.bindSync('tcp://127.0.0.1:6701');
console.log('Publisher bound to port 3000');
setInterval(function(){
console.log('sending a multipart message envelope');
sock.send(['kitty cats', 'meow!']);
}, 500);
The NodeJS publisher works with both the Python and the NodeJS subscriber but the Python publisher works only with the Python subscriber and not the NodeJS one.
Fixed.
Removing node_modules and reinstaling it by npm install zmq.
Here are the "sender" and "receiver" I am using.
sender.py
import time
import zmq
context = zmq.Context()
sender = context.socket(zmq.PUSH)
sender.connect("tcp://127.0.0.1:5557")
c = 0
while True:
sender.send_json(c)
print "Sent {}".format(c)
c += 1
time.sleep(1)
reciever.py
import zmq
context = zmq.Context()
receiver = context.socket(zmq.PULL)
receiver.connect("tcp://127.0.0.1:5557")
while True:
m = receiver.recv_json()
print m
When I start the receiver then start the sender nothing is written to stdout on the receiver's cmd (I am trying to get this to work on windows 7). I've added inbound and outbound exceptions to the firewall for port 5557 and running netstat -an doesn't show anything running on port 5557. Lastly I am using zmq 2.2.0.
In your sender.py you should write sender.bind, not sender.connect.