I have searched both the web and stockexchange for an explanation of this strange behavior I experience when connecting to MQTT. But I couldn't find any similar case and I would like to understand where the issue comes from.
So, I have set up a Mosquitto MQTT broker on my Raspberry Pi to listen on port 1883. I have also set up port forwarding on my router, so I can reach the Pi from outside my home network (though I can reproduce the behavior below with a public server like broker.hivemq.com as well). When I execute the following Node.js script, I can connect to the broker and subscribe and publish messages, it works perfectly:
const mqtt = require('mqtt')
const client = mqtt.connect('tcp://my.address.net:1883')
client.on('connect', () => {
console.log('Connected!')
client.subscribe("chat")
})
var readline = require('readline');
var rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
terminal: false
});
rl.on('line', function(line){
client.publish('chat',line)
})
client.on('message', (topic, message) => {
console.log('>> '+message)
})
However, when I leave out the tcp:// protocol and have
const client = mqtt.connect('my.address.net:1883')
as the host address I don't get a connection, but also no error message. The program just hangs until I terminate it. I don't understand that. Isn't MQTT using TCP by default?
Does that have to do with my client or with my broker? Could it have something to do with my system (OSX)?
Maybe this has nothing to do with it, but a similar behavior I get when I use the Paho MQTT package in Python, and this is actually the more important case for me, because here I don't get it to run at all. This is my code:
import paho.mqtt.client as paho
def on_connect(client, userdata, flags, rc):
print("connected")
client.disconnect()
def on_disconnect(client, userdata, rc):
print("disconnected")
client = paho.Client()
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.connect("my.address.net", 1883)
Using this version without protocol, I don't get any response. The program runs for a second and then terminates. If I use the protocol
client.connect("tcp://my.address.net", 1883)
I get the error
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
I don't know what I am missing.
Can someone explain the difference of stating the tcp:// protocol in the address vs. leaving it out?
This is two question.
For the python part, the python client expects host and port separately as its connection arguments. It is not a URI. Hence no TCP//.
The python program does exactly what you ask - connect, then exit. You'll have to add more code beneath the connect call if you want it to do anything else. What that code is depends on what you want to do, but loop_forever() is a good start.
The other client expects a URI. I believe it uses TCP/SSL to distinguish between plain and encrypted connections.
The 2 libraries are requiring different things.
The NodeJS library is asking for a URI which includes
a schema ("tcp://")
a hostname/IP address ("local host")
a port number (":1883")
The host and port can easily be parsed from this.
Where as the Python library is explicitly asking for 2 separate things
the hostname
the port
These are taken as separate variables.
This just down to different authors choosing different approaches to gather information.
Related
I'm having a lot of difficulty with a very simple task. I'm attempting to set up a socket.io client in node js, which should then communicate with a local socket.io server setup in python (using the python bindings here. The issue I'm having is the server is detecting the client, but the client never seems to receive the 'connect' event. I suspect this is an issue with the way I've deployed the server asynchronously, but I'm really not sure. The code for each file is below:
server.py
import socketio
from aiohttp import web
HOST = '127.0.0.1'
PORT = 10001
# create a Socket.IO server
sio = socketio.AsyncServer(async_mode='aiohttp', logger=True, engineio_logger=True)
app = web.Application()
sio.attach(app)
#sio.on('connect')
def connect(sid, environ):
print('connect ', sid)
if __name__ == '__main__':
web.run_app(app, host=HOST, port=PORT)
client.js
const io = require('socket.io-client');
const HOST = '127.0.0.1';
const PORT = '10001';
const socket = io(`http://${HOST}:${PORT}`);
console.log('Socket instantiated!');
socket.on('connect', () => {
console.log(`socket connected: ${socket.connected}`);
});
The output I would expect is to see the server print out that the client has connected, and then for the client to print out that it has connected too. However, the client never seems to receive the 'connect' event, so never prints anything to the console.
Finally, an example of the server's output is:
Server initialized for aiohttp.
======== Running on http://127.0.0.1:10001 ========
(Press CTRL+C to quit)
1c17586e4c7e49b48abefea2fba460e6: Sending packet OPEN data {'sid': '1c17586e4c7e49b48abefea2fba460e6', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
connect 1c17586e4c7e49b48abefea2fba460e6
1c17586e4c7e49b48abefea2fba460e6: Sending packet MESSAGE data 0
While the client's output is annoyingly just
$ node mySocket.js
Socket instantiated!
and then it just hangs doing nothing.
I'm clearly misunderstanding something here, so thank you in advance!
Small update
I quickly tested using the python socketio client, and succesfully got an actual connection, so this should narrow it down to something I've done in the JS client.
Well, I ended up downgrading from socket.io-client 3.00 (did not see there was this major release 3 days ago) back to 2.3.1, and everything started working again! However, based on the lack of issues listed on Github, I'm guessing this is not a bug that is affecting everyone.
Suppose we have an echo server and a file transfer app.
There is a sender (Client) to send files and a receiver (Server) to receive files. The echo server will echo whatever received from the Client and the Server.
However, Client and Server cannot communicate directly, i.e., all packets have to go through the echo server. For example, the Client sends a UDP packet to the Echo server, and the Echo server echo that packet to Server and Server send Acknowledgment to the Echo server, and the Echo server echo that ack packet to the Client.
The objective is to implement a reliable UDP for file transfer. And we have only one UDP socket.
This figure demonstrates what the setup is
Client, Server and Echo Server
I have tried to use multi-thread and select.select and both do not work perfectly
The issue with multi-thread is that since Client and Server cannot communicate internally, and we have only one socket, it is difficult to choose who should send or receive now.
The issue with select.select is that the return list always has writeable non-empty, which makes the Client continues to send a bunch of packets before the readable is ready.
Here is the implementation for both Client and Server inside one file (say transceiver.py) what I do not use select.select (instead using send bool variable) but it seems to work fine. But I do believe this is bad practice, so I wonder what can I do to improve my design.
def create_transceiver(ip, port):
address = (ip, port)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.settimeout(1)
inout = [udp]
client = Client(udp, address)
server = Server(udp, address)
client_to_server = True
send = True
while True:
# infds, outfds, errfds = select.select(inout, inout, [])
if not send: # len(infds) != 0
if client_to_server:
server.start_recv()
client_to_server = False
else:
client.start_recv()
client_to_server = True
send = True
elif send: # len(outfds) != 0
if client_to_server:
if client.has_ack_all():
print(server.write_content())
break
client.start_send()
client_to_server = True
else:
server.start_send()
client_to_server = False
send = False
Here is the implementation of Echo Server:
import socket
ip = "10.10.1.100"
port = 8888
address = (ip, port)
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(address)
while True:
data, address = udp_socket.recvfrom(2048)
udp_socket.sendto(data, address)
And we have only one UDP socket.
As far as I understand this objective, you have a server process somewhere (you call it echo server) that is listening on a specific port. Also there is a client that wants to send data to some sort of server.
You provided code shows an implementation of what is called a state machine that (in your case) switches between receiving and sending.
However, Client and Server cannot communicate directly
The scenario you are describing makes your echo server a classic server that handles different types of clients. In your case this would be your "client" and your "server". I would like to call these just client-A and client-B. Most tutorials on the internet would call them Alice and Bob, I guess.
The objective is to implement a reliable UDP for file transfer.
So you want to transfer files between different clients using UDP as the base protocol.
UDP is not very well suited for this purpose. It does not garantee delivery of each packet transmitted. It is possible that packets arrive in different order than they were sent.
Usually you would use TCP for this kind of transmission. UDP is usually used for live streaming data like audio/video calls and stuff like that.
For more information on the differences between UDP and TCP you might check out the wikipedia pages for each:
https://en.wikipedia.org/wiki/User_Datagram_Protocol
https://en.wikipedia.org/wiki/Transmission_Control_Protocol
It is possible to use UDP for your transfers but you would have to implement all the safeties provided by TCP yourself.
I assume, your client and your server are actually different programs. Otherwise there would be a way they could communicate directly.
If that is the case, this tutorial might give you a starting point:
https://pythonprogramming.net/server-chatroom-sockets-tutorial-python-3/
In short, I have a Paho/MQTT sample set that works fine, but I'm having trouble detecting errors. Specifically, I'm not getting an on_connect callback and when an invalid UserID/Pswd combination is given, it fails silently.
In fact, by every indication, it all worked fine!
What am I doing wrong?
(snip)
def on_connect(client, userdata, flags, rc):
print("Connected with flags [%s] rtn code [%d]"% (flags, rc) )
def on_disconnect(client, userdata, rc):
print("disconnected with rtn code [%d]"% (rc) )
def on_publish(client, userdata, msgID):
print("Published with MsgID [%d]"% (msgID) )
mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_disconnect = on_disconnect
mqttc.on_publish = on_publish
mqttc.username_pw_set(Q_USER, Q_PSWD)
rc=mqttc.connect(Q_BROKER, Q_PORT)
print "Return="+str(rc)
mqttc.loop_start()
rc=mqttc.publish(Q_TOPIC, "Hello, World!")
print "Return="+str(rc)
mqttc.disconnect()
mqttc.loop_stop()
Output when UserID or Pswd is deliberately wrong:
Return=0
Published with MsgID [1]
Return=(0, 1)
disconnected with rtn code [0]
Turns out that it was a few different issues.
First, Connect returns without truly being connected. Adding a wait loop with a Sleep until the Connect callback is received is crucial. Only the callback tells you the Connect succeeded or failed.
Second, once I saw the return code, I saw mine was failing on Protocol Version Mismatch. The Mosquitto version my Pi was pulling down was REALLY old. Found a post that pointed my system to a newer version. Solved the version mismatch problem.
Third, adding the "loop_start()" before Connect does not work. The intuitive place is not the right place.
Fourth, one needs to add Sleep() calls before Disconnect, or you will not see all the Callbacks.
Bottom line, while the docs show how to use the APIs, they don't show you how to write a robust program for production use.
I'm still still looking for ways to know when all in-flight publish calls have cleared, rather than using a blind Sleep call, but my original issue is resolved.
When you send mqtt CONNECT packet, you should receive CONNACK response. This response contains the following codes
0 - success, connection accepted
1 - connection refused, bad protocol
2 - refused, client-id error
3 - refused, service unavailable
4 - refused, bad username or password
5 - refused, not authorized
As you can see, your response should be 4. But it is zero. It might be that you broker doesn't check credentials so your connect message is accepted. Client looks fine.
Try to put the publish into a while loop like this
while True:
mqttc.loop()
rc=mqttc.publish(Q_TOPIC, "Hello, World!")
print("Return=" + str(rc))
if sth:
break
mqttc.disconnect()
In this way, the callback function may be executed before the program ends.
So i'm trying to write a TCP server and client so that when the client connects, a file is sent back from the server. Here's my code for the server:
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9999
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(10)
file_to_send = ('file_to_send.txt')
print '[*] Listening on %s:%d' % (bind_ip,bind_port)
def handle_client(client_socket):
request = client_socket.recv(1024)
print '[*] Received %s' % request
client_socket.send('')
client_socket.close(file_to_send)
while True:
client,addr = server.accept()
print '[*] Accepted connection from: %s:%d' % (addr[0],addr[1])
client_handler = threading.Thread(target=handle_client,args=(client,))
client_handler.start()
And here is my code for the client:
import socket
target_host = '0.0.0.0'
target_port = 9999
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((target_host,target_port))
client.send('ACK!')
response = client.recv(4096)
print response
When the server and client are run, the server returns the error 'Only one usage of each socket address (protocol/network address/port) is normally permitted' and when the client is run I get the error ' The requested address is not valid in its context'
Does anyone know why these errors are occurring or how I might be able to fix it.
I think this is a case of inaccurate example code. The server code you posted does not cause the issue you're describing (the client code does, but we'll get to that).
Server
The issue with the server code is that you're binding to the same (address, port) twice. Whether this is from wrong indentation or wrong logic it's tough to say, but that error message comes from binding the same protocol to the same address with the same port number more than once at the same time.
The rest of the code seems fine, though you can't do client_socket.close("some string") as you're doing here. socket.close does not accept any arguments.
Client
There's a simple solution here -- just change the target_host to something reasonable. You cannot connect to 0.0.0.0. This should be the addressable IP of the server in the smallest scope possible. Presumably if this is a toy program this is something like localhost.
(N.B. you bind the server to '0.0.0.0' to tell it to accept connections going to any destination IP. You could bind the server instead to '127.0.0.1' to inform the server that it will ONLY be known as localhost and never anything else.)
I've had a similar issue before and it was that I was running old versions of the program on the same port, restarting the PC or closing the processes in task manager should fix it.
I'm aware that this was asked over a year ago, so OP has probably restarted their PC since then, but hopefully this will help someone looking for a solution for a similar problem.
i am relatively new to python, so please be considerate...
i'm implementing a server and a client via raw_sockets.
i have the necessary privileges.
now, the server i defined so:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockSer = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
sockSer.bind(address)
sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
packet, addr = sockSer .recvfrom(4096) # wait for packet from client
Q1) why can't i simply type: hosts = 'localhost'.
if i do so, it doesn't allow me to write the line: sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON). and then the server doesn't receive my client's messages.
only when doing gethostbyname(socket.gethostname()) i get 192.168.1.101
and then it works.
in a different class:
the client socket:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockCli = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
Q2) do i also need to type: sockCli.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
or maybe sockCli.connect(address)? seems that it works without the connect command.
for the client socket?
now, the problems arise when i do the following:
1) send a packet from client to server:
header=...
payload='a'
sockCli.sendto(header + payload, address)
2) receive packet in server and send something back to client:
while(true):
data, addr = sockSer.recvfrom(4096)
header2=...
payload2='b'
sockSer.sendto(header2 + payload2, addr)
now, my important question is:
Q3) the server sent only 1 packet to client, with payload 'b'.
what happens is, my client actually receives 2 packets in the while loop:
first packet is what the client itself sent to server, and the other packet is what the client got from the server.
hence my output is 'ab' instead of simply 'b'
why is this happening???
NOTE: i didn't type the entire code, but i think my syntax,parsing,header composition etc.. are correct.
is there an obvious problem in my code?
if necessary i'll upload the entire code.
thanks
I got this too.
my solution is add a judge in the receive code,such as if I send Ping package so I only want ECHO　Reply( type 0 code 0), I write
if type != 0:
continue
and you also can write as
if addr == my_ip:
continue
It seems not has any smooth solution
Q1: I was able to bind to localhost and call IOCTL with both parameters just fine. Assuming your client is also running on the same system, ensure the client is sending to "localhost", otherwise your server will never receive the packets. If your client is on another system, obviously your server will never receive the packets.
Q2: You do not need IOCTL for sending the packet. Just send it via sendto().
Q3: The reason you're seeing two replies is, the kernel is also processing the echo request, in addition to your own user-space code.
Although you can use ICMP for arbitrary message passing, as someone else pointed out this isn't its intended design. You may find that your data portion is truncated out in message replies. For example, when sending echo requests, your reply likely will contain everything you sent; however, a reply that is type 3 code 3 may not include your data, but only the first 8 bytes of the ICMP header.