I am working on a program using TCP protocol to collect ADS-B messages from an antenna. Since I am new to Python, I used the following scripts to establish connection. The problem is that I receive several messages at the same time (since TCP is stream-oriented). I would like to separate each message using a "\n" delimiter for instance (each message has "#" at the beginning and ";" at the end and the length varies). I have no idea of how to tell Python to separate each message like this, do you have any idea ? Thanks a lot
Python version 3.7.6, Anaconda, Windows 10
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
# The request handler class for our server.
# It is instantiated once per connection to the server, and must
# override the handle() method to implement communication to the
# client.
# """
def handle(self):
# self.rfile is a file-like object created by the handler;
# we can now use e.g. readline() instead of raw recv() calls
self.data = self.rfile.readline().strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# Likewise, self.wfile is a file-like object used to write back
# to the client
self.wfile.write(self.data.upper())
if __name__ == "__main__":
print ("Server online")
HOST, PORT = "localhost", 10100
# Create the server, binding to localhost on port 10002
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
import socket
import sys
def tcp_client():
HOST, PORT = "192.168.2.99", 10002
data = " ".join(sys.argv[1:])
# Create a socket (SOCK_STREAM means a TCP socket)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
# Connect to server and send data
sock.connect((HOST, PORT))
while True :
sock.sendall(bytes(data + "\n", "utf-8"))
# Receive data from the server
received = str(sock.recv(1024), "utf-8")
print("{}".format(received))
You could try using:
acumulator = ""
while True:
received = str(sock.recv(1024), "utf-8")
divided_message = received.split('\n')
if len(divided_message) >= 2:
print('One mesage: ', acumulator + divided_message[0].strip())
for i in range(1, len(divided_message) - 1):
print('One mesage: ', divided_message[i].strip())
if '\n' in divided_message[-1]:
print('One mesage: ', divided_message[-1].strip())
acumulator = ''
else:
acumulator = divided_message[-1]
else:
acumulator += divided_message[0]
If the message is separated by /n you can divide the message applying a selection technique, like the one presented above. If your messages have a fixed length you could just modify your delimiter.
I would suggest you use the following approach:
Assuming your messages can't be more than 4GB long, just send the
length, packed into exactly 4 bytes, and then you send the data
itself. So, the other side always knows how much to read: Read exactly
4 bytes, unpack it into a length, then read exactly as many bytes as
that:
def send_one_message(sock, data):
length = len(data)
sock.sendall(struct.pack('!I', length))
sock.sendall(data)
def recv_one_message(sock):
lengthbuf = recvall(sock, 4)
length, = struct.unpack('!I', lengthbuf)
return recvall(sock, length)
That's almost a complete protocol. The only problem is that Python
doesn't have a recvall counterpart to sendall, but you can write it
yourself:
def recvall(sock, count):
buf = b''
while count:
newbuf = sock.recv(count)
if not newbuf: return None
buf += newbuf
count -= len(newbuf)
return buf
More detailed description here
Related
I am capturing screenshots from the server, then sending it to the client, but the images get all sent as one big file to the client that keeps expanding in size. This only happens when i send from one machine to another (I am working on a local netwrok) but when running both client and server from my machine they work fine.
Note: for the client on the other machine, I packaged it into an exe using pyinstaller, since this machine does not have python.
server code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
data = f.read(4096)
while data:
q.send(data)
data = f.read(4096)
q.send(b"full")
i += 1
time.sleep(0.3)
client code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
i = 0
while True:
with open(str(i) + "s.jpg", "wb") as f:
recv_data = s.recv(4096)
while recv_data:
f.write(recv_data)
recv_data = s.recv(4096)
if(recv_data == b"full"):
break
i += 1
There various wrong assumptions here which lead to the problem you see. The wrong assumptions are:
that send(data) will write all data It might send less. You need to check the return value or use sendall to be sure.
that a single send in the sender is matched by exactly a single recv in the recipientTCP is only an unstructured byte stream. send does not add message semantics, so a single send might lead to multiple recv, multiple send might lead to a single recv etc. Specifically send("data") followed by send("full") might be recv(4096) as "datafull", thus missing your code to detect end of image.
As for why does it work on the local machine but not on the remote - the chance in the latter case is higher that send get combined together and recv as one.
As stated by Steffen Ulrich you should use sendall for sending and for receiving we create a specialized function my_recv that will repeatedly call socket.recv until the expected number of bytes have been received. Also, a 4-byte header (you can make the length greater if your file sizes warrant this) that contains a binary representation of the file length precedes the sending of the actual file data. In this way the client knows exactly how much data it should receive for each file.
Server Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
s.listen(1) # This should be called
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
# Get length by positioning to end of file
image_length = f.seek(0, 2)
f.seek(0, 0) # Seek back to beginning of file
# Convert image length to a 4-byte array:
image_length_bytes = image_length.to_bytes(4, 'big')
q.sendall(image_length_bytes)
data = f.read(4096)
while len(data):
q.sendall(data)
data = f.read(4096)
i += 1
Client Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
def my_recv(msg_length):
chunks = []
bytes_to_recv = msg_length
while bytes_to_recv:
chunk = s.recv(bytes_to_recv)
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_to_recv -= len(chunk)
return b''.join(chunks)
i = 0
while True:
image_length_bytes = my_recv(4)
image_length = int.from_bytes(image_length_bytes, 'big')
with open(str(i) + "s.jpg", "wb") as f:
bytes_to_recv = image_length
while bytes_to_recv:
recv_data = my_recv(min(4096, bytes_to_recv))
f.write(recv_data)
bytes_to_recv -= len(recv_data)
i += 1
I've written a basic client/server interface using Python socket (quoted only relevant part of code, for full script: (SERVER: https://github.com/mydomo/ble-presence/blob/master/server.py)
(CLIENT: https://github.com/mydomo/ble-presence/blob/master/clients/DOMOTICZ/ble-presence/plugin.py)
The issue is when the script run from some hours and the result list is getting bigger sometimes the reply is exactly as it should be, other times it's cutted, not complete... it's random, like if the socket closed for no reason earlier or the reply is not fully read.
Can you please help me?
SERVER:
def client_thread(conn, ip, port, MAX_BUFFER_SIZE = 32768):
# the input is in bytes, so decode it
input_from_client_bytes = conn.recv(MAX_BUFFER_SIZE)
# MAX_BUFFER_SIZE is how big the message can be
# this is test if it's too big
siz = sys.getsizeof(input_from_client_bytes)
if siz >= MAX_BUFFER_SIZE:
print("The length of input is probably too long: {}".format(siz))
# decode input and strip the end of line
input_from_client = input_from_client_bytes.decode("utf8").rstrip()
res = socket_input_process(input_from_client)
#print("Result of processing {} is: {}".format(input_from_client, res))
vysl = res.encode("utf8") # encode the result string
conn.sendall(vysl) # send it to client
conn.close() # close connection
##########- END FUNCTION THAT HANDLE SOCKET'S TRANSMISSION -##########
def start_server():
global soc
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# this is for easy starting/killing the app
soc.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#print('Socket created')
try:
soc.bind((socket_ip, socket_port))
# print('Socket bind complete')
except socket.error as msg:
# print('Bind failed. Error : ' + str(sys.exc_info()))
sys.exit()
#Start listening on socket
soc.listen(10)
#print('Socket now listening')
# for handling task in separate jobs we need threading
#from threading import Thread
# this will make an infinite loop needed for
# not reseting server for every client
while (not killer.kill_now):
conn, addr = soc.accept()
ip, port = str(addr[0]), str(addr[1])
#print('Accepting connection from ' + ip + ':' + port)
try:
Thread(target=client_thread, args=(conn, ip, port)).start()
except:
print("Terible error!")
import traceback
traceback.print_exc()
soc.close()
CLIENT:
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SERV_ADDR = str(Parameters["Address"])
SERV_PORT = int(Parameters["Port"])
soc.connect((SERV_ADDR, SERV_PORT))
if BATTERY_REQUEST == True:
clients_input = str(BATTERY_DEVICE_REQUEST)
else:
clients_input = "beacon_data"
soc.send(clients_input.encode()) # we must encode the string to bytes
result_bytes = soc.recv(32768) # the number means how the response can be in bytes
result_string = result_bytes.decode("utf8") # the return will be in bytes, so decode
Method recv() does not guarantee receiving the full message in the first call so you have to try getting the full message by calling recv() multiple times.
If recv() does return an empty string, connection is closed in the client side.
Using this while loop you can get full stream from client into data:
data = b'' # recv() does return bytes
while True:
try:
chunk = conn.recv(4096) # some 2^n number
if not chunk: # chunk == ''
break
data += chunk
except socket.error:
conn.close()
break
TCP is a streaming protocol, meaning it has no concept of what constitutes a complete message. You have to implement your own message protocol layer on top of TCP to make sure you send and receive complete messages. You are responsible for buffering data received until you have a complete message, and you have to define what a complete message is. Some options:
Send fixed length messages.
Send a fixed number of bytes representing the length of the message, then the message.
Separate messages with a sentinel byte.
Then, call recv and accumulate the results until you have a complete message in the buffer.
Im trying to write a basic server / client application in python, where the clients sends the numbers 1-15 to the server, and the server prints it on the server side console.
Code for client:
import socket
clientsocket.connect(('localhost', 8303))
def updateX():
x = 0
while (x < 15):
x
clientsocket.send(format(x))
print x
x = x+1
updateX()
server:
import socket
HOST = 'localhost'
PORT = 8303
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(5) # become a server socket, maximum 5 connections
connection, address = s.accept()
while True:
connection, address = s.accept()
buf = connection.recv(64)
print buf
The output of running the client while the server is live results in either no output, prints only 1, or prints only 12. Ideas?
Before entering the main loop on the server side, you accept a connection:
connection, address = s.accept()
But then in the loop itself you begin by accepting a connection:
while True:
connection, address = s.accept()
buf = connection.recv(64)
print buf
As a result, you never read from the first connection. That's why you don't see any output.
Note also that it's wrong (for what you're trying to do) to accept a new connection on every iteration. Even if you keep making new client connections, the server will accept a connection on each iteration and read from the socket once, but then continue the next iteration and wait for a new connection, never reading more data sent by a client. You should be making multiple recv calls on the same connection object instead.
You might find this tutorial helpful.
There are multiple errors:
socket.send() might send only partial content, use socket.sendall() instead
format(12) returns '12' therefore even if your code sends all numbers and the server correctly receives them then it sees '01234567891011121314' i.e., individual numbers are not separated
double socket.accept() mentioned by #Alp leads to ignoring the very first connection
socket.recv(64) may return less than 64 bytes, you need a loop until it returns an empty bytestring (meaning EOF) or use socket.makefile()
Client:
#!/usr/bin/env python
"""Send numbers in the range 0..14 inclusive as bytes e.g., 10 -> b'\n'
Usage: python client.py [port]
"""
import sys
import socket
from contextlib import closing
port = 8686 if len(sys.argv) < 2 else int(sys.argv[1])
with closing(socket.create_connection(('localhost', port))) as sock:
sock.sendall(bytearray(range(15))) # send each number as a byte
sock.shutdown(socket.SHUT_RDWR) # no more sends/receives
You need to know how numbers are separated in the data. In this case, a fixed format is used: each number is a separate byte. It is limited to numbers that are less than 256.
And the corresponding server:
#!/usr/bin/env python
"""
Usage: python server.py [port]
"""
from __future__ import print_function
import sys
import socket
from contextlib import closing
host = 'localhost'
port = 8686 if len(sys.argv) < 2 else int(sys.argv[1])
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # ipv4 version
try:
s.bind((host, port))
s.listen(5)
print("listening TCP on {host} port {port}".format(**vars()))
while True:
conn, addr = s.accept()
with closing(conn), closing(conn.makefile('rb')) as file:
for byte in iter(lambda: file.read(1), b''):
# print numerical value of the byte as a decimal number
print(ord(byte), end=' ')
print("") # received all input from the client, print a newline
except KeyboardInterrupt:
print('Keyboard interrupt received, exiting.')
finally:
s.close()
Ok,
I hope that I can explain what I am trying to do. I have an external device that accepts 4 byte commands with a checksum value. I need to send these to an external ethernet to serial interface and then read back the data recevied from the device. So far I can do this from a python script to a python script. WHere I am running into a problem is when sending the data to the device. I have tried using b'\x00\x00\x00\x00\x00' format, I have tried byte arrays and so on. I have wiresharked the data going out and see the data payload looking correct. But, I am not getting a response to the query I am sending.
import struct
import sys
import string
import select
import time
import binascii
import SocketServer
port = 8010
# region Read Codes
getSerial = [0x26, 0x0A, 0x00, 0xF0]
#Communication utilities
class CommUtilities():
#Convert to bytearray
def hexit2(self, inp):
return bytearray(inp)
def calculateChecksum(self, buf):
checksum = 0x5A #Starting checksum
#For each byte, bitwise XOR it with our checksum
for b in buf:
checksum ^= b
return checksum
#Append the one byte checksum to a command
#returns the complete byte array
def appendChecksum(self, buff):
buff.append(self.calculateChecksum(buff))
return buff
class TCPHandler():
""" Notes here.... TCP Handler """
def recv_timeout(self, the_socket, timeout=2):
total_data=[]
data=''
begin=time.time()
while l:
#if you got some data, then break after wait a sec
if total_data and time.time()-begin>timeout:
break
#if you got not data at all, wait a little longer
elif time.time()-begin>timeout*2:
break
try:
data=the_socket.recv(1024)
if data:
total_data.append(data)
begin=time.time()
else:
time.sleep(0.1)
except:
pass
return ''.join(total_data)
def start_server_receive(host):
sock=socket(AF_INET,SOCK_STREAM)
sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
sock.bind(('0.0.0.0', port))
sock.listen(5)
print 'started on',port
while True:
host = '0.0.0.0'
newsock,host=sock.accept()
print 'connected'
result= self.recv_timeout(newsock)
print 'got', result
sock.close()
return result
def send_server(data, host):
sock=socket(AF_INET,SOCK_STREAM)
sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
sock.connect((host, port))
print data
#Is this strip needed? Tried both with and without....
data = data.strip()
sock.sendall(data)
print 'Sent Data'
sock.close()
def main():
comm = CommUtilities()
tcp = TCPHandler()
host = "172.16.2.52"
#get data to send
data = comm.appendChecksum(getSerial)
data = comm.hexit2(data)
#Create a socket (SOCK_STREAM means a TCP socket)
while True:
try:
#Connect to server and send data
send_server(data, host)
#Receive data from the server and shut down
received = start_server_receive(host)
finally:
pass
print "send: {}".format(data)
#print "received:{}".format(received)
time.sleep(2)
if __name__ == '__main__':
main()
So any suggestions? I have example code that works in C# (which I didn't write since I don't know C# well at all), but how would I import that into python or use it? Otherwise I need a way to make the device respond...
Any and all suggestions would be appreciated.
Is it because I am not using struct.pack? If that is it, then how would I create the struct.pack function to make this work? I am not very well versed in the struct.pack method yet.
Here's a simple python 3.x TCP server:
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
print(str(self.client_address[0]) + " wrote: " + str(self.data.decode()))
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
and client:
import socket
import sys
HOST, PORT = "localhost", 9999
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
while( True ):
data = input("Msg: ")
if data == "exit()":
print("Exiting...")
sock.close()
exit();
sock.sendall(bytes(data, "utf-8"))
#numBytes = ....?
#print("Sent: " + str( numBytes ) + " bytes\n")
I can't figure out how to view the exact number of bytes that I send in a message. I can use len(data), but it doesn't account for the null terminator and such.... Is null terminator being sent as well, or is it irrelevant? I tried researching on an exact byte count of a sent/received message, but I couldn't find any python-specific documentation and only have seen examples of people using len(), which I don't think is exact...
Any ideas?
There is no null terminator in Python strings. If you want to send one, you have to do it explicitly: sock.sendall(bytes(data, "utf-8") + b'\0').
However, there's no good reason to add a null terminator in the first place, unless you're planning to use it as a delimiter between messages. (Note that this won't work for general Python strings, because they're allowed to include null bytes in the middle… but it will work fine for real human-readable text, of course.)
Using null bytes as a delimiter is not a bad idea… but your existing code needs to actually handle that. You can't just call recv(1024) and assume it's a whole message; you have to keep calling recv(1024) in a loop and appending to a buffer until you find a null—and then save everything after that null for the next time through the loop.
Anyway, the sendall method doesn't return the number of bytes sent because it always sends exactly the bytes you gave it (unless there's an error, in which case is raises). So:
buf = bytes(data, "utf-8") + b'\0'
sock.sendall(buf)
bytes_sent = len(buf)
And on the server side, you might want to write a NullTerminatedHandler class like this:
class NullTerminatedHandler(socketserver.BaseRequestHandler):
def __init__(self):
self.buf = b''
def handle(self):
self.buf += self.request.recv(1024)
messages = self.buf.split(b'\0')
for message in messages[:-1]:
self.handle_message(message)
self.buf = self.buf[:-1]
Then you can use it like this:
class MyTCPHandler(NullTerminatedHandler):
def handle_message(self, message):
print(str(self.client_address[0]) + " wrote: " + str(message.decode()))
While we're at it, you've got some Unicode/string issues. From most serious to least:
You should almost never just call decode with no argument. If you're sending UTF-8 data on one side, always explicitly decode('utf-8') on the other.
The decode method is guaranteed to return a str, so writing str(message.decode()) just makes your code confusing.
There's a reason the sample code uses format instead of calling str on a bunch of objects and concatenating them—it's usually a lot easier to read.
It's generally more readable to say data.encode('utf-8') than bytes(data, 'utf-8').