We have a string which should be send to a service via a websocket. This string containt some data which is seperated by some tags. One row of data always ends with <\r><\n>.
So we want to send each row of data to this service which is implemented as a TCPServer. This is working perfectly fine in case we use normal strings, but as soon as we try to incorporate the tags I mentioned, Python just keeps appending them in the infinite loop and sends them as one huge string when the loop is exited:
Consider the following implementation:
import random
from datetime import datetime
import time
import socket
import sys
HOST = "xx.xx.xx.xx"
PORT = 12345
srv = (HOST, PORT)
cls = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
cls.connect(srv)
# DATA:
# Variable to change status text
dec = 0
while 1:
data = "foo\\nbar"
size = cls.send(data.encode())
print(size)
time.sleep(1)
Then the string is built up in python until I quit the loop manually to: foo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbarfoo\\nbar and sent to the socket.
What we want to accomplish is, that those strings are sent after each other like:
foo\nbar
foo\nbar
foo\nbar
Is there any possibility to send those special characters with the string together?
Related
I am trying to send an image via python sockets. I wrote a simple program for doing that.
server.py
import socket
import cv2
import sys
c = 0
q1 = '127.0.0.1'
q2 = 10001
s = socket.socket()
s.bind((q1, q2))
s.listen(5)
image_path = sys.argv[1]
with open(image_path, 'rb') as fh:
print(fh)
print(dir(fh))
l = list(fh)
print(len(l))
c, addr = s.accept()
if c != 0:
for i in fh:
c.send(i)
client.py
import socket
import cv2
import time
s = socket.socket()
q = '127.0.0.1'
p = 10001
condition = True
s.connect((q,p))
counter = 0
with open('recieved_image.jpg', 'wb') as fh:
while condition:
counter = counter+1
img = s.recv(1024)
if img == b'':
condition = False
fh.write(img)
print(counter)
Now this is the problem. In my case, when I run python server.py test.jpg, the length of list(fh) is 374. And those parts from list(fh) are sent one at a time via socket. But when I recieve them from client.py, the counter variable increments only one time. Isn't it supposed to increment until 374? Can someone clarify on the things that I have understood wrong?
It's not a peculiarity of "Python sockets", it's actually how TCP sockets behave. When you call a low level function like send with TCP, it may or may not send every single byte you requested to be sent, and you have to check its return value in order to determine how much was actually sent. In this specific case, you're calling the send function for every line in the file (because you're using the file descriptor as an iterable). On client side, you try to read up to 1024 bytes from the socket, but just like send, it is not guaranteed you will read all of the data within a single recv call. Obviously, since your counter is incremented just one time, that means that recv receives everything in one stand, in that particular occasion. Learn more about sockets, there are many good tutorials and documentations, even on Wikipedia.
I am writing a simple client server program using Sockets in python. In my example, 1st I want to send a simple "text" in a packet for example, 2nd I will send an image for example.
In the client I want that if I receive a text, I print it on my console. But, if I receive a file (like an image), I save it on my hard drive.
How can I differentiate between the packets I receive on the client side?
#Server Code
import socket
host = socket.gethostname()
s_client=socket.socket()
port_server = 8081
s_client.bind((host, port_server))
s_client.listen(1)
print("Waiting for any incoming connections from clients ...")
conn_client, addr_client = s_client.accept()
#Send a text
conn_client.send("text".encode())
#Send a file (image for example)
f = open("image", "rb")
l = f.read(1024)
while (l):
conn_client.send(l)
l = f.read(1024)
#Client code
import socket
s = socket.socket()
host = socket.gethostname()
port_server = 8081
s.connect((host, port_server))
print("Connected...")
while True:
data = s.recv(1024)
#I WANT TO DIFFERENTIATE HERE
if data:
print(data)
Python's socket object is designed for low-level network communication. In other words, you can use it to send some raw data from one communication endpoint to another one. The socket object itself does not have any means to differentiate between data types or content. This typically takes place on a higher protocol layer.
For example, when a browser receives some data from a server via http, usually the content type is transmitted so the browser knows how to handle the data.
For your purpose, it might be perfectly sufficient if the server would in a similar way send some kind of prefix before the actual data - for example the string 'TXT' if the subsequent data is a text, and 'IMG' if the subsequent data is an image. The client could then evaluate these first three characters and interpret the data appropriately.
I have .NET client/server applications that communicate with a line protocol using TcpClient and an SSLStream. Basically they exchange some information and then enter a heartbeat loop where the client periodically sends a message to the server and receives an answer.
This works well and can do so for days, but sometimes the connection breaks with a Socket error code of either 10054 or 10060, saying the host is unresponsive.
For testing I got rid of most code and tried it with the very minimal components.
On the server I use a TcpListener and run this code for new clients in a new task (please note that I have stripped logging code from the example):
Private Sub HandleClient(c As TcpClient)
c.NoDelay = True 'I tried with and without this algorithm
Dim stream = c.GetStream
Dim buffer(1024) As Byte
c.ReceiveTimeout = 60000
Do
Dim value As Integer = 0
Try
value = stream.Read(buffer, 0, buffer.Length)
Catch ex As Exception
End Try
For i = 0 To value - 1
buffer(i) = 0 'Just to get rid of residuals for error checking
Next
stream.Write({1, 45, 26}, 0, 3)
Loop
End Sub
Most is just logging, the values I send are just some random numbers. So here I read the values from a NetworkStream and write some numbers back. Note: In the real application I use WriteLine and ReadLine.
The client code is equally short, it is also started in a new task:
Private Sub Connect()
'Get the ip to connect to from the command line, or use localhost
Dim ip = If(My.Application.CommandLineArgs.Count > 0, My.Application.CommandLineArgs(0), "127.0.0.1")
log("Trying to parse " & ip)
Dim parsed As IPAddress = Nothing
If IPAddress.TryParse(ip, parsed) = False Then parsed = IPAddress.Parse("127.0.0.1")
Dim c As New TcpClient()
c.NoDelay = True
c.Connect(parsed, 16543)
Dim stream = c.GetStream
Dim buffer(1024) As Byte
Do
stream.Write({33, 75, 123, 3}, 0, 4)
Dim value As Integer
Try
value = stream.Read(buffer, 0, buffer.Length)
Catch ex As Exception
End Try
For i = 0 To value - 1
buffer(i) = 0 'Just to get rid of residuals for error checking
Next
'Write a message every 5 seconds
Threading.Thread.Sleep(5000)
Loop
End Sub
I run this alongside my normal client and server applications and it works flawlessly but disconnects at the exact same time as my normal client/server connection. The quick conclusion is that there is a problem with the internet connection on this device.
However, I also adapted a small python script to do basically the some thing, it connects and exchanges Hello World strings:
Server:
import socket
import time
from datetime import tzinfo, timedelta, datetime
TCP_IP = 'this server'
TCP_PORT = 7777
BUFFER_SIZE = 20
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn,addr = s.accept()
while 1:
while 1:
data = conn.recv(BUFFER_SIZE)
if not data: break
conn.send("World!")
conn.close()
Client:
import socket
import time
from datetime import tzinfo, timedelta, datetime
TCP_IP = 'my server'
TCP_PORT = 7777
BUFFER_SIZE = 1024
MESSAGE = "Hello"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while 1:
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
time.sleep(5)
s.close()
I run this as well on the same machines and it has no problem whatsoever. It just keeps on sending, so I guess the internet connection is ok.
The client is behind a NAT router while the server is running on a vServer off-site.
So, to summarize:
Two .NET client/server applications experience disconnects at the same time, at random intervals (the can run for hours before a disconnect happens).
A python client/server application that seems to do the same task does not experience disconnects.
I am truly out of ideas what the cause of this problem may be.
Things I have tried are:
Using Async read/write operations instead of synchronous, enabling/disabling Nagle's algorithm (NoDelay), sending raw bytes instead of using WriteLine/ReadLine.
It may be machine or connection dependent, as two other test clients on other connections seem to be running stable for the last few days. But it also seems implementation dependent, because the Python script works forever.
Sorry for the wall of text/code, I'm grateful for any idea about where I went wrong.
I have LIFX lights and a VOIP phone system. I'm trying to figure out how to trigger a light to turn on or off based on if a phone is in use. When a user picks up or hangs up a handset, the phone broadcasts its state over UDP 2088.
Using Wireshark, I know that the payload for offhook is 0b1f1001c0a8fa33dcd19b200000000813ed020412020066
and the payload for on-hook is 0b1f1001c0a8fa33dcd19b200000000813ed000412020066 (although if I copy straight from Wireshark, it inserts colons when I paste 0b:1f:10:01:c0...)
I've figured out how to turn my office light on or off using the LIFXLan module and have gotten to the point where I can trigger lights manually by sending plaintext from a client to a server listening on whatever UDP port. However, I'm stuck at trying to use data from the phones to trigger the lights. I'm not 100% sure it can be done, in fact. My code is as follows:
import Bulb_Off
import Bulb_On
import sys
import socket
import codecs
import binascii
#I may have imported unnecessary modules
UDP_PORT = 2088
OFF_HOOK = "0b1f1001c0a8fa33dcd19b200000000813ed020412020066"
ON_HOOK = "0b1f1001c0a8fa33dcd19b200000000813ed000412020066" #I am hoping I can simply match the data in the data field of the packet
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', UDP_PORT))
while True:
data, addr = sock.recvfrom(1024)
if data == ON_HOOK:
Bulb_Off.main()
print "received message:", data
print "1"
elif data == OFF_HOOK:
Bulb_On.main()
print "received message:", data
print "2"
else:
print data
The output I usually get is gibberish such as al▒.~5Vn]D▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}▒▒▒▒▒▒▒▒
I know I have to convert the payload, but I've been stuck (I just started yesterday.) I'm guessing this is something extremely easy or ridiculously hard.
Got it figured out. Needed to add
.decode("hex")
to the end of the declarations for ON_HOOK and OFF_HOOK
I am running a very simple python (3.x) client-server program (both locally on my PC) for a school project (not intended for the real world) which just sends messages back-and-forth (like view customers, add customer, delete customer, etc... real basic).
Sometimes the data can be multiple records which I had stored as namedTuples (just made sense) and then went down the path of using Pickle to transfer then.
So for example on the client I do something like this:
s.send(message.encode('utf-8'))
pickledResponse = s.recv(4096);
response = pickle.loads(pickledResponse)
Now ever so often I get the following error:
response = pickle.loads(pickledResponse)
EOFError: Ran out of input
My fear is that this has something to do with my socket (TCP) transfer and maybe somehow I am not getting all the data in time for my pickle.loads - make sense? If not I am really lost as to why this would be happening so inconsistently.
However, even if I am right I am not sure how to fix it (quickly), I was considering dropping pickle and just using strings (but couldn't this suffer from the same fate)? Does anyone have any suggestions?
Really my message are pretty basic - usually just a command and some small data like "1=John" which means command (1) which is FIND command and then "John" and it returns the record (name, age, etc...) of John (as a namedTuple - but honestly this isn't mandatory).
Any suggestions or help would be much appreciated, looking for a quick fix...
The problem with your code is that recv(4096), when used on a TCP socket, might return different amount of data from what you might have expected, as they are sliced at packet boundaries.
The easy solution is to prefix each message with length; for sending like
import struct
packet = pickle.dumps(foo)
length = struct.pack('!I', len(packet)
packet = length + packet
then for receiving
import struct
buf = b''
while len(buf) < 4:
buf += socket.recv(4 - len(buf))
length = struct.unpack('!I', buf)[0]
# now recv until at least length bytes are received,
# then slice length first bytes and decode.
However, Python standard library already has a support for message oriented pickling socket, namely multiprocessing.Connection, that supports sending and receiving pickles with ease using the Connection.send and Connection.recv respectively.
Thus you can code your server as
from multiprocessing.connection import Listener
PORT = 1234
server_sock = Listener(('localhost', PORT))
conn = server_sock.accept()
unpickled_data = conn.recv()
and client as
from multiprocessing.connection import Client
client = Client(('localhost', 1234))
client.send(['hello', 'world'])
For receiving everything the server sends until it closes its side of the connection try this:
import json
import socket
from functools import partial
def main():
message = 'Test'
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect(('127.0.0.1', 9999))
sock.sendall(message.encode('utf-8'))
sock.shutdown(socket.SHUT_WR)
json_response = b''.join(iter(partial(sock.recv, 4096), b''))
response = json.loads(json_response.decode('utf-8'))
print(response)
if __name__ == '__main__':
main()
I've used sendall() because send() has the same ”problem” as recv(): It's not guaranteed everything is sent. send() returns the number of bytes actually sent, and the programmer has to make sure that matches the length of the argument and if not to send the rest until everything is out. After sending the writing side of the connection is closed (shutdown()) so the server knows there is no more data coming from the client. After that, all data from the server is received until the server closes its side of the connection, resulting in the empty bytes object returned from the recv() call.
Here is a suitable socketserver.TCPServer for the client:
import json
from socketserver import StreamRequestHandler, TCPServer
class Handler(StreamRequestHandler):
def handle(self):
print('Handle request...')
message = self.rfile.read().decode('utf-8')
print('Received message:', message)
self.wfile.write(
json.dumps(
{'name': 'John', 'age': 42, 'message': message}
).encode('utf-8')
)
print('Finished request.')
def main():
address = ('127.0.0.1', 9999)
try:
print('Start server at', address, '...')
server = TCPServer(address, Handler)
server.serve_forever()
except KeyboardInterrupt:
print('Stopping server...')
if __name__ == '__main__':
main()
It reads the complete data from the client and puts it into a JSON encoded response with some other, fixed items. Instead of the low level socket operations it makes use of the more convenient file like objects the TCPServer offers for reading and writing from/to the connection. The connection is closed by the TCPServer after the handle() method finished.