I would like to implement a TCP server using a Python script.
The server should basically do the following task:
It will be cyclically polled by a remote client, then reads certain data on its local workstation and sends it to the client polling.
One can assume the following:
There is always only one client connecting to the server (via Ethernet)
Client and server are running on a Windows platform
Python 2.5.1 will be used for implementation
Goal:
The server should be as efficient as possible with respect to reading/writing data from/to the client
The server shall stress the local workstation as less as possible
The server shall not cause system instability
Since I have only a little experience with the topic, I would like to discuss here how my current (very simple) code could be optimized to meet the requirements previously mentioned.
So far I have the following:
import socket
import sys
port_number = 12345
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = (socket.gethostname(), port_number)
sock.bind(server_address)
sock.listen(1)
while True:
connection, client_address = sock.accept()
try:
while True:
data = connection.recv(4096)
if data:
connection.sendall(getdesireddata(data))
else:
break
finally:
connection.close()
Thank you for all your responses.
First of all, there's minor bug in the code. The line...
data = connection.recv(4096)
...will attempt to read up to 4k from a single TCP packet, but if the client sends more than 4k, or decides to break up the data into several packets, you may not get all the data in a single call.
You'd typically continue to read data from the socket, appending to a buffer, until either you have a complete message as defined by your protocol specification, or until the remote host closes the outbound half of its TCP connection.
You'll also have issues with the server getting stuck if the client crashes, and doesn't shut down the socket properly. To avoid this, you'll need to look at either non-blocking IO, which would be something like...
connection.setblocking(0)
buffer = ''
while 1:
try:
data = connection.recv(4096)
if not data:
# Client has closed outbound connection
break
else:
# Append to buffer
buffer += data
except socket.error, e:
code = e.args[0]
if code == 11:
# Got EWOULDBLOCK/EAGAIN
time.sleep(0.1)
else:
# Got another error
raise
do_something_with(buffer)
...or look at using the Python select module to do the same thing.
With regards to the lesser concern of performance, I don't see any major improvements you can make with such a small code sample.
Related
I'm trying to make a simple UDP receiver, using python, to read a continuous data stream from a data aquisition device. This device can be configured to send data using UDP protocol to a certain IP and port. I configured it to send two data streams on port 4000 and 4001 both to ip: 192.168.0.10 (see wireshark screenshot below). The stream to port 4001 has a higher package rate. My laptop ip for the ethernet interface used is set statically at this ip in order to receive the data. I can receive the slow stream at port 4000 which works as expected. When I use the same code and set the port to 4001 I expected to read this data stream, instead I get a timeout exception.
I expected the biggest problem with high package rate udp would be loosing data because the buffer overflows which is not a big problem since loosing some packets is acceptable for my case. However if I get a timeout exception during the socket.recv(1024) call I thought that means there is no data in the buffer? Since wireshark does receive data I don't understand what is happening.
Here is a minimal example of what I am trying to do.
import socket
for i in range(3): # try multiple times
try:
socket_i = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
socket_i.settimeout(5)
socket_i.bind(('192.168.0.10', 4001)) #4000 works, 4001 does not
raw = socket_i.recv(1024)
socket_i.close()
print(raw)
except Exception as ex: # Exception is raised when using 4001: timed out
socket_i.close()
print('failed')
print(ex)
Wireshark printscreen; only 1 packet from port 4000 is shown, since this stream is slow.
I read similar questions:
Receive an high rate of UDP packets with python
Here it was concluded that python was the bottle neck. Also they received data, I don't receive anything, instead the socket.recv() times out. I don't think my package rate is that high, anyone thinks otherwise?
Receive an high rate of UDP packets with python
Here most of the data was received and only after some time the algorithm 'hangs on socket.recv()'. I tried increasing the buffer as suggested, using:
socket_i.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024*1024*1024)
Various values for buffer size are tried without luck. Setting large time out did not help.
Finally I tried using 'socketserver' package (the example udp server script):
import socketserver
class MyUDPHandler(socketserver.BaseRequestHandler):
"""
This class works similar to the TCP handler class, except that
self.request consists of a pair of data and client socket, and since
there is no connection the client address must be given explicitly
when sending data back via sendto().
"""
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
# socket.sendto(data.upper(), self.client_address) #don't want to send stuff back
if __name__ == "__main__":
with socketserver.UDPServer(('192.168.0.10', 4001), MyUDPHandler) as server:
server.serve_forever()
Using port 4000 this gives the expected behaviour, for port 4001 the code hangs without doing anything (probably large timeout by default?). I guess 'socketserver' package runs ontop of'socket' package?
I hope someone can help!
Update:
I fixed my problem by making the udp receiver using pyshark. This is a bit hacky and I'm not satisfied with the method however for now it works. I figured if wireshark receives data pyshark might work as well. Any suggestions using 'socket' package are welcome!
code for hacky udp receiver
import pyshark
ip = '192.168.0.10'
port = 4001
capture = pyshark.LiveCapture(bpf_filter=f'dst host {ip} and dst port {port}', use_json=True, include_raw=True)
for pkt in capture.sniff_continuously():
raw = bytearray.fromhex(pkt.udp.payload_raw[0])
print(raw)
I have written small programs to create both client and server on the host machine using python sockets. On the other side of the connection, I have my FPGA board which runs LwIP client and server also, and the connection is Gbit ethernet. My aim is to send and receive dummy data and monitor the throughput of both client and server.
client.py
from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_SNDBUF
send_data = []
for i in range(1458):
send_data.append(i % 256)
send_data = bytearray(send_data)
CLIENT_SOCK = socket(AF_INET, SOCK_STREAM)
CLIENT_SOCK.setsockopt(SOL_SOCKET, SO_SNDBUF, 524288)
CLIENT_SOCK.connect(("192.168.1.10", 42000))
print("CLIENT CONNECTED")
i=0
while True:
#print(i)
#i+=1
CLIENT_SOCK.sendall(send_data)
server.py
from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_RCVBUF
SERV_SOCK = socket(AF_INET, SOCK_STREAM)
SERV_SOCK.setsockopt(SOL_SOCKET, SO_RCVBUF, 524288)
SERV_SOCK.bind(("192.168.1.1", 42001))
SERV_SOCK.listen(1)
conn, b = SERV_SOCK.accept()
print("CLIENT ACCEPTED")
i=0
while True:
#print(i)
#i+=1
data = conn.recv(16384)
I run these codes in separate command lines. When I run only one of them, I get ~800 Mbit/s throughput for server.py, and ~950 Mbit/s for client.py. However, the problem arises when I run them at the same time. At this time, I still get the same speed for server.py but I only get a fluctuating throughput between ~100 Kbit/s and ~10 Mbit/s throughput for client.py.
When I uncomment the print statements in the while loops, I saw that in server.py the print statements are printing on the command line smoothly and very fast, but on the other hand client.py prints some and then stops for a while and then prints again but not in a fast manner. I tried to reduce the bandwidth of the server.py by decreasing the size of the receive bytes from 16384 to 256, and I saw that when I run it individually the throughput is around ~500 Mbit/s. However, again, when I run them at the same time, client.py throughput showed a similar pattern as previous. I tried to increase/decrease the send_data size, or adding "\r\n" at the end of the send_data but it didn't work.
Does anyone have any idea what might be the problem and the possible solution?
Thanks.
python socket client slows down when server is also running
In TCP a client will only send data as fast as the server can receive and ACK these (and vice versa). This means another perspective of looking at your problem is not a slow Python client but a slow LwIP server.
Since you run both LwIP client and server on the FPGA there might be a bad management of resources between these. If there is only one LwIP server/client running (and thus only one Python client/server), it will get all the resources on the FPGA and thus can run with max speed. If instead both Python client/server are running there will also be both LwIP both server/client running on the FPGA - and the system resources must be properly managed between these.
The results you get suggest that your LwIP client on the FPGA is getting most of the resources though, which means that the Python server will get max bandwidth. The LwIP server on the other hand will get far less resources which makes it slow - which thus causes the Python client to be slow.
Thus, don't look on the Python side but instead look on your resource managed between LwIP client and server on the FPGA.
I'm sending data via sockets between godot and python like this:
godot:
var socket = PacketPeerUDP.new()
socket.set_dest_address("127.0.0.1", 6000)
var data={...}
socket.put_packet(JSON.print(data).to_ascii())
python server:
s= socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 6000))
while True:
data = s.recvfrom(1024)
but the problem is even when the python server is not running the godot code sends the data instead of giving an error that the server is not available
I even tried var err=socket.set_dest_address("127.0.0.1", 6000) hopin this would print out the error
but it always prints 0 whether the python server is running or not
so how do I check if the server is available or not?
This is UDP we are talking about. So there isn't really a session or connection established. Also there isn't really an acknowledged package. So at the end the only solution is to implement your own reliability protocol on top of it (e.g. have the server respond and the client wait for the response). Try searching dor UDP reliability on the gamedev site.
The return values for set_dest_address are ERR_CANT_RESOLVE (the IP is not valid) or OK.
The returns values of put_packet. It can return ERR_BUSY (send buffers are full), FAILED (the socket is otherwise in use) or OK.
I am implementing a socket in Python to pass data back and forth between two scripts running on the same machine as part of a single Tkinter application.
This data, in many cases, will be highly sensitive (i.e. personal credit card numbers).
Does passing the data between scripts in this way open me up to any security concerns?
Server side:
import socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(5) # become a server socket, maximum 5 connections
while True:
connection, address = serversocket.accept()
buf = connection.recv(64)
if len(buf) > 0:
print buf
break
Client side:
import socket
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))
clientsocket.send('hello')
Code source.
Additional considerations:
This will only ever function as part of a single Tkinter application, on a single machine. Localhost will always be specified.
I am unable to use multiprocessing or threading; please no suggestions for using one of those or an alternative, other than varieties of socket. For more info as to why, see this SO question, answers, and comments. It has to do with this needing to function on Windows 7 and *nix, as well as my desired set-up.
Yes, passing the data between scripts in this way may raise a security concerns. If the attacker has an access to the same machine - he can easily sniff the traffic using the tool like tcpdump for example.
To avoid this you should encrypted your traffic - I have posted a comment below your question with an example solution.
I'm currently working with python's socket library for the first time and i'm not very experienced with computer networking.
I'm able to connect to the server and the tcp handshake has happened as viewed by wireshark. After establishing a connection to the server(I have no control over the server), the connection stays open for a while, but after a small amount of time, the server sends a "FIN, ACK" and the connection is terminated. I'm trying to understand how I can keep this connection alive while the client is capable of reaching the server.
Looking at a tcp connection, it seems a packet can be sent every so often. Maybe a sort of keep alive message. I had thought using socket.send('hello') every 5 seconds in another thread would keep the connection with the server open, but I still get the "FIN, ACK" after some time.
In the documentation I found a setsockopt() but using this made no noticeable difference. I've tried client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) both before and after the connection is made. I don't completely understand how this method is supposed to work, so maybe I used it incorrectly. There isn't much mention of this. I read somewhere about it being broken on windows. I don't know the truth in that.
What am I missing? The documentation for sockets doesn't seem to have anything about this unless I may have missed something.
import socket
import time
import threading
SERVER_IP = 'THE SERVER'
SERVER_PORT = SERVER_PORT
SOURCE_IP = socket.gethostname()
SOURCE_PORT = 57004
KEEP_ALIVE_INTERVAL = 5
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def keep_alive(interval):
data = 'hello'
while True:
client.send(data)
time.sleep(interval)
client.connect((SERVER_IP, SERVER_PORT))
t = threading.Thread(target=keep_alive, args = (KEEP_ALIVE_INTERVAL,))
t.start()
while True:
data = client.recv(1024)
if not data:
break
print data
client.close()
For enabling keep alive there is a duplicate question at How to change tcp keepalive timer using python script?
Keep in mind some servers and intermediate proxies forcibly close long lived connections regardless of keep alives being used or not, in which case you will see a FIN,ACK after X amount of time no matter what.