Background: I have a TCP Socket python script set up on a Raspberry Pi 4 to speak with an off-network machine. The machine has commands set up that I send it a request over the socket essentially querying "What is the value of A?" and the machine response is the value of A. While I am not super familiar with the coding of the machine, my understanding is that when queried in this way, it prints out the value. I am receiving that data through the socket by simply putting a socket.sendall("What is the value of A") followed immediately by a socket.recv(SIZE). My analogy is its like throwing a ball against the wall with one hand, and catching it with the other. The connection is made with an Ethernet cable approximately 100ft long.
Problem: I can query across this socket quite a bit, up to the point that I can send 60 entry arrays back and forth between 5-10 times, but eventually, the connection closes up. I'm not quite sure why this is. It gets to the point where I have to unplug everything, close out all the sockets, and just give it some time before trying again. While it may not be clear to diagnose the problem without fully understanding the machine (which I can't really give more information about), I am leaning towards using threading to run the two processes independent: one thread to query, one thread to receive. My guess would be that the recv() misses the proverbial ball, and then sits waiting to hear back from the machine, but the machine never talks again because the code is just left to wait. I don't have a lot of experience with threading, and therefore would appreciate some suggestions. Another thought is that the Ethernet cable is too long for the Pi to handle pushing that much data across. This feels more naive, but I am not a network engineer and therefore don't claim to fully understand that process.
Thanks in advance, feel free to ask any clarifying questions.
Related
I have a connection to a websocket server that some python code runs, and after some time (doesn't happen right away), it begins to send massive amounts of data "out" as shown in the screenshot. It's aroudn 4.8 megabytes per second on average. Prior to it starting to do this, it sends maybe 30 kb/s in what I assume is normal operation.
It can add up to hundreds of gigabytes in a day or two, depending on network speeds.
If I kill the original python process that was running and using the websocket client (kill -9 1234), it does not stop the traffic. I'm using Little Snitch for MacOS to obtain this information, and I feel like there is more info available under the surface I should be able to get to find out what's sending/receiving this data and what the data is.
If I terminate iTerm itself, the traffic still doesn't stop. If I launch regular Mac Terminal and do a ps aux | grep iterm I get nothing, and I get the same 4 processes shown in the screen shot if I do a | grep python...
This kind of throughput is really high, it's enough to be streaming my screen, uploading my entire hard drive, etc. or maybe it's just a code bug and it's sending garbage.
The only other relevant things I can think of adding right now are:
This is a brand new Macbook Pro M1 chip 13"
According to MenuMeters (resource monitor) there have been 16 billion page faults.. no idea if that's normal.
I have tried testing this by rebooting and just "not launching" my python code to get the websocket data, and basically I can wait a while and nothing seems to happen, so I think it's only happened after I launch the connection, and then I wait a while.
Sorry, I wish I knew how to get more relevant information for you, but if anyone has a good idea of how I can generate better logs or dig deeper I'd appreciate it.
screenshot of little snitch traffic
Sorry, I wish I knew how to get more relevant information for you, but
if anyone has a good idea of how I can generate better logs or dig
deeper I'd appreciate it.
Wireshark will allow you to track all connections and check what is inside the packets.
I have a python socket reader to listen for incoming UDP packets from about 5000 clients every minute. As I started rolling it out it was working fine but now that I'm up to about 4000 clients I'm losing about 50% of the data coming in. The VM has plenty of memory and CPU so I assume it's something with my UDP socket listener on the server getting too much data at once. Via cron, every minute the clients send in this data:
site8385','10.255.255.255','1525215422','3.3.0-2','Jackel','00:15:65:20:39:10'
This is the socket reader portion of my listener script.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
port = 18000
s.bind(('', port))
while True:
# Establish connection with client.
d = s.recvfrom(1024)
Could it be the buffer size is too small? How do I determine the size of the packets coming in so I can adjust the 1024 value?
Every 60 seconds, you get a storm of ~5000 messages. You process them sequentially, and it takes "quite a bit" of time. So pretty quickly, one of your buffers gets full up and either your OS, your network card, or your router starts dropping packets. (Most likely it's the buffer your kernel sets aside for this particular socket, and the kernel is dropping the packets, but all of the other options are possible too.)
You could try increasing those buffers. That will give yourself a lot more "allowed lag time", so you can get farther behind before the kernel starts dropping packets. If you want to go down this road, the first step is setsockopt to raise the SO_RCVBUF value, but you really need to learn about all the issues that could be involved here.1
If you control the client code, you could also have the clients stagger their packets (e.g., just sleeping for random.random() * 55 before the send).
But it's probably better to try to actually service those packets as quickly as possible, and do the processing in the background.2
Trying to do this in-thread could be ideal, but it could also be very fiddly to get right. A simpler solution is to just a background thread, or a pool of them:
def process_msg(d):
# your actual processing code
with concurrent.futures.ThreadPoolExecutor(max_workers=12) as x:
while True:
d = s.recvfrom(1024)
x.submit(process_msg, d)
This may not actually help. If your processing is CPU-bound rather than I/O-bound, the background threads will just be fighting over the GIL with the main thread. If you're using Python 2.7 or 3.2 or something else old, even I/O-bound threads can interfere in some situations. But either way, there's an easy fix: Just change that ThreadPoolExecutor to a ProcessPoolExecutor (and maybe drop max_workers to 1 fewer than the number of cores you have, to make sure the receiving code can have a whole core to itself).
1. Redhat has a nice doc on Network Performance Tuning. It's written more from the sysadmin's point of view than the programmer's, and it expects you to either know, or know how to look up, a lot of background information—but it should be helpful if you're willing to do that. You may also want to try searching Server Fault rather than Stack Overflow if you want to go down this road.
2. Of course if there's more than a minute's work to be done to process each minute's messages, the queue will just get longer and longer, and eventually everything will fail catastrophically, which is worse than just dropping some packets until you catch up… But hopefully that's not an issue here.
I am trying to create a multiplayer game for iPhone(cocos2d), which is almost finished, but the multiplayer part is left. I have searched the web for two days now, and can´t find any thing that answers my question.
I have created a search room(tcp socket on port 2000) that matches players that searches for a quick match to play. After two players has been matched, that server disconnects them from the search room to leave space for incoming searchers(clients/players)
Now I´m wondering how to create the play room(where two players interact and play)?
I was thinking I could create a new tcp/udp socket on a new port and let the matched(matched in the search room) players connect to that socket and then have a perfect isolated room for the two to interact with each another.
Or do I need a new server (machine/hardware) and than create a new socket on that one and let the peered players connect to it.
Or maybe there is another way of doing this.
OBS. I am not going to have the game running on the server to deal with cheaters for now. Because this will be to much load for the server cpu on my setup.
I was thinking I could create a new tcp/udp socket on a new port and let the matched(matched in the search room) players connect to that socket and then have a perfect isolated room for the two to interact with each another.
Yes, you can do that. And there shouldn't be anything hard about it at all. You just bind a socket on the first available port, pass that port to the two players, and wait for them to connect. If you're worried about hackers swooping in by, e.g., portscanning for new ports opening up, there are ways to deal with that, but given that you're not attempting any cheap protection I doubt it's an issue.
Or do I need a new server (machine/hardware) and than create a new socket on that one and let the peered players connect to it.
Why would you need that? What could it do for you? Sure, it could take some load off the first server, but there are plenty of ways you could load-balance if that's the issue; doing it asymmetrically like this tends to lead to one server at 100% while the other one's at 5%…
Or maybe there is another way of doing this.
One obvious way is to not do anything. Just let them keeping talking to the same port they're already talking to, just attach a different handler (or a different state in the client state machine, or whatever; you haven't given us any clue how you're implementing your server). I don't know what you think you're getting out of "perfect isolation". But even if you want it to be a different process, you can just migrate the two client sockets over to the other process; there's no reason to make them connect to a new port.
Another way to do it is to get the server out of the way entirely—STUN or hole-punch them together and let them P2P the game protocol.
Anything in between those two extremes doesn't seem worth doing, unless you have some constraints you haven't explained.
OBS. I am not going to have the game running on the server to deal with cheaters for now. Because this will be to much load for the server cpu on my setup.
I'm guessing that if putting even minimal game logic on the server for cheat protection is too expensive, spinning off a separate process for every pair of clients may also be too expensive. Another reason not to do it.
I have a program that sniffs network data and stores it in a database using pcapy (based on this). I need to make the data available in realtime over a network connection.
Right now when i run the program it will start a second thread for the sniffer and a Twisted server on the main thread, however i have no idea how to get clients to 'tap into' the sniffer that's running in the background.
The end result should be that a client enters an url and the connection will be kept open until the client disconnects (even when there's nothing to send), whenever the server has network activity the sniffer will sniff it and send it to the clients.
I'm a beginner with Python so i'm quite overwhelmed so if anyone could point me in the right direction it would be greatly appreciated.
Without more information (a simple code sample that doesn't work as you expect, perhaps) it's tough to give a thorough answer.
However, here are two pointers which may help you:
Twisted Pair, a (unfortunately very rudimentary and poorly documented) low-level/raw sockets networking library within Twisted itself, which may be able to implement the packet capture directly in a Twisted-friendly way, or
The recently-released Crochet, which will allow you to manage the background Twisted thread and its interactions with your pcapy-based capture code.
Update to original post: A colleague pointed out what I was doing wrong.
I'll give the explanation at the bottom of the post, as it might be helpful
for others.
I am trying to get a basic understanding of the limits on network performance
of python programs and have run into an anomaly. The code fragment
while 1:
sock.sendto("a",target)
sends UDP packets to a target machine, as fast as the host will send.
I measure a sending rate of just over 4000 packets per second, or 250 us
per packet. This seems slow, even for an interpreted language like python
(the program is running on a 2 GHz AMD opteron, Linux, python version 2.6.6).
I've seen much better performance in python for TCP, so I find this a bit weird.
If I run this in the background and run top, I find that python is using
just 25% of the cpu, suggesting that python may be artificially delaying
the transmission of UDP packets.
Has anyone else experienced anything similar? Does anyone know if python
does limit the rate of packet transmission, and if there is a way to turn
this off?
BTW, a similar C++ program can send well over 200,000 packets per second,
so it's not an intrinsic limit of the platform or OS.
So, it turns out I made a silly newbie mistake. I neglected to call gethostbyname
explicitly. Consequently, the target address in the sendto command contained
a symbolic name. This was triggering a name resolution every time a packet was
sent. After fixing this, I measure a maximum sending rate of about 120,000 p/s.
Much better.
You might want to post a more complete code sample so that others can repeat your benchmark. 250μs per loop iteration is too slow. Based on daily experience with optimizing Python, I would expect Python's interpreter overhead to be well below 1μs on a modern machine. In other words, if the C++ program is sending 200k packets per second, I would expect Python to be in the same order of magnitude of speed.
(In light of the above, the usual optimization suggestions such as moving the attribute lookup of sock.sendto out of the loop do not apply here because the slowness is coming from another source.)
A good first step to be to use strace to check what Python is actually doing. Is it a single-threaded program or a multithreaded application that might be losing time waiting on the GIL? Is sock a normal Python socket or is it part of a more elaborate API? Does the same happen when you directly call os.write on the socket's fileno?
Have you tried doing a connect() first, then using send() instead of sendto()? (UDP connect() just establishes the destination address, it doesn't actually make a "connection".) I'm rusty on this, but I believe Python does more interpretation on the address parameter than C sockets, which might be adding overhead.