Develop Reference

ZMQ Pub-Sub latency spikes

I'm using a ZMQ pub-sub to pass messages from a C++ program to a Python script.
I'm passing around 5000 messages per second, and they vary in weight from 100 to 4500 bytes, with an average of about 300 bytes each. On average, I'm sending about 2.5 megabytes worth of data per second via this socket.
My monitoring script works like this:
Bind C++ program to the socket and leave it running.
Add a timestamp to each message going out of the C++ program.
Start python script, connect to socket, receive msg, get a timestamp and subtract the output timestamp, get the latency measurement.
Keep track of the average latency rolling over x messages.
My problem:
When I start the script, the socket latency hovers around 200 microseconds. That's within my expectations, and seems correct. However, after 1 minute avg latency is around 1.5 milliseconds, around 10-15ms after 4 minutes, and stabilizes at around 50ms after 10 minutes.
If I restart my Python script, latency goes back to 200 microseconds. I don't think the problem is message backlogging because if that was the case, the latency would keep increasing, and it would occasionally decrease too, instead of increasing linearly up to a certain threshold in a predictable way.
What could possibly be the issue?
Is there a way to either change ZMQ settings to try and improve this, or see if it's having some internal backlogging problem?

When you mentioned 50ms I immediately related to the Nagle algorithm.
However as you are saying that you use ZMQ on both sides and Nagle is disabled by default then I am sort of taken back.
Here are some ideas to debug:
Start the C++ process under strace with strace <binary> args and look at the logs, in particular the ones right after the socket creation. Check that TCP_NODELAY is set on the socket.
Start wireshark as admin and capture on the interface you are publishing. Follow the given TCP stream and try to check for blatant errors (eg retransmission) and patterns.

UDP buffers, where are they, what's guaranteed?

I'm experimenting with the python socket library (3.5, on linux mint 18), trying to understand UDP. I'm a hardware person dabbling in software, and UDP seems simpler to get my head around than TCP. I am well aware that UDP does not guarantee to deliver packets one for one.
So far, I can follow the tutorials to echo data back from a server to a client.
However, I like to push things to see what happens when applications don't follow the expected path, I detest writing things that 'hang' when unexpected things happen.
If a server binds a socket to a port number, then the client sends several messages to that port, before the server calls recvfrom() several times, I find that each call returns one message, with the messages in order. In other words, the messages have been buffered, later messages have not overwritten earlier messages in the queue. I was not surprised to see this happen, but also would not have been surprised to find only the last received message available, aka buffer length of one.
Is this buffer, and its depth, a python implementation detail, a linux mint/ubuntu detail, or defined by the UDP protocol?

Is this buffer, and its depth, a python implementation detail, a linux
mint/ubuntu detail, or defined by the UDP protocol?
The UDP socket's buffer sizes are an implementation detail of your OS's networking stack. Each OS tries to set reasonable default size based on its expected use-cases, but you can override the OS's default size (up to some maximum value, anyway) on a per-socket basis by calling socket.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, newSizeInBytes) and/or socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, newSizeInBytes)
The buffers will queue up as many packets as they have space to hold, then drop any incoming packets that they can't fully fit into the remaining space.

UDP buffers are in the operating system's network stack. The size of the buffers will depend on how much memory your computer has and kernel configuration settings. On modern computers with gigabytes of memory, it's likely that the OS will have plenty of space for UDP buffers, and it will be difficult to overflow them unless the computer is extremely overloaded.
There might be some way for you to configure the OS to limit the amount of memory used for UDP buffers, so that you can cause overflows and see what the symptoms are in your test application. I don't know the configuration settings, you could try asking in Unix & Linux or AskUbuntu.com.

Maximum UDP-packet number which can be stored in the socket buffer? (Ubuntu)

Client:
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
msg = b"X"
for i in range(1500):
s.sendto(msg,("<IP>",<PORT>))
Server:
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.bind(("",>PORT>))
counter = 0
for i in range(1500):
s.recv(1)
counter += 1
I have two machines - the first one with Windows7 and the second one with Ubuntu 16.04.
Now the problem:
If I try to send 1500 UDP-packets (for example) from the client to the server, then:
Windows7 is Client and Ubuntu16.04 is server:
server only receives between 200 and 280 packets
Ubuntu16.04 is Client and Windows7 is server:
server receives all 1500 packets
My first question:
What is the reason for this? Are there any limitations on the OS?
Second question:
Is it possible to optimize the sockets in Python?
I know that it will be possible, that UDP-packages can get lost - but up to 4/5 of all packets?
edit:
Why this kind of question?
Imagine I have a big sensor-network... and one server. Each sensor-node should send his information to the server. The program on the server can only be programmed in an asynchronious way - the server is only able to read the data out of the socket at a specific time. Now I want to calculate how many sensor-nodes can send data via UDP-packets to the server during the period of time where the server is not able to read out his buffer. With the information how many different UDP-packets can be stored in the buffer, I can calculate how many sensor-nodes I can use...

Instead of writing a cluttered comment trail, here's a few cents to the problem.
As documented by redhat the default values for the different OS:es in this writing moment is:
Linux: 131071
Windows: No known limit
Solaris: 262144
FreeBSD, Darwin: 262144
AIX: 1048576
These values should correspond to the output of:
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
print(s.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF))
These numbers represents how many bytes can be held at any given moment in the socket receive buffer. The numbers can be increased at any given time at the cost of RAM being reserved for this buffer (or at least that's what I remember).
On Linux (And some BSD flavors), to increase the buffer you can use sysctl:
sudo sysctl -w net.core.rmem_max=425984
sudo sysctl -w net.core.rmem_default=425984
This sets the buffer to 416KB. You can most likely increase this to a few megabytes if buffering is something you see a lot of.
However, buffers usually indicate a problem because your machine should rarely have much in the buffer at all. It's a mechanism to handle sudden peaks and to serve as a tiny platter for your machine to store work load. If it gets to full, either you have a really slow code that needs to get quicker or you need to offload your server quite a bit. Because if the buffer fills up - no matter how big it is, eventually it will get full again.
Supposedly you can also increase the buffer size from Python via:
s.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF, 1024)
However, again, if your OS is capped at a certain roof - that will supersede any value you put in your python program.
tl;dr:
Every OS has limitations based on optimizations/performance reasons. Sockets, file handles (essentially any I/O operation) has them.
It's common, you should find a lot of information on it. All this information above was mostly found via a search on "linux udp recieve buffer".
Also, "windows increase udp buffer size" landed me on this: Change default socket buffer size under Windows
Final note
As you mentioned, the performance, amount etc can vary vastly due to the fact that you're using UDP. It is prone to data loss at benefit of speed. Distance between servers, drivers, NIC's (especially important, some NIC's have a limited hardware buffer that can cause these things) etc all impact the data you'll receive. Windows do a lot of auto-magic as well in these situations, make sure you tune your Linux machine to the same parameters. A UDP packet consists not only of the ammount of data you send.. but all the parameters in the headers before it (in the IP packet, for instance TTL, Fragmentation, ECN, etc.).
For instance, you can tune how much memory your UDP stack can eat under certain loads, to find out your lower threshold (UDP won't bother checking RAM usage), pressure threshold (memory management under load) and the max value UDP sockets can use per socket.
sudo sysctl net.ipv4.udp_mem
Here's a good article on UDP tuning from ESnet:
https://fasterdata.es.net/network-tuning/udp-tuning/
Beyond this, you're tweaking to your grave. Most likely, your problem can be solved by redesigning your code. Because unless you're actually pushing 1-10GB/s from your network, the kernel should be able to handle it assuming you process the packets fast enough, rather than piling them up in a buffer.

Muliple clients sending UDP data to python socket are getting lost

I have a python socket reader to listen for incoming UDP packets from about 5000 clients every minute. As I started rolling it out it was working fine but now that I'm up to about 4000 clients I'm losing about 50% of the data coming in. The VM has plenty of memory and CPU so I assume it's something with my UDP socket listener on the server getting too much data at once. Via cron, every minute the clients send in this data:
site8385','10.255.255.255','1525215422','3.3.0-2','Jackel','00:15:65:20:39:10'
This is the socket reader portion of my listener script.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
port = 18000
s.bind(('', port))
while True:
# Establish connection with client.
d = s.recvfrom(1024)
Could it be the buffer size is too small? How do I determine the size of the packets coming in so I can adjust the 1024 value?

Every 60 seconds, you get a storm of ~5000 messages. You process them sequentially, and it takes "quite a bit" of time. So pretty quickly, one of your buffers gets full up and either your OS, your network card, or your router starts dropping packets. (Most likely it's the buffer your kernel sets aside for this particular socket, and the kernel is dropping the packets, but all of the other options are possible too.)
You could try increasing those buffers. That will give yourself a lot more "allowed lag time", so you can get farther behind before the kernel starts dropping packets. If you want to go down this road, the first step is setsockopt to raise the SO_RCVBUF value, but you really need to learn about all the issues that could be involved here.1
If you control the client code, you could also have the clients stagger their packets (e.g., just sleeping for random.random() * 55 before the send).
But it's probably better to try to actually service those packets as quickly as possible, and do the processing in the background.2
Trying to do this in-thread could be ideal, but it could also be very fiddly to get right. A simpler solution is to just a background thread, or a pool of them:
def process_msg(d):
# your actual processing code
with concurrent.futures.ThreadPoolExecutor(max_workers=12) as x:
while True:
d = s.recvfrom(1024)
x.submit(process_msg, d)
This may not actually help. If your processing is CPU-bound rather than I/O-bound, the background threads will just be fighting over the GIL with the main thread. If you're using Python 2.7 or 3.2 or something else old, even I/O-bound threads can interfere in some situations. But either way, there's an easy fix: Just change that ThreadPoolExecutor to a ProcessPoolExecutor (and maybe drop max_workers to 1 fewer than the number of cores you have, to make sure the receiving code can have a whole core to itself).
1. Redhat has a nice doc on Network Performance Tuning. It's written more from the sysadmin's point of view than the programmer's, and it expects you to either know, or know how to look up, a lot of background information—but it should be helpful if you're willing to do that. You may also want to try searching Server Fault rather than Stack Overflow if you want to go down this road.
2. Of course if there's more than a minute's work to be done to process each minute's messages, the queue will just get longer and longer, and eventually everything will fail catastrophically, which is worse than just dropping some packets until you catch up… But hopefully that's not an issue here.

udp rate limits in python?

Update to original post: A colleague pointed out what I was doing wrong.
I'll give the explanation at the bottom of the post, as it might be helpful
for others.
I am trying to get a basic understanding of the limits on network performance
of python programs and have run into an anomaly. The code fragment
while 1:
sock.sendto("a",target)
sends UDP packets to a target machine, as fast as the host will send.
I measure a sending rate of just over 4000 packets per second, or 250 us
per packet. This seems slow, even for an interpreted language like python
(the program is running on a 2 GHz AMD opteron, Linux, python version 2.6.6).
I've seen much better performance in python for TCP, so I find this a bit weird.
If I run this in the background and run top, I find that python is using
just 25% of the cpu, suggesting that python may be artificially delaying
the transmission of UDP packets.
Has anyone else experienced anything similar? Does anyone know if python
does limit the rate of packet transmission, and if there is a way to turn
this off?
BTW, a similar C++ program can send well over 200,000 packets per second,
so it's not an intrinsic limit of the platform or OS.
So, it turns out I made a silly newbie mistake. I neglected to call gethostbyname
explicitly. Consequently, the target address in the sendto command contained
a symbolic name. This was triggering a name resolution every time a packet was
sent. After fixing this, I measure a maximum sending rate of about 120,000 p/s.
Much better.

You might want to post a more complete code sample so that others can repeat your benchmark. 250μs per loop iteration is too slow. Based on daily experience with optimizing Python, I would expect Python's interpreter overhead to be well below 1μs on a modern machine. In other words, if the C++ program is sending 200k packets per second, I would expect Python to be in the same order of magnitude of speed.
(In light of the above, the usual optimization suggestions such as moving the attribute lookup of sock.sendto out of the loop do not apply here because the slowness is coming from another source.)
A good first step to be to use strace to check what Python is actually doing. Is it a single-threaded program or a multithreaded application that might be losing time waiting on the GIL? Is sock a normal Python socket or is it part of a more elaborate API? Does the same happen when you directly call os.write on the socket's fileno?

Have you tried doing a connect() first, then using send() instead of sendto()? (UDP connect() just establishes the destination address, it doesn't actually make a "connection".) I'm rusty on this, but I believe Python does more interpretation on the address parameter than C sockets, which might be adding overhead.