I am sending 20000 messages from a DEALER to a ROUTER using pyzmq.
When I pause 0.0001 seconds between each messages they all arrive but if I send them 10x faster by pausing 0.00001 per message only around half of the messages arrive.
What is causing the problem?
What is causing the problem?
A default setup of the ZMQ IO-thread - that is responsible for the mode of operations.
I would dare to call it a problem, the more if you invest your time and dive deeper into the excellent ZMQ concept and architecture.
Since early versions of the ZMQ library, there were some important parameters, that help the central masterpiece ( the IO-thread ) keep the grounds both stable and scalable and thus giving you this powerful framework.
Zero SHARING / Zero COPY / (almost) Zero LATENCY are the maxims that do not come at zero-cost.
The ZMQ.Context instance has quite a rich internal parametrisation that can be modified via API methods.
Let me quote from a marvelous and precious source -- Pieter HINTJENS' book, Code Connected, Volume 1.
( It is definitely worth spending time and step through the PDF copy. C-language code snippets do not hurt anyone's pythonic state of mind as the key messages are in the text and stories that Pieter has crafted into his 300+ thrilling pages ).
High-Water Marks
When you can send messages rapidly from process to process, you soon discover that memory is a precious resource, and one that can be trivially filled up. A few seconds of delay somewhere in a process can turn into a backlog that blows up a server unless you understand the problem and take precautions.
...
ØMQ uses the concept of HWM (high-water mark) to define the capacity of its internal pipes. Each connection out of a socket or into a socket has its own pipe, and HWM for sending, and/or receiving, depending on the socket type. Some sockets (PUB, PUSH) only have send buffers. Some (SUB, PULL, REQ, REP) only have receive buffers. Some (DEALER, ROUTER, PAIR) have both send and receive buffers.
In ØMQ v2.x, the HWM was infinite by default. This was easy but also typically fatal for high-volume publishers. In ØMQ v3.x, it’s set to 1,000 by default, which is more sensible. If you’re still using ØMQ v2.x, you should always set a HWM on your sockets, be it 1,000 to match ØMQ v3.x or another figure that takes into account your message sizes and expected subscriber performance.
When your socket reaches its HWM, it will either block or drop data depending on the socket type. PUB and ROUTER sockets will drop data if they reach their HWM, while other socket types will block. Over the inproc transport, the sender and receiver share the same buffers, so the real HWM is the sum of the HWM set by both sides.
Lastly, the HWM-s are not exact; while you may get up to 1,000 messages by default, the real buffer size may be much lower (as little as half), due to the way libzmq implements its queues.
Related
I'm experimenting with the python socket library (3.5, on linux mint 18), trying to understand UDP. I'm a hardware person dabbling in software, and UDP seems simpler to get my head around than TCP. I am well aware that UDP does not guarantee to deliver packets one for one.
So far, I can follow the tutorials to echo data back from a server to a client.
However, I like to push things to see what happens when applications don't follow the expected path, I detest writing things that 'hang' when unexpected things happen.
If a server binds a socket to a port number, then the client sends several messages to that port, before the server calls recvfrom() several times, I find that each call returns one message, with the messages in order. In other words, the messages have been buffered, later messages have not overwritten earlier messages in the queue. I was not surprised to see this happen, but also would not have been surprised to find only the last received message available, aka buffer length of one.
Is this buffer, and its depth, a python implementation detail, a linux mint/ubuntu detail, or defined by the UDP protocol?
Is this buffer, and its depth, a python implementation detail, a linux
mint/ubuntu detail, or defined by the UDP protocol?
The UDP socket's buffer sizes are an implementation detail of your OS's networking stack. Each OS tries to set reasonable default size based on its expected use-cases, but you can override the OS's default size (up to some maximum value, anyway) on a per-socket basis by calling socket.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, newSizeInBytes) and/or socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, newSizeInBytes)
The buffers will queue up as many packets as they have space to hold, then drop any incoming packets that they can't fully fit into the remaining space.
UDP buffers are in the operating system's network stack. The size of the buffers will depend on how much memory your computer has and kernel configuration settings. On modern computers with gigabytes of memory, it's likely that the OS will have plenty of space for UDP buffers, and it will be difficult to overflow them unless the computer is extremely overloaded.
There might be some way for you to configure the OS to limit the amount of memory used for UDP buffers, so that you can cause overflows and see what the symptoms are in your test application. I don't know the configuration settings, you could try asking in Unix & Linux or AskUbuntu.com.
I have a python socket reader to listen for incoming UDP packets from about 5000 clients every minute. As I started rolling it out it was working fine but now that I'm up to about 4000 clients I'm losing about 50% of the data coming in. The VM has plenty of memory and CPU so I assume it's something with my UDP socket listener on the server getting too much data at once. Via cron, every minute the clients send in this data:
site8385','10.255.255.255','1525215422','3.3.0-2','Jackel','00:15:65:20:39:10'
This is the socket reader portion of my listener script.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
port = 18000
s.bind(('', port))
while True:
# Establish connection with client.
d = s.recvfrom(1024)
Could it be the buffer size is too small? How do I determine the size of the packets coming in so I can adjust the 1024 value?
Every 60 seconds, you get a storm of ~5000 messages. You process them sequentially, and it takes "quite a bit" of time. So pretty quickly, one of your buffers gets full up and either your OS, your network card, or your router starts dropping packets. (Most likely it's the buffer your kernel sets aside for this particular socket, and the kernel is dropping the packets, but all of the other options are possible too.)
You could try increasing those buffers. That will give yourself a lot more "allowed lag time", so you can get farther behind before the kernel starts dropping packets. If you want to go down this road, the first step is setsockopt to raise the SO_RCVBUF value, but you really need to learn about all the issues that could be involved here.1
If you control the client code, you could also have the clients stagger their packets (e.g., just sleeping for random.random() * 55 before the send).
But it's probably better to try to actually service those packets as quickly as possible, and do the processing in the background.2
Trying to do this in-thread could be ideal, but it could also be very fiddly to get right. A simpler solution is to just a background thread, or a pool of them:
def process_msg(d):
# your actual processing code
with concurrent.futures.ThreadPoolExecutor(max_workers=12) as x:
while True:
d = s.recvfrom(1024)
x.submit(process_msg, d)
This may not actually help. If your processing is CPU-bound rather than I/O-bound, the background threads will just be fighting over the GIL with the main thread. If you're using Python 2.7 or 3.2 or something else old, even I/O-bound threads can interfere in some situations. But either way, there's an easy fix: Just change that ThreadPoolExecutor to a ProcessPoolExecutor (and maybe drop max_workers to 1 fewer than the number of cores you have, to make sure the receiving code can have a whole core to itself).
1. Redhat has a nice doc on Network Performance Tuning. It's written more from the sysadmin's point of view than the programmer's, and it expects you to either know, or know how to look up, a lot of background information—but it should be helpful if you're willing to do that. You may also want to try searching Server Fault rather than Stack Overflow if you want to go down this road.
2. Of course if there's more than a minute's work to be done to process each minute's messages, the queue will just get longer and longer, and eventually everything will fail catastrophically, which is worse than just dropping some packets until you catch up… But hopefully that's not an issue here.
So, I'm trying to write a very simple Client-Server system that places some congestion control over UDP. Simply, the server sends data packets and receives feedback packets from the client(s). I'm fairly new to this and am not certain about:
(1) How many users can be connected to the same socket?
(2) Can a socket be used by multiples users simultaneously (aka do reads/writes in parallel)?
(3) I need to use threading so that for each user, I can send data packets to them and receive feedback packets from them in parallel. I read in another post that one shouldn't be opening threads for each user (and I'm opening two threads per user). Am I wrong in doing this? Do I have an inherent misunderstanding? Does anyone have a suggestion for how to handle this?
Sorry, I'm very new to this.
(1) The number of connections to a socket is limited by the operative system but can go up to 100k and more check the C10K problem
(2) A socket may be used by multiple threads and even multiple processes using the pre-fork approach
(3) You may use a new thread per connection, but the best approach is to use a thread pool, alternatively you may use an asynchronous approach and avoid the use of threads
According to the documentation of socket.listen():
Listen for connections made to the socket. The backlog argument specifies the maximum number of queued connections and should be at least 0; the maximum value is system-dependent (usually 5), the minimum value is forced to 0.
So technically the request aren't processed in parallel but queued, until a usual maximum of 5. I don't know the reasons the author of the post you mention has to state that you shouldn't start a thread per user, but I have read that most servers use one thread per request. Also the thread pool is a common pattern, and it can be easily implemented using a syncronized object like Queue (this example may be useful if you want to see a small implementation).
I wrote an audio broadcasting server with Python/Twisted. It works fine, but the usage of memory grows too fast! I think that's because some user's network might not be good enough to download the audio in time.
My audio server broadcast audio data to different listener's client, if some of them can't download the audio in time, that means, my server keep the audio data until listeners received. And what's more, my audio server is a broadcasting server, it receive audio data, and send them to different clients, I though Twisted copy those data in different buffer, even they are same audio piece.
I want to reduce the usage of memory usage, so I need to know when is the audio received by the client, so that I can decide when to discard some slow clients. But I have no idea how to achieve that with Twisted. Do anyone have idea?
And what else can I do to reduce usage of memory usage?
Thanks.
Victor Lin.
You didn't say, but I'm going to assume that you're using TCP. It would be hard to write a UDP-based system which had ever increasing memory because of clients who can't receive data as fast as you're trying to send it.
TCP has built-in flow control capabilities. If a receiver cannot read data as fast as you'd like to send it, this information will be made available to you and you can send more slowly. The way this works with the BSD socket API is that a send(2) call will block or will return 0 to indicate it cannot add any bytes to the send buffer. The way it works in Twisted is by a system called "producers and consumers". The gist of this system is that you register a producer with a consumer. The producer calls write on the consumer repeatedly. When the consumer cannot keep up, it calls pauseProducing on the producer. When the consumer is again ready for more data, it calls resumeProducing on the producer.
You can read about this system in more detail in the producer/consumer howto, part of Twisted's documentation.
Make sure you're using Python's garbage collector and then go through and delete variables you aren't using.
I have to write a reliable, totally-ordered multicast system from scratch in Python. I can't use any external libraries. I'm allowed to use a central sequencer.
There seems to be two immediate approaches:
write an efficient system, attaching a unique id to each multicasted message,
having the sequencer multicast sequence numbers for the message id's it receives,
and sending back and forth ACK's and NACK's.
write an inefficient flooding system, where each multicaster simply re-sends each
message it receives once (unless it was sent by that particular multicaster.)
I'm allowed to use the second option, and am inclined to do so.
I'm currently multicasting UDP messages (which seems to be the only option,) but that means that some messages might get lost. That means I have to be able to uniquely identify each sent UDP message, so that it can be re-sent according to #2. Should I really generate unique numbers (e.g. using the sender address and a counter) and pack them into each and every UDP message sent? How would I go about doing that? And how do I receive a single UDP message in Python, and not a stream of data (i.e. socket.recv)?
The flooding approach can cause a bad situation to get worse. If messages are dropped due to high network load, having every node resend every message will only make the situation worse.
The best approach to take depends on the nature of the data you are sending. For example:
Multimedia data: no retries, a dropped packet is a dropped frame, which won't matter when the next frame gets there anyway.
Fixed period data: Recipient node keeps a timer that is reset each time an update is received. If the time expires, it requests the missing update from the master node. Retries can be unicast to the requesting node.
If neither of these situations applies (every packet has to be received by every node, and the packet timing is unpredictable, so recipients can't detect missed packets on their own), then your options include:
Explicit ACK from every node for each packet. Sender retries (unicast) any packet that is not ACKed.
TCP-based grid approach, where each node is manually repeats received packets to neighbor nodes, relying on TCP mechanisms to ensure delivery.
You could possibly rely on recipients noticing a missed packet upon reception of one with a later sequence number, but this requires the sender to keep the packet around until at least one additional packet has been sent. Requiring positive ACKs is more reliable (and provable).
The approach you take is going to depend very much on the nature of the data that you're sending, the scale of your network and the quantity of data you're sending. In particular it is going to depend on the number of targets each of your nodes is connected to.
If you're expecting this to scale to a large number of targets for each node and a large quantity of data then you may well find that the overhead of adding an ACK/NAK to every packet is sufficient to adversely limit your throughput, particularly when you add retransmissions into the mix.
As Frank Szczerba has said multimedia data has the benefit of being able to recover from lost packets. If you have any control over the data that you're sending you should try to design the payloads so that you minimise the susceptibility to dropped packets.
If the data that you're sending cannot tolerate dropped packets and you're trying to scale to high utilisation of your network then perhaps udp is not the best protocol to use. Implementing a series of tcp proxies (where each node retransmits, unicast, to all other connected nodes - similar to your flooding idea) would be a more reliable mechanism.
With all of that said, have you considered using true multicast for this application?
Just saw the "homework" tag... these suggestions might not be appropriate for a homework problem.
IMHO, you should choose an existing reliable UDP protocol. There are several you can choose, take a look at this SO question: What do you use when you need reliable UDP?
I personally like and use MoldUDP, which is the protocol used by Nasdaq's ITCH market data feed.