Develop Reference

Python - Read remaining data from socket after TCP RST

I'm implementing a file transfer protocol with the following use case:
The server sends the file chunk by chunk inside several frames.
The client might cancel the transfer: for this, it sends a message and disconnects at TCP level.
What happened in that case on server side (Python running on Windows) is that I catch a ConnectionResetException (this is normal, the client has disconnected the socket) while sending the data to the client. I would want to read the latest data sent by the client (the message used to abort the call), but calling mysocket.recv() still raises a ConnectionResetException.
With a wireshark capture, I can clearly see that the message was properly sent by the client prior to TCP disonnection.
Any idea floks? Thanks!
VR

In order to understand what to do about this situation, you need to understand how a TCP connection is closed (see, e.g. this) and how the socket API relates to a clean shutdown (without fail, see this).
Your client is most likely calling close to terminate the connection. The problem with this is that there may be unread data in the socket receive queue or data arriving shortly from the other end that you will no longer be able to read, which is basically an error condition. To signal to the other end that data sent cannot be delivered to the receiving application, a reset is sent (well, technically, "SHOULD be sent" as per the RFC) and the TCP connection is abnormally terminated.
You might think that enabling SO_LINGER will help (many, many bits have been spilt over this so I won't elaborate further), but it won't solve the problem of unread data by the client causing the reset.
The client needs to instead call shutdown(SHUT_WR) to indicate that it is done sending, and then continue to call recv() until it reads 0 bytes indicating the other side is done sending. You may then call close().
Note that the Python 2 socket documentation states that
Depending on the platform, shutting down one half of the connection can also close the opposite half (e.g. on Mac OS X, shutdown(SHUT_WR) does not allow further reads on the other end of the connection).
This sounds like a bug to me. To get around this, you would have to send your cancel message, then keep reading until you get 0 bytes so that you know the server received the cancel message. You may then close the socket.
The Python 3.8 docs make no such disclaimer.

How python socket detect the server is closed when continue sending data to server?

I use python socket to send data to server, and the code like:
When I close the server, it will send the data twice, and then, it will go to the "except" code. If I set the SEND_INTERVAL too long, it will be a disaster. So, how to get the error immediately when the server is closed or downtime?

Nothing happens immediatly over the network. That's one thing.
Secondly the underlying OS will detect broken connections (and Python gets that info in the form of an exception), but usually this takes time. And that's why you still send messages even though the connection is already dead. But since OS operates on network layer (as opposed to the application layer) then there's an issue: the connection may be dead but OS may never detect this. For example this will happen when the server is dead but behind alive proxy.
Thirdly the most reliable way to know that a server is alive is when it sends something back to the client. So you should always .recv() (with timeout) after .sendall() call and the server should always .sendall() after .recv() (the request-response pattern, even when the response is a simple "I received message"). If you can't modify the server side and (in worst case) if the server doesn't send anything back to the client then there's no reliable way to know this.
That's why you need some form of framing/correctness protocol. Simple .sendall() won't do.

web.py - server and simultaneous socket connection

I'm working on an application in web.py which can send commands to a device through a website with buttons.
I know which buttons are pushed on my website and I get some parameters back in my python serverprogram. Python-program -> gets the basic commands out of an SQLite-database -> adds the received parameters. But I need to simultaneously set up a connection with the remote device through Ethernet (simple socket connection) to send these commands to the device. That's where I got stuck.
So I have the website working correctly, I also have a small seperate terminal program written to just make connection with the device and with a simple terminal interface to send commands. So basically I have the 2 major parts of the program working, but not simultaneously and I can't figure out how to fit them together.
I have been reading through some information to let the webserver run in a separate thread or maybe I have to connect and close the socket connection with the device each time I get information (command/parameters) from the website? Can somebody push me somewhat in the right direction?
NB: the server is running on a Raspberry Pi

Yes, your problem appears to be caused by socket connection not being thread-safe.
Each request to web.py server runs in its own thread, and if you want to access socket connection to your device, then you have to use locks or manage connection pool, if your device supports multiple connections.
To force web.py running in single thread mode please see the following answer:
Forcing single threaded request handling with web.py
Note that you don't have to lock all requests (as in that answer) and may put lock only on the part of the code where connection is used.

udp socket stops receiving data

I am learning network programing in python and I'm trying to write a Toy vpn forked from android sdk https://github.com/android/platform_development/tree/master/samples/ToyVpn.
My Toy vpn is https://github.com/325862401/ToyVPN.
It's only for Linux.
My home network is behind NAT.
I can use this vpn to surf the internet after connect to remote sever.
But about half an hour or some time later the client udp socket stops receiving any data but the server can receive and send normally.
At this point I must terminate my client and run ToyVpnClient again.
It works normal for some time until it stop receiving again.
Please help me check the client logs.
>2013-08-24 11:42:38 INFO receive data from the tunnel timeout`
you can see that when problem happens, the socket always sends, not receive.
> means send, < means receive
I want to know why the udp socket stops receiving data.
Is there any debug method to find the cause?
For now I've just used logging to debug my program.

Since you're trying your client on the Internet, there is the whole universe of possible causes represented by all the Internet newtwork.
There's not a simple way of debugging here. Possible causes could be of course a software error but also some intermediate network configurations between you and the remote server.
You should capture the udp traffic using the good wireshark or the commandline tcpdump between you and the server and check if you're stopping sending packets or if the server is stopping receiving them.
If you send packets but your server doesn't receive them ( tcpdump on the server ) then there is something on the network which decides to filter your packets. And if it's not on the server (firewall rules to rate limit packets for example or something like that) then there's nothing you can do to that without modifying the logic of your program. Like changing UDP port every X seconds or using a persistent tcp connection.

A udp socket is not stable and may become null once a scanning or other event occupy your network interface for a while (especially true on Android). Using tcp avoids this problem. If you wants to maintain a stable udp, keep monitoring the status of your udp socket; if it becomes null or any unusual things happens, delete this socket and create a new one. Put this reactivating staff in a loop so that your udp socket is always alive.

104, 'Connection reset by peer' socket error, or When does closing a socket result in a RST rather than FIN?

We're developing a Python web service and a client web site in parallel. When we make an HTTP request from the client to the service, one call consistently raises a socket.error in socket.py, in read:
(104, 'Connection reset by peer')
When I listen in with wireshark, the "good" and "bad" responses look very similar:
Because of the size of the OAuth header, the request is split into two packets. The service responds to both with ACK
The service sends the response, one packet per header (HTTP/1.0 200 OK, then the Date header, etc.). The client responds to each with ACK.
(Good request) the server sends a FIN, ACK. The client responds with a FIN, ACK. The server responds ACK.
(Bad request) the server sends a RST, ACK, the client doesn't send a TCP response, the socket.error is raised on the client side.
Both the web service and the client are running on a Gentoo Linux x86-64 box running glibc-2.6.1. We're using Python 2.5.2 inside the same virtual_env.
The client is a Django 1.0.2 app that is calling httplib2 0.4.0 to make requests. We're signing requests with the OAuth signing algorithm, with the OAuth token always set to an empty string.
The service is running Werkzeug 0.3.1, which is using Python's wsgiref.simple_server. I ran the WSGI app through wsgiref.validator with no issues.
It seems like this should be easy to debug, but when I trace through a good request on the service side, it looks just like the bad request, in the socket._socketobject.close() function, turning delegate methods into dummy methods. When the send or sendto (can't remember which) method is switched off, the FIN or RST is sent, and the client starts processing.
"Connection reset by peer" seems to place blame on the service, but I don't trust httplib2 either. Can the client be at fault?
** Further debugging - Looks like server on Linux **
I have a MacBook, so I tried running the service on one and the client website on the other. The Linux client calls the OS X server without the bug (FIN ACK). The OS X client calls the Linux service with the bug (RST ACK, and a (54, 'Connection reset by peer')). So, it looks like it's the service running on Linux. Is it x86_64? A bad glibc? wsgiref? Still looking...
** Further testing - wsgiref looks flaky **
We've gone to production with Apache and mod_wsgi, and the connection resets have gone away. See my answer below, but my advice is to log the connection reset and retry. This will let your server run OK in development mode, and solidly in production.

I've had this problem. See The Python "Connection Reset By Peer" Problem.
You have (most likely) run afoul of small timing issues based on the Python Global Interpreter Lock.
You can (sometimes) correct this with a time.sleep(0.01) placed strategically.
"Where?" you ask. Beats me. The idea is to provide some better thread concurrency in and around the client requests. Try putting it just before you make the request so that the GIL is reset and the Python interpreter can clear out any pending threads.

Don't use wsgiref for production. Use Apache and mod_wsgi, or something else.
We continue to see these connection resets, sometimes frequently, with wsgiref (the backend used by the werkzeug test server, and possibly others like the Django test server). Our solution was to log the error, retry the call in a loop, and give up after ten failures. httplib2 tries twice, but we needed a few more. They seem to come in bunches as well - adding a 1 second sleep might clear the issue.
We've never seen a connection reset when running through Apache and mod_wsgi. I don't know what they do differently, (maybe they just mask them), but they don't appear.
When we asked the local dev community for help, someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server. There's a bug there, but it is going to be hard to find it.

Normally, you'd get an RST if you do a close which doesn't linger (i.e. in which data can be discarded by the stack if it hasn't been sent and ACK'd) and a normal FIN if you allow the close to linger (i.e. the close waits for the data in transit to be ACK'd).
Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving?

I had the same issue however with doing an upload of a very large file using a python-requests client posting to a nginx+uwsgi backend.
What ended up being the cause was the the backend had a cap on the max file size for uploads lower than what the client was trying to send.
The error never showed up in our uwsgi logs since this limit was actually one imposed by nginx.
Upping the limit in nginx removed the error.