Develop Reference

Does setting socket timeout cancel the initial request

I have a request that can only run once. At times, the request takes much longer than it should.
If I were to set a default socket timeout value (using socket.setdefaulttimeout(5)), and it took longer than 5 seconds, will the original request be cancelled so it's safe to retry (see example code below)?
If not, what is the best way to cancel the original request and retry it again ensuring it never runs more than once.
import socket
from googleapiclient.discovery import build
from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_exception_type
#retry(
retry=retry_if_exception_type(socket.timeout),
wait=wait_fixed(4),
stop=stop_after_attempt(3)
)
def create_file_once_only(creds, body):
service = build('drive', 'v3', credentials=creds)
file = service.files().create(body=body, fields='id').execute()
socket.setdefaulttimeout(5)
create_file_once_only(creds, body)

It's unlikely that this can be made to work as you hope. An HTTP POST (as with any other HTTP request) is implemented by sending a command to the web server, then receiving a response. The python requests library encapsulates a lot of tedious parts of that for you, but at the core, it's going to do a socket send followed by a socket recv (it may of course require more than one send or recv depending on the size of the data).
Now, if you were able to connect to the web server initially (again, this is taken care of for you by the requests library but typically only takes a few milliseconds), then it's highly likely that the data in your POST request has long since been sent. (If the data you are sending is megabytes long, it's possible that it's only been partially sent, but if it is reasonably short, it's almost certainly been sent in full.)
That in turn means that in all likelihood the server has received your entire request and is working on it or has enqueued your request to work on it eventually. In either case, even if you break the connection to the server by timing out on the recv, it's unlikely that the server will actually even notice that until it gets to the point in its execution where it would be sending its response to your request. By that point, it has probably finished doing whatever it was going to do.
In other words, your socket timeout is not going to apply to the "HTTP request" -- it applies to the underlying socket operations instead -- and almost certainly to the recv part on the tail end. And just breaking the socket connection doesn't cancel the HTTP request.
There is no reliable way to do what you want without designing a transactional protocol with the close cooperation of the HTTP server.
You could do something (with the cooperation of the HTTP server still) that could do something approximating it:
Create a unique ID (UUID or the like)
Send a request to the server that contains that UUID along with the other account info (name, password, whatever else)
The server then only creates the account if it hasn't already created an account with the same unique ID.
That way, you can request the operation multiple times, but know that it will only actually be implemented once. If asked to do the same operation a second time, the server would simply respond with "yep, already did that".

HTTP status code 200 vs 202

I have a Python+requests script.
Steps that script should execute:
send file to DB;
approve this file (change file state in DB);
download file.
The constraint:
Only approved file could be downloaded
My code:
requests.post(url_to_create, files={"file": open(path_to_file)})
requests.post(url_to_approve, data={'id': file_id})
requests.get(url_to_download, data={'id': file_id})
The problem:
This code works almost perfectly, but sometimes I get no file. I found that the first and the third requests return 200 status code while the second returns 202. As I understand (tell me if I wrong) status 202: Accepted means that server accept request and return status code without actual request completion
The question:
Does it mean that request to download could be send even if request to approve hasn't been already completed and, if it is so, how can I wait till approval-request completed before send download-request?

It depends on your server implementation and your server decides how 202 will be processed.
202 Accepted
The request has been accepted for processing, but the processing has
not been completed. The request might or might not eventually be acted
upon, as it might be disallowed when processing actually takes place.
There is no facility for re-sending a status code from an asynchronous
operation such as this.
The 202 response is intentionally non-committal. Its purpose is to
allow a server to accept a request for some other process (perhaps a
batch-oriented process that is only run once per day) without
requiring that the user agent's connection to the server persist until
the process is completed. The entity returned with this response
SHOULD include an indication of the request's current status and
either a pointer to a status monitor or some estimate of when the user
can expect the request to be fulfilled.
If response body is empty, makes sense to check response headers that should have additional information.
Reference - https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

How do I terminate a long-running Django request if the XHR gets an abort()?

I initiate a request client-side, then I change my mind and call xhr.abort().
How does Django react to this? Does it terminate the thread somehow? If not, how do I get Django to stop wasting time trying to respond to the aborted request? How do I handle it gracefully?

Due to how http works and that you usually got a frontend in front of your django gunicorn app processes (or uswgi etc), your http cancel request is buffered by nginx. The gunicorns don't get a signal, they just finish processing and then output whatever to the http socket. But if that socket is closed it will have an error (which is caught as a closed connection and move one).
So it's easy to DOS a server if you can find a way to spawn many of these requests.
But to answer your question it depends on the backend, with gunicorn it will keep going until the timeout.

Just think of the Web as a platform for building easy-to-use, distributed, loosely couple systems, with no guarantee about the availability of resources as 404 status code suggests.
I think that creating tightly coupled solutions such as your idea is going against web principles and usage of REST. xhr.abort() is client side programming, it's completely different from server side. It's a bad practice trying to tighten client side technology to server side internal behavior.
Not only this is a waste of resources, but also there is no guarantee on processing status of the request by web server. It may lead to data inconsistency too.
If your request generates no server-side side effects for which the client
can be held responsible. It is better just to ignore it, since these kind of requests does not change server state & the response is usually cached for better performance.
If your request could cause changes in server state or data, for the sake of data consistency you can check whether the changes have taken effect or not using an API. In case of affection try to rollback using another API.

Python HTTP client with request pipelining

The problem: I need to send many HTTP requests to a server. I can only use one connection (non-negotiable server limit). The server's response time plus the network latency is too high – I'm falling behind.
The requests typically don't change server state and don't depend on the previous request's response. So my idea is to simply send them on top of each other, enqueue the response objects, and depend on the Content-Length: of the incoming responses to feed incoming replies to the next-waiting response object. In other words: Pipeline the requests to the server.
This is of course not entirely safe (any reply without Content-Length: means trouble), but I don't care -- in that case I can always retry any queued requests. (The safe way would be to wait for the header before sending the next bit. That'd might help me enough. No way to test beforehand.)
So, ideally I want the following client code (which uses client delays to mimic network latency) to run in three seconds.
Now for the $64000 question: Is there a Python library which already does this, or do I need to roll my own? My code uses gevent; I could use Twisted if necessary, but Twisted's standard connection pool does not support pipelined requests. I also could write a wrapper for some C library if necessary, but I'd prefer native code.
#!/usr/bin/python
import gevent.pool
from gevent import sleep
from time import time
from geventhttpclient import HTTPClient
url = 'http://local_server/100k_of_lorem_ipsum.txt'
http = HTTPClient.from_url(url, concurrency=1)
def get_it(http):
print time(),"Queueing request"
response = http.get(url)
print time(),"Expect header data"
# Do something with the header, just to make sure that it has arrived
# (the greenlet should block until then)
assert response.status_code == 200
assert response["content-length"] > 0
for h in response.items():
pass
print time(),"Wait before reading body data"
# Now I can read the body. The library should send at
# least one new HTTP request during this time.
sleep(2)
print time(),"Reading body data"
while response.read(10000):
pass
print time(),"Processing my response"
# The next request should definitely be transmitted NOW.
sleep(1)
print time(),"Done"
# Run parallel requests
pool = gevent.pool.Pool(3)
for i in range(3):
pool.spawn(get_it, http)
pool.join()
http.close()

Dugong is an HTTP/1.1-only client which claims to support real HTTP/1.1 pipelining. The tutorial includes several examples on how to use it, including one using threads and another using asyncio.
Be sure to verify that the server you're communicating with actually supports HTTP/1.1 pipelining—some servers claim to support HTTP/1.1 but don't implement pipelining.

I think txrequests could get you most of what you are looking for, using the background_callback to en-queue processing of responses on a separate thread. Each request would still be it's own thread but using a session means by default it would reuse the same connection.
https://github.com/tardyp/txrequests#working-in-the-background

It seems you are running python2.
For python3 >= 3.5
you could use async/await loop
See asyncio
Also, there is a library built on top for better, easier use
called Trio, available on pip.
Another thing I can think of is multiple threads with locks.
I will think on how to better explain this or could it even work.

What happens if a HTTP connection is closed while AppEngine is still running

The real question is if Google App Engine guarantees it would complete a HTTP request even if the connection is no longer existed (such as terminated, lost Internet connection).
Says we have a python script running on Google App Engine:
db.put(status = "Outputting")
print very_very_very_long_string_like_1GB
db.put(status = "done")
If the client decides to close the connection in the middle (too much data coming...), will status = "done" be executed? Or will the instance be killed and all following code be ignored?

If the client breaks the connect, the request will continue to execute. Unless it reaches the deadline of 60 seconds.

GAE uses Pending Queue to queue up requests. If client drops connection and request is already in the queue or being executed, then it will not be aborted. Afaik all other http servres behave the same way.
This will be a real problem when you make requests that change state (PUT, POST, DELETE) on mobile networks. On Edge networks we see about 1% of large requests (uploads, ~500kb) dropped in the middle of request executing (exec takes about 1s): e.g. server gets the data and processes it, but client does not receive response, triggering it to retry. This could produce duplicate data in the DB, breaking integrity of this data.
To alleviate this you will need to make your web methods idempotent: repeating the same method with same arguments does not change state. The easiest way to achieve this would be one of:
Hash relevant data and compare to existing hashes. In you case it would be the string you are trying to save (very_very_very_long_string_like_1GB). You can do this server side.
Client provides unique request-scoped ID, and sever checks if this ID was already used.