I have a google function that gets data of few websites
with http request.
I want to send the data back as it comes in and not wait for all of it.
I was thinking streaming the data should work.
But I have no idea how to do it
not on server or client side.
I already trade 20 example of google and non worked for me.
Tried:
response.write
response.send
and
import json
import requests
r = requests.get('https://httpbin.org/stream/20', stream=True)
for line in r.iter_lines():
# filter out keep-alive new lines
if line:
decoded_line = line.decode('utf-8')
print(json.loads(decoded_line))
Not sure which side I am doing wrong the client or server but this not working.
Anyone know how to do streaming data on python from http request / to http res?
Related
There's a website that has a button which downloads an Excel file. After I click, it takes around 20 seconds for the server API to generate the file and send it back to my browser for download.
If I monitor the communication after I click the button, I can see how the browser sends a POST request to a server with a series of headers and form values.
Is there a way that I can simulate a similar POST request programmatically using Python, and retrieve the Excel file after the server sends it over?
Thank you in advance
The requests module is used for sending all kinds of request types.
requests.post sends the post requests synchronously.
The payload data can be set using data=
The response can be accessed using .content.
Be sure to check the .status_code and only save on a successful response code
Also note the use of "wb" inside open, because we want to save the file as a binary instead of text.
Example:
import requests
payload = {"dao":"SampleDAO",
"condigId": 1,
...}
r = requests.post("http://url.com/api", data=payload)
if r.status_code == 200:
with open("file.save","wb") as f:
f.write(r.content)
Requests Documentation
I guess You could similarly do this:
file_info = request.get(url)
with open('file_name.extension', 'wb') as file:
file.write(file_info.content)
I honestly do not know how to explain this tho since I have little understanding how it works
I'm making a request to an endpoint that returns a PDF as streamable binary. This endpoint uses mutual TLS authentication so when I hit the endpoint I must send a client certificate. To achieve this I am using https://pypi.org/project/requests-pkcs12/ which supports the Python requests library.
I would like to download this PDF from the client.
Ideally when the end user clicks 'download' it hits the endpoint and directly streams the data and downloads it.
I am struggling to do this in one single step.
Currently what I'm doing is downloading the PDF to a file, then sending this file back to the client. Writing to the file is slow and I'd like to avoid the download-to-file step and simply send a streaming response back somehow.
Is there a way to stream this directly using Python's Request?
#hit the mutual tls authenticated endpoint
response = post(f'{url}, stream=True, pkcs12_filename=client_certificate_path,
pkcs12_password=client_certificate_passphrase)
#Write the returned data to a file
with open('/tmp/newfile.pdf', 'wb') as f:
f.write(response.content)
#Send the file back to client with Django's FileResponse
return FileResponse(open('/tmp/newfile.pdf', 'rb'))
While I am using Django which seems to handle this problem nicely with StreamingHttpResponse, I was unable to get this working as it doesn't allow me to send a client certificate and password protected client certificate key.
I am writing a small application that interprets the http response of a request. I am writing the application in python. I have not found anything that allows me to send the body + headers stored in one file. I can send certain parts like the headers but not the entire request.
For example, if the request is:
GET /index.html HTTP/1.1
Host: localhost
Cookie: bob=lemon
I want to send this entire request in one go. How would I do this in python?
Check out the python requests library. https://requests.readthedocs.io/en/master/user/quickstart/#make-a-request
For the request above it would look something like
import requests
url = 'http://localhost:[YOUR PORT HERE]/'
cookies = {bob : lemon}
r = requests.get(url, cookies=cookies)
To check if you had a successful request you should get a 200 code from.
r.status_code
Check out the library for more, it is very extensive.
I need to get some json data using API
import requests
url = 'https://example.com/api/some-info/'
response = requests.get(url)
print(response.text) # Here is JSON needeed
And everything fine, except I need to make such requests very often, and API provider says:
You'll be banned if you make more than 5 requests per second, so use
sockets
So, how can I make this work via sockets?
Big thx for advices.
I have a google app engine request handler that processes multipart/form-data request from file uploads. It works fine with a html form but with the flash uploader it seems that no parameters are passed through.
Here is the data I have put together:
https://gist.github.com/martinheidegger/b8ee4982eb9580b9baa9
It contains the two http requests, one working, one broken together with the log output for logging.info("%s" % self.request.POST);
What am I missing? Why does the flash request not "arrive" at the GAE?
Are you receiving the POST request but missing all the POST data?
It could be due to the HTTP headers were not sent properly in Flash.
Try to read and parse directly from the HTTP request body and you should get something.