Develop Reference

How to read from json-stream API endpoint in Python?

I am contacting an API endpoint, which requires you send an "Accept" parameter in the header, the value of which should be "application/x-json-stream", "text/event-stream".
If I don't add this to the header I get a 406 response.
If I do it seems like the output in the Python console is redirected somewhere. No commands in the console lead to new output anymore.
I am using the requests library and its get method. The endpoint is supposed to be some stream of information, perhaps it is something akin to a websocket.
The endpoint is here.

Python requests being fingerprinted?

I'm hacking together an amazon api and when only using python requests without proxying, it prompts for a captcha. When routing this python requests traffic through fiddler, it seems to pass without a problem. Is it possible that amazon is fingerprinting python requests and fiddler changes the fingerprint since it's a proxy?
I viewed headers sent from fiddler and python requests and they are the same.
There is no exra proxying/fiddler rules/filters set on fiddler to create a change.
To be clear, all mentioned proxying is only done locally, so it will not change the public ip address.
Thank you!

The reason is that websites are fingerprinting your requests with TLS hello package. There exist libraries like JA3 to generate a fingerprint for each request. They will intentionally block http clients like requests or urllib. If you uses a MITM proxy, because the proxy server create a new TLS connection with the server, the server only sees proxy server's fingerprint, so they will not block it.
If the server only blocks certain popular http libraries, you can simply change the TLS version, then you will have different fingerprint than the default one.
If the server only allows popular real-world browsers, and only accepts them as valid requests, you will need libraries that can simulate browser fingerprints, one of which is curl-impersonate and its python binding curl_cffi.
pip install curl_cffi
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser

'CORS request did not succeed' when uploading an image and Flask raises error (Firefox only)

I am trying to debug a CORS issue with my app. Specifically, it fails only in Firefox and, it seems, only with somewhat bigger files.
I am using flask on the backend and I am trying to upload a "faulty" image to my service. When I say faulty, I mean that the backend should reject the image with a 400 (only accept PNG, not JPG). Uploading a PNG of any size works ok. However, when I reject the JPG file, the browser request fails with Network error and I cannot capture the 400-error to display a user-friendly message. From the backend's side, everything is the same, always same headers returned, be it accepted or rejected request, POST or OPTIONS.
However, I have noticed that it only fails with somewhat bigger files. If I send a JPG of a few KBs, it works. If I send a JPG of a few MBs, it fails.
I have looked at everything
curl-ing the backend gives all the right headers
there are no OPTIONS requests logged by the browsers, but if there were, I've also checked those with curl for the right headers
I'm only using HTTP (not HTTPS), so no problems with certificates
I have disabled all extensions, so no possible blocking from the browser
maybe other things that I cannot remember
What can possibly be the cause? Note that everything works as expected
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8083/api/image. (Reason: CORS request did not succeed).

Well, after a couple of hours of trials, it appears this has nothing to do with CORS. This is probably the most confusing error message. To cite from Firefox' documentation on this (emphasis mine):
The HTTP request which makes use of CORS failed because the HTTP connection failed at either the network or protocol level. The error is not directly related to CORS, but is a fundamental network error of some kind. In many cases, it is caused by a browser plugin (e.g. an ad blocker or privacy protector) blocking the request.
So, this should actually indicate that the problem is on the backend, although it is very subtle.
Since in my code I am rejecting the request based on the trasmitted filename, I never read the content of the request if the name ends with .jpg. Instead, I reject it immediately. This is a problem with Flask's development server, which does not empty the input stream in such cases (issue here).
So, if you want to deal with this while keeping the development server, you should consume the input. In my case, I added a custom error handler, like so:
class BadRequestError(ValueError):
"""Raised when a request does not conform to the protocol"""
pass
#app.errorhandler(BadRequestError)
def bad_request_handler(error):
# throw away the request data to avoid closing the connection before receiving all of it
# http://flask.pocoo.org/snippets/47/
_ = request.data
_ = request.form
response = jsonify(str(error))
response.status_code = 400
return response
and then, in the code, I always raise BadRequestError('...'), instead of just returning a 400-response.

How to transform CoAP to HTTP?

I searched but did not found any example showing on how to convert CoAP request or response to HTTP request.
Basically what I want to do is CoAP request POST some data from device to a server which will translate it and do HTTP request POST to other server to be save inside the database.
While the part to save the data is not a major problem right now, I did not managed to find any example script showing how to convert from CoAP to HTTP.
I already looked at coapthon , aiocoap but since aiocoap requires python 3.5,(I use python 2.7) that left me with coapthon. Unfortunately coapthon only has HTTP to CoAP proxy while CoAP to HTTP is still in development.
If anyone know other project regarding this or has any opinion on how to solve this, I am glad if you can share it. Thank you.

That is called Protocol Interoperability. You Need a CoAP - HTTP and HTTP - CoAP proxy that can translate the messages between them.
Here is californium-proxy on GitHub, I am using it already. Here is the example that shows how to use it.

Python-Scapy HTTP Traffic Manipulation

I need to intercept an HTTP Response packet from the server and replace it with my own response, or at least modify that response, before it arrives to my browser.
I'm already able to sniff this response and print it, the problem is with manipulating/replacing it.
Is there a way to do so wiht scapy library ?
Or do i have to connect my browser through a proxy to manipulate the response ?

If you want to work from your ordinary browser, then you need proxy between browser and server in order to manipulate it. E.g. see https://portswigger.net/burp/ which is a proxy specifically created for penetration testing with easy replacing of responses/requests (which is sriptable, too).
If you want to script all your session in scapy, then you can create requests and responses to your liking, but response does not go to the browser. Also, you can record ordinary web session (with tcpdump/wireshark/scapy) into pcap, then use scapy to read pcap modify it and send similar requests to the server.