So I've got a python application that is using requests.post to make a post request with json headers, body info, etc.
Problem is that in my dictionary that gets sent as headers, I have a variable that often contains character groups like %25"" or "%2F", etc. I've seen this cause problems before if sent in body data, but that can be fixed by sending the body data as a sting rather than a dictionary. Haven't figured out how to make this work with the headers though, as you can't simply delimit the parameters with an ampersand like in body data.
How do I make sure that my cookie value is not altered in the process of the post request?
For instance, headers :
Host : blahblah.com
Connection : Keep-Alive
Cookie : My sensitive string with special characters
etc.
Note : Nothing server-side can be changed. The python application is being used for hired pentesting services.
A common technique for sending data that becomes a mess when transmitted is to encode it, especially as base64
Sender:
import base64
...
encoded_data = "base64:{}".format(base64.b64encode(data))
Receiver:
import base64
...
if encoded_data.startswith("base64:"):
data = base64.b64decode(encoded_data.split(':')[1])
Related
I am doing a straight forward request as follows.
import requests
def user_transactions():
url = 'https://webapi.coinfloor.co.uk/v2/bist/XBT/GBP/user_transactions/'
data = {'key':'value'}
r = requests.post(url, data=data, auth=("some_username", "some_password") )
print(r.status_code)
print(r.text)
return
Even though data= is optional in the documents.
https://www.w3schools.com/python/ref_requests_post.asp
If i comment out the data variable then the routine returns a
status_code=415 error.
If i include in the data variable then the routine returns a status_code=200 success.
I have tried to look this up, for example here:
Python request gives 415 error while post data , but with no answer.
The question is: Why is it the case that [1] fails but [2] works ?
Yes, data is optional on the python side. The requests library will happily send a empty request to the server, as you can see. If the argument was not optional, the program would crash before sending a request so there would be no status code.
However, the server needs to be able to process the request. If it does not like what you sent for whatever reason, it might send back a 4xx status code, or otherwise not do what you expect.
In this case, it throws an error that the data is in invalid format. How can a empty request be in invalid format? Because the format is specified in a header. If you supply a data argumet requests will send data in urlencoded format, and specify in the header what format the data is in. If the data is empty, the request will be empty but the header will still be there. This site apparently requires the header to specify a data format it knows.
You can solve this in two ways, giving an empty object:
r = requests.post(url, data={}, auth=("some_username", "some_password") )
Or by explicitly specifying the header:
r = requests.post(url, auth=(...), headers={'Content-Type': 'application/x-www-form-urlencoded'})
Side note: You should not be using W3Schools as a source. It is frequently inaccurate and often recommends bad practices.
I think you are mistaking the documentation of the requests.post function signature with API documentation. It is saying that data is a keyword argument, not that the API optionally takes data.
It depends on the API endpoint you are trying to use. That endpoint must require data to be sent with the request. If you look at the documentation for the API you are using, it will mention what needs to be sent for a valid request.
The request's content-type is application/json, but I want to get the request body bytes. Flask will auto convert the data to json. How do I get the request body?
You can get the non-form-related data by calling request.get_data() You can get the parsed form data by accessing request.form and request.files.
However, the order in which you access these two will change what is returned from get_data. If you call it first, it will contain the full request body, including the raw form data. If you call it second, it will typically be empty, and form will be populated. If you want consistent behavior, call request.get_data(parse_form_data=True).
You can get the body parsed as JSON by using request.get_json(), but this does not happen automatically like your question suggests.
See the docs on dealing with request data for more information.
To stream the data rather than reading it all at once, access request.stream.
If you want the data as a string instead of bytes, use request.get_data(as_text=True). This will only work if the body is actually text, not binary, data.
Files in a FormData request can be accessed at request.files then you can select the file you included in the FormData e.g. request.files['audio'].
So now if you want to access the actual bytes of the file, in our case 'audio' using .stream, you should make sure first that your cursor points to the first byte and not to the end of the file, in which case you will get empty bytes.
Hence, a good way to do it:
file = request.files['audio']
file.stream.seek(0)
audio = file.read()
If the data is JSON, use request.get_json() to parse it.
I am trying to test my update method on my viewset. The viewset is a modelviewset taken from drf. To update i would need to send a put request. As this is not always supported there are 2 ways to tell the server i am making a put request, the first which does not fit my needs is to use an additional field to form called _method and set it to put. As i need to post json data i need to use the second way, which uses the X-HTTP-Method-Override header.
To post my data in the testcase i use the following code:
header = {'X_HTTP_METHOD_OVERRIDE': 'PUT'}
response = client.post('/model/1/', content_type='application/json', data=post_data_clean, **header)
But unfortunately the result I get is {'detail':'Method POST not allowed.'}. I tested the behavior of the server using a addon (Postman) where i specified the X-HTTP-Method-Override header too. No exception is raised. I need to know now how to correctly pass the header to the django test client, otherwise testing will get really annoying over here.
You need to specify header as 'HTTP_X_HTTP_METHOD_OVERRIDE' instead of 'X_HTTP_METHOD_OVERRIDE' i.e. add HTTP_ at the beginning of the header.
header = {'HTTP_X_HTTP_METHOD_OVERRIDE': 'PUT'}
response = client.post('/model/1/', content_type='application/json', data=post_data_clean, **header)
From the Django documentation:
HTTP headers in the request are converted to META keys by converting
all characters to uppercase, replacing any hyphens with underscores
and adding an HTTP_ prefix to the name. So, for example, a header
called X-Bender would be mapped to the META key HTTP_X_BENDER.
Also, you can pass headers to the constructor of the Client:
from django.test import Client
client = Client(HTTP_USER_AGENT="Mozilla/5.0 ...", HTTP_X_USER_ID="982734")
This way every request will contain default headers.
PS: This approach is valid for DRF TestApiClient and ApiRequestFactory.
I am using mitmproxy for intercepting traffic. What I want is to be able to get the entire request and response in a string. I know that you have def response(context, flow) and that the HTTPFlow object has the request and response objects. What I want is simply something like this in a string
GET http://www.google-analytics.com/collect?v=1& HTTP/1.1
Header 1: value
Header 2: value
request body
and this
HTTP/1.1 301 Moved Permanently
Header 1: value
Header 2: value
response body
Now I've been trying this by joing the different parts and bits of the requests and responses but that is prone to errors. Is there a better way to do this?
Also, does mitm handle Gzip encoded response bodies?
If some one bumps into this; the above answer does not work for mitmproxy 4. Instead one can use this:
from mitmproxy.net.http.http1.assemble import assemble_request
def response(flow):
print(assemble_request(flow.request).decode('utf-8'))
You can get the whole request/response object as a string using flow.request.assemble(). If you want the request/response without transfer-encoding (gzip), you can use the decoded decorator:
from libmproxy.protocol.http import decoded
with decoded(flow.request):
data = flow.request.assemble()
Apart from that, you may find https://github.com/mitmproxy/mitmproxy/tree/master/examples very useful.
Note that the following pieces of code are used for a remote file inclusion exploit in a controlled environment (not doing anything malicious here).
I'm trying to perform a post request to a URL:
resp = requests.post("http://example.com/test/index.php",data=post_data,cookies=cookie,proxies=proxies,config={'encode_uri': False})
One of the data parameters is a url which is used for file inclusion, at the end it has a nullbyte:
http://mysite.org/simple-backdoor.php%00
But what requests is doing is re-encoding the nullbyte at the end, making it useless
http%3A%2F%2Fmysite.org%2Fsimple-backdoor.php%2500
I tried appending config={'encode_uri': False}) but this results in the same behavior. Does anyone have a clue how to disable this encoding or how to introduce a nullbyte character which gets encoded to %00?
Requests v2.0.0 onwards doesn't have (thus respect) encode_uri. It tries to encode data if data isn't a string.
Use a unicode null-byte instead of %00, OR manually encode every component of data and form data as a string.