I wrote a Django view that responses ether a text/html or a application/json depending on request.is_ajax().
So far so good, but when I use my browsers history buttons, I end up getting a JSON response rather than the HTML.
I can't figure out the problem. It's true an jQuery ajax request is getting the same url after the page was loaded, but that shouldn't end up in the history, or should it?
Thanks, Joe
If you send different content depending on request.is_ajax(), you need to send Vary: X-Requested-With to the browser. That way, the browser will be able to distinguish the two kinds of response based on the value of the X-Requested-With header on the request. You can do that via:
from django.views.decorators.vary import vary_on_headers
#vary_on_headers('X-Requested-With')
def yourview(request, ...):
pass
Related
Hay ! I am new here so let me describe clearly my issue,Please Ignore mistakes.
I am making request on a page which literlaly works on js.
Acually its the page of paytm payemnt response through UPI.
When ever i do the requests the response is {'POLL_STATUS':"STOP_POLLING"}
But the problem is the reqest is giving this response while the browser is giving another response with loaded html.
I tried everyting like stopeed redirects and printing raw content nothing works.
I just think may be urllib post request may be work but i do not know the uses.
Can anyone please tell me how to get the exact html response as the browser gives.
Note[0]:Please dont provide answer of selenium because this issue having in the middle of my script.
Note[1]:Friendly answer appriciated.
for i in range(0,15):
resp_check_transaction=self.s.post("https://secure.website.in/theia/upi/transactionStatus?MID="+str(Merchant_ID)+"&ORDER_ID="+str(ORDER_ID),headers=check_transaction(str(ORDER_ID)),data=check_transaction_payload(Merchant_ID,ORDER_ID,TRANSID,CASHIERID))
print(resp_check_transaction.text)
resp_check_transaction=resp_check_transaction.json()
if resp_check_transaction['POLL_STATUS']=="STOP_POLLING":
print("Breaking looop")
break
time.sleep(4)
self.clear_header()
parrms={
"MID": str(Merchant_ID),
"ORDER_ID": str(ORDER_ID)
}
resp_transaction_pass=requests.post("https://secure.website.in/theia/upi/transactionStatus",headers=transaction_pass(str(ORDER_ID)),data=transaction_pass_payload(CASHIERID,UPISTATUSURL,Merchant_ID,ORDER_ID,TRANSID,TXN_AMOUNT),params=parrms,allow_redirects=True)
print("Printing response")
print(resp_transaction_pass.text)
print(resp_transaction_pass.content)
And in the web browser its showing that Status Code: 302 Moved Temporarily in the bank response of Bank response. :(
About the 302 status code
You mention that the web browser is sends a 302 status code in response to the request. In the simplest terms the 302 status code is just the web servers way of saying "Hey I know what you're looking for but it is actually located at this other URL.".
Basically all modern browsers and HTTP request libraries like Python's Requests will automatically follow a 302 redirect and act as though you send the request to the new URL instead. (Your browser's developer tools may show that a 302 redirect has happened but as far as the JavaScript is concerned it just got a normal 200 response).
If you really want to see if your Python script receives a 302 status you can do so by setting the allow_redirects option to False, but this means you will manually have to get the stuff from the new URL.
import requests
r1 = requests.get('https://httpstat.us/302', allow_redirects=False)
r2 = requests.get('https://httpstat.us/302', allow_redirects=True)
print('No redirects:', r1.status_code) # 302
print('Redirects on:', r2.status_code) # 200 (status code of page it redirects to)
Note that allow_redirects is already set to True by default, I just wanted to make the example a bit more verbose so the difference is obvious.
So why is the response content different?
So even though the browser and the Requests library are both automatically following the 302 redirect the response they get is still different, you didn't share any screenshots of the browsers requests or responses so I can only give a few educated guesses but it boils down to the fact that the request made by your Python code is somehow different from the JavaScript loaded by the web browser.
Some things to consider:
Are you sure you are using the he correct HTTP method? Is the browser also making a POST request?
If so are you sure the body of the request is the same/of the same format as the one sent by the web browser?
Perhaps the browser has a session cookie it is sending along with the request (Note this usually not explicitly said in the JS but happens automatically).
Alternatively the JS might include some API key/credentials in the HTTP auth header (this should be explicitly visible in JS).
Although unlikely it could be that whatever API you're trying to query is trying to block reverse engineering attempts by blocking the Requests library's user agent string.
Luckily all of these differences can be easily examined with some print statements and your browser's developer tools :p.
I am adding a custom route handler to a Playwright page and I am trying to inspect the request passed into the handler. For context here is a following code snippet:
def handler(route: Route, request: Request):
# Do things with `request`
...
await page.route('**/*', handler=handler)
For POST/PUT requests with a Content-Type of application/json, I have been able to successfully inspect the payload by using request.post_data_buffer. However, when the Content-Type is multipart/form-data, I have not been able locate where I can get the form data. All of the post_data, post_data_buffer, and post_data_json properties have a value of None, and I couldn't see anything else in the documentation which could contain the form_data.
The issue had nothing to do with really any details in my original post. The issue was I was using Chromium, and it is a known bug that post_data does not contain file/blob data.
I am using Python flask. I have a POST request with some payload coming on say:
abc.com/hello/hello1
I want to redirect this (302) to:
xyz.com/hello/hello1
only changing the domain name while keeping the remaining part as it is and also the payload. Is there a simple way to do this?
As per RFC, redirect requests (all 3xx) cannot contain request data or headers. You will miss the payload, supplied via POST in original request.
There are two possible workaround I could think of right away:
Give the client new URL, and implement further logic on client side;
Create a proxy handler on backend, which will do a request by itself and give the answer back as it's own.
EDIT: As per Andrejs Cainikovs's comment below, this would not work for a POST with payload.
In your endpoint, get the url that was used using request.url (see request API here for more options). Then you can rewrite it and make a redirect.
newUrl = "xyz.com/" + route
return redirect(newUrl, code=302)
How to scrape the list of product from this page with scrapy?
I have tried the ajax request url the browser sends:
https://www.amazon.cn/gp/profile/A34PAP6LGJIN6N/more?next_batch_params%5Breview_offset%5D=10&_=1469081762384
but it returns 404.
You need to replicate the headers you see in the request.
If you inspect the response headers you can see:
from this you need to update your scrapy.Request.headers attribute. With few of these values. For the most part you can skip the Cookie since scrapy manages this one by itself and usually for ajax requests like this it's meaningless.
For this case I've manage to get a successful response by replicating only X-Requested-With header. This header is used to indicate that ajax request is happening.
You can actually test out and engineer this real time:
scrapy shell <url>
# gives you 403
request.headers.update({'X-Requested-With': 'XMLHttpRequest'})
request.headers.update({'User-Agent': <some user agent>})
fetch(request)
# now the request is redownloaded and it's 200!
i have form.html and form.py files. In form.html file i have a text box and a button when i enter some keyword in the text box and submit it, it executes form.py based on the keyword entered and it displays latest tweets from twitter. now my requirement is to refresh broswer automatically for every 5 sec to get latest tweets.
i have tried meta http-equiv="refresh" content='5' but it is refreshing only form.py without taking keyword from form.html
is there any way to do it, please suggest me....
When you 1st call form.py, send an header to set a cookie with the keyword, then on subsequent refreshes you should get the cookie and not the keyword as post daya, so read it from there.
In case both post data and cookie are present you should ignore the cookie and treat it as just post data and set the new cookie in the HTTP headers.
To set a cookie you must send an header like this
Set-Cookie: name=value
or like this to have an expiration
Set-Cookie: name2=value2; Expires=Wed, 09-Jun-2021 10:18:14 GMT
I think in the 1st case the cookie would be treated as a session cookie, and be removed when you close the browser.
Check the python doc to see how to set and read headers with the CGI module.