I have a daily cron which handles some of the recurring events at my app, and from time to time I notice one weird error that pops up in logs. The cron, among other things, does a validation of some codes, and it uses the webapp running on the same server, so the validation request is made via POST request with some data.
url = 'https://example.com/validate/'
payload = {'pin': pin, 'sku': sku, 'phone': phone, 'AR': True}
validation_post = requests.post(url, data=payload)
So, this makes the actual request and I log the response. From time to time, and recently up to 50% of the request, the response contains the following message from nginx:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method GET is not allowed for the requested URL.</p>
So, the actual request was made using the GET method, not the POST as it was instructed in the code. In the nginx access.log I can see that entry:
123.123.123.123 - - [18/Feb/2015:12:26:50 -0500] "GET /validate/ HTTP/1.1" 405 182 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-37-generic"
And the uwsgi log for the app shows the similar thing:
[pid: 6888|app: 0|req: 1589/58763] 123.123.123.123 () {40 vars in 613 bytes} [Mon Apr 6 11:42:41 2015] GET /validate/ => generated 182 bytes in 1 msecs (HTTP/1.1 405) 4 headers in 234 bytes (1 switches on core 0)
So, everything points out that the actual request was not made using the POST. The app route that handles this code is simple, and this is an excerpt:
#app.route('/validate/', methods=['POST'])
#login_required
def validate():
if isinstance(current_user.user, Sales):
try:
#do the stuff here
except Exception, e:
app.logger.exception(str(e))
return 0
abort(403)
The app route can fail, and there are some returns inside the try block, but even if those fails or there is an expcetion, there is nothing that could raise the 405 error code in this block, only 403 which rarely happens since I construct and login the user manually from the cron.
I have found similar thing here but the soultion there was that there was a redirect from HTTP to HTTPS version, and I also have that redirect present in the server, but the URL the request is being made has the HTTPS in it, so I doubt this is the cause.
The stack I am running this on is uwsgi+nginx+flask. Can anyone see what might be causing this? To repeat, its not happening always, so sometimes its working as expected, sometimes not. I recently migrated from apache and mod_wsgi to this new stack and from that point I have started encontering this error; can't recally ever seeing it on apache environment.
Thanks!
The only time we ever change a POST request to a GET is when we're handling a redirect. Depending on the redirect code, we will change the request method. If you want to be sure that we don't follow redirects, you need to pass allow_redirects=False. That said, you need to figure out why your application is generating redirects (including if it's redirecting to HTTP or to a different domain, or using a specific status code).
Not sure if it's by design, but removing the forward slash at the end of the URL fixed it for me:
url = 'https://example.com/validate/' # remove the slash
payload = {'pin': pin, 'sku': sku, 'phone': phone, 'AR': True}
validation_post = requests.post(url, data=payload)
Related
Hay ! I am new here so let me describe clearly my issue,Please Ignore mistakes.
I am making request on a page which literlaly works on js.
Acually its the page of paytm payemnt response through UPI.
When ever i do the requests the response is {'POLL_STATUS':"STOP_POLLING"}
But the problem is the reqest is giving this response while the browser is giving another response with loaded html.
I tried everyting like stopeed redirects and printing raw content nothing works.
I just think may be urllib post request may be work but i do not know the uses.
Can anyone please tell me how to get the exact html response as the browser gives.
Note[0]:Please dont provide answer of selenium because this issue having in the middle of my script.
Note[1]:Friendly answer appriciated.
for i in range(0,15):
resp_check_transaction=self.s.post("https://secure.website.in/theia/upi/transactionStatus?MID="+str(Merchant_ID)+"&ORDER_ID="+str(ORDER_ID),headers=check_transaction(str(ORDER_ID)),data=check_transaction_payload(Merchant_ID,ORDER_ID,TRANSID,CASHIERID))
print(resp_check_transaction.text)
resp_check_transaction=resp_check_transaction.json()
if resp_check_transaction['POLL_STATUS']=="STOP_POLLING":
print("Breaking looop")
break
time.sleep(4)
self.clear_header()
parrms={
"MID": str(Merchant_ID),
"ORDER_ID": str(ORDER_ID)
}
resp_transaction_pass=requests.post("https://secure.website.in/theia/upi/transactionStatus",headers=transaction_pass(str(ORDER_ID)),data=transaction_pass_payload(CASHIERID,UPISTATUSURL,Merchant_ID,ORDER_ID,TRANSID,TXN_AMOUNT),params=parrms,allow_redirects=True)
print("Printing response")
print(resp_transaction_pass.text)
print(resp_transaction_pass.content)
And in the web browser its showing that Status Code: 302 Moved Temporarily in the bank response of Bank response. :(
About the 302 status code
You mention that the web browser is sends a 302 status code in response to the request. In the simplest terms the 302 status code is just the web servers way of saying "Hey I know what you're looking for but it is actually located at this other URL.".
Basically all modern browsers and HTTP request libraries like Python's Requests will automatically follow a 302 redirect and act as though you send the request to the new URL instead. (Your browser's developer tools may show that a 302 redirect has happened but as far as the JavaScript is concerned it just got a normal 200 response).
If you really want to see if your Python script receives a 302 status you can do so by setting the allow_redirects option to False, but this means you will manually have to get the stuff from the new URL.
import requests
r1 = requests.get('https://httpstat.us/302', allow_redirects=False)
r2 = requests.get('https://httpstat.us/302', allow_redirects=True)
print('No redirects:', r1.status_code) # 302
print('Redirects on:', r2.status_code) # 200 (status code of page it redirects to)
Note that allow_redirects is already set to True by default, I just wanted to make the example a bit more verbose so the difference is obvious.
So why is the response content different?
So even though the browser and the Requests library are both automatically following the 302 redirect the response they get is still different, you didn't share any screenshots of the browsers requests or responses so I can only give a few educated guesses but it boils down to the fact that the request made by your Python code is somehow different from the JavaScript loaded by the web browser.
Some things to consider:
Are you sure you are using the he correct HTTP method? Is the browser also making a POST request?
If so are you sure the body of the request is the same/of the same format as the one sent by the web browser?
Perhaps the browser has a session cookie it is sending along with the request (Note this usually not explicitly said in the JS but happens automatically).
Alternatively the JS might include some API key/credentials in the HTTP auth header (this should be explicitly visible in JS).
Although unlikely it could be that whatever API you're trying to query is trying to block reverse engineering attempts by blocking the Requests library's user agent string.
Luckily all of these differences can be easily examined with some print statements and your browser's developer tools :p.
I am trying to connect to a Confluence page using the python wrapper on the API (as I am not familiar with any of this) but I keep getting the following error:
requests.exceptions.HTTPError: 401 Client Error
I know that people talk about this being caused by the necessity of using an API token but the page runs on an old version of Confluence and I have been told that we cannot use access tokens.
So has anyone any other idea? Here's a small code:
from atlassian import Confluence
confluence = Confluence(
url='https://address',
username='name',
password='pwd'
)
confluence.create_page(
space='Test',
title='A title',
body='something')
I have tried to use an older version of atlassian-python-api just in case there was some conflict but it got me the same error.
Your code looks ok. Authenticating to Confluence using Basic Auth should work without generating an API token, afaik.
The 401 status definitely suggests a problem with the authentication though. The obvious reason for this would be of course wrong credentials, but I assume that you have double checked that the credentials work when interactively logging into confluence with a browser.
To get a better sense of the error, you can import logging to debug your requests and response:
from atlassian import Confluence
import logging
logging.basicConfig(filename='conf_connect.log', filemode='w', level=logging.DEBUG)
try:
c = Confluence(url='https://conf.yoursystem.com', username='name', password='pwd')
# atlassian API does not raise error on init if credentials are wrong, this only happens on the first request
c.get_user_details_by_username('name')
except Exception as e:
logging.error(e)
The Confluence module internally also uses logging, so the requests and responses will appear in your conf_connect.log logfile:
DEBUG:atlassian.rest_client:curl --silent -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' 'https://conf.yoursystem.com/rest/api/user?username=name'
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conf.yoursystem.com:443
DEBUG:urllib3.connectionpool:https://conf.yoursystem.com:443 "GET /rest/api/user?username=name HTTP/1.1" 401 751
DEBUG:atlassian.rest_client:HTTP: GET rest/api/user -> 401
DEBUG:atlassian.rest_client:HTTP: Response text -> <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Basic Authentication Failure - Reason : AUTHENTICATED_FAILED</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.33</h3></body></html>
ERROR:root:401 Client Error: for url: https://conf.yoursystem.com/rest/api/user?username=name
The response body may include some information on the reason:
HTTP Status 401 – UnauthorizedType Status ReportMessage Basic Authentication Failure - Reason : AUTHENTICATED_FAILEDDescription The request has not been applied because it lacks valid authentication credentials for the target resource.
The reason AUTHENTICATED_FAILED suggests something is likely wrong with your credentials. If you want to dig deeper into that, you can use this SO answer to also display the headers that are being sent with your request.
However, if your reason is AUTHENTICATION_DENIED the problem is likely the following: If you have too many failed authentication attempts in a row, a CAPTCHA challenge is triggered, and this error will occur until the Failed Login Count is reset. This can easily happen when you are developing a script and test it frequently. To remedy this, either open a browser and manually (re-)logon to Confluence, completing the CAPTCHA, or resolve it from the Confluence User Management.
I've defined custom templates for errors 400, and 404 for my Django project. When I try to access the production version of my site, the error 404 template is correctly loaded for missing pages. However, if I send a bad request to my Apache/Django server (e.g. http://mysite.example.com/%), the template for the error 400 is not loaded, instead, the regular Apache error page is rendered:
Bad Request
Your browser sent a request that this server could not understand.
Apache/2.4.18 (Ubuntu) Server at mysite.example.com Port 80
Is apache relaying this request to Django at all, or do I need to define handler400 in my Django project in order for this to work (though I didn't have to do that for the 404.html)?
The crucial point here is that your apache is acting as a proxy for your usgi server. It's forwarding all valid requests to usgi, a request for a non existent request is a valid request as far as apache is concerned and needs the forwarded to the django router - which will find that the url mapping does not exist and raise a 404 error. This error is done internally by django and results in the django 404 page being shown.
Some requests, most notably the django rest framework produce 400 responses internally when the serializers fail to validate the incoming json request. Those will also result in the django 400 page being shown.
However if the request itself is malformed, it will never be forwarded to the usgi server and django will never see it. it will be handled internally by apache hence the reason that the apache 400 html is shown.
The simplest solution would be to replace all the apache error pages with the corresponding django one (if these are templates, render them and save the html)
I have a JavaScript bookmarklet that POSTs information to a (Flask powered) server while the user is on some other page (i.e. not one on my server). I don't want to interrupt the user's browsing by hijacking their session with my server response.
My initial thought was that I could suppress the HTTP response from Flask somehow; prevent it from sending anything to the client so they aren't mysteriously redirected. I was hoping I could do this by perhaps having a null return from a view.
I then thought that might be some HTTP response that lets the client know the information was successfully submitted, but will leave the client on their current page. Suppose a header value like "Here is the result of your request, but you should not alter your current display"?
To answer your amended question, yes there is such a response. From RFC 2616-section 10 (emphasis added):
10.2.5 204 No Content
The server has fulfilled the request but does not need to return an
entity-body, and might want to return updated metainformation. The
response MAY include new or updated metainformation in the form of
entity-headers, which if present SHOULD be associated with the
requested variant.
If the client is a user agent, it SHOULD NOT change its document view
from that which caused the request to be sent. This response is
primarily intended to allow input for actions to take place without
causing a change to the user agent's active document view, although
any new or updated metainformation SHOULD be applied to the document
currently in the user agent's active view.
The 204 response MUST NOT include a message-body, and thus is always
terminated by the first empty line after the header fields.
Thus from flask you can do something like this. Remember, the response must not include a message body, so any data you want to send back should be put into a cookie.
#app.route('/')
def index():
r = flask.Response()
r.set_cookie("My important cookie", value=some_cool_value)
return r, 204
No, it is not possible. Flask is built on Werkzeug, which implements the WSGI spec. The WSGI cycle requires sending a response to each request. Droping the response would require control over the TCP/IP connection at a far lower level even that HTTP. This is outside the domain of WSGI, therefore outside the domain of Flask.
You could return an error code, or an empty body, but you have to return something.
return '' # empty body
I'm dusting off an app that worked a few months ago. I've made no changes. Here's the code in question:
result = urlfetch.fetch(
url=url,
deadline=TWENTY_SECONDS)
if result.status_code != 200: # pragma: no cover
logging.error('urlfetch failed.')
logging.error('result.status_code = %s' % result.status_code)
logging.error('url =')
logging.error(url)
Here's the output:
WARNING 2015-04-20 01:13:46,473 urlfetch_stub.py:118] No ssl package found. urlfetch will not be able to validate SSL certificates.
ERROR 2015-04-20 01:13:46,932 adminhandlers.py:84] urlfetch failed. url =
ERROR 2015-04-20 01:13:46,933 adminhandlers.py:85] http://www.stubhub.com/listingCatalog/select/?q=%2Bevent_date%3A%5BNOW%20TO%20NOW%2B1DAY%5D%0D%0A%2BancestorGeoDescriptions:%22New%20York%20Metro%22%0D%0A%2BstubhubDocumentType%3Aevent&version=2.2&start=0&rows=1&wt=json&fl=name_primary+event_date_time_local+venue_name+act_primary+ancestorGenreDescriptions+description
When I use a different url, e.g., "http://www.google.com/", the fetch succeeds.
When I paste the url string from the output into Chrome I get this response, which is the one I'm looking for:
{"responseHeader":{"status":0,"QTime":19,"params":{"fl":"name_primary event_date_time_local venue_name act_primary ancestorGenreDescriptions description","start":"0","q":"+event_date:[NOW TO NOW+1DAY]\r\n+ancestorGeoDescriptions:\"New York Metro\"\r\n+stubhubDocumentType:event +allowedViewingDomain:stubhub.com","wt":"json","version":"2.2","rows":"1"}},"response":{"numFound":26,"start":0,"docs":[{"act_primary":"Waka Flocka Flame","description":"Waka Flocka Flame Tickets (18+ Event)","event_date_time_local":"2015-04-20T20:00:00Z","name_primary":"Webster Hall","venue_name":"Webster Hall","ancestorGenreDescriptions":["All tickets","Concert tickets","Artists T - Z","Waka Flocka Flame Tickets"]}]}}
I hope I'm missing something simple. Any suggestions?
Update May 30, 2015
Anzel's suggestion of Apr 23 was correct. I need to add a user agent header. The one supplied by the AppEngine dev server is
AppEngine-Google; (+http://code.google.com/appengine)
The one supplied by hosted AppEngine is
AppEngine-Google; (+http://code.google.com/appengine; appid: s~MY_APP_ID)
The one supplied by requests.get() in pure Python (no AppEngine) on MacOS is
python-requests/2.2.1 CPython/2.7.6 Darwin/14.3.0
When I switch in the Chrome user agent header all is well in pure Python. Stubhub must have changed this since I last tried it. Curious that they would require an interactive user agent for a service that emits JSON, but I'm happy they offer the service at all.
When I add that header in AppEngine, though, AppEngine prepends it to its own user-agent header. Stubhub then turns down the request.
So I've made some progress, but have not yet solved my problem.
FYI:
In AppEngine I supply the user agent like this:
result = urlfetch.fetch(
url=url,
headers = {'user-agent': USER_AGENT_STRING}
)
This is a useful site for determining the user agent string your code or browser is sending:
http://myhttp.info/
I don't have priveledges yet to post comments, so here goes.
Look at the way you are entering the URL into the var 'url'. Is it already encoded as the error message says? I would try to make sure the url is a regular, non-encoded one, and test that, perhaps the library is re-encoding it again, causing problems. If you could give us more surrounding code, that may help in our diagnosis.