I'm currently having a hard time getting some of my Django tests to work.
What I'm trying to do is test if a given URL (the REST API I want to consume) is up and running (returning status code 200) and later on if it's responding the expected values.
However, all I get returned is a status code 404 (Page not found), even though the URL is definitely the right one. (Tried the exact string in my browser)
This is the code:
from django.test import TestCase
class RestTest(TestCase):
def test_api_test_endpoint(self):
response = self.client.get("http://ip.to.my.api:8181/test/")
self.assertEqual(response.status_code, 200, "Status code not equals 200")
It always returns a 404 instead of a 200...
Anyone knows what I do wrong here?
self.client is not a real HTTP client; it's the Django test client, which simulates requests to your own app for the purposes of testing. It doesn't make HTTP requests, and it only accepts a path, not a full URL.
If you really needed to check that an external URL was up, you would need a proper HTTP client like requests. However this doesn't seem to be an appropriate thing to do in a test case, since it depends on an external API; if that API went down, suddenly your tests would fail, which seems odd. This could be something that you do in a monitoring job, but not in a test.
Related
I'm trying to make one simple request:
ua=UserAgent()
req = requests.get('https://www.casasbahia.com.br/' , headers={'User-Agent':ua.random})
I would understand if I received <Response [403] or something like that, but instead, a recive nothing, the code keep runing with no response.
using logging I see:
I know I could use a timeout to avoid keeping the code running, but I just want to understand why I don't get an response
thanks in advance
I never used this API before, but from what I researched on here just now, there are sites that can block requests from fake users.
So, for reproducing this example on my PC, I installed fake_useragent and requests modules on my Python 3.10, and tried to execute your script. It turns out that with my Authentic UserAgent string, the request can be done. When printed on the console, req.text shows the entire HTML file received from the request.
But if I try again with a fake user agent, using ua.random, it fails. The site was probably developed to detect and reject requests from fake agents (or bots).
Though again, this is just theory. I have no ways to access this site's server files to comprove it.
I am trying to debug a CORS issue with my app. Specifically, it fails only in Firefox and, it seems, only with somewhat bigger files.
I am using flask on the backend and I am trying to upload a "faulty" image to my service. When I say faulty, I mean that the backend should reject the image with a 400 (only accept PNG, not JPG). Uploading a PNG of any size works ok. However, when I reject the JPG file, the browser request fails with Network error and I cannot capture the 400-error to display a user-friendly message. From the backend's side, everything is the same, always same headers returned, be it accepted or rejected request, POST or OPTIONS.
However, I have noticed that it only fails with somewhat bigger files. If I send a JPG of a few KBs, it works. If I send a JPG of a few MBs, it fails.
I have looked at everything
curl-ing the backend gives all the right headers
there are no OPTIONS requests logged by the browsers, but if there were, I've also checked those with curl for the right headers
I'm only using HTTP (not HTTPS), so no problems with certificates
I have disabled all extensions, so no possible blocking from the browser
maybe other things that I cannot remember
What can possibly be the cause? Note that everything works as expected
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8083/api/image. (Reason: CORS request did not succeed).
Well, after a couple of hours of trials, it appears this has nothing to do with CORS. This is probably the most confusing error message. To cite from Firefox' documentation on this (emphasis mine):
The HTTP request which makes use of CORS failed because the HTTP connection failed at either the network or protocol level. The error is not directly related to CORS, but is a fundamental network error of some kind. In many cases, it is caused by a browser plugin (e.g. an ad blocker or privacy protector) blocking the request.
So, this should actually indicate that the problem is on the backend, although it is very subtle.
Since in my code I am rejecting the request based on the trasmitted filename, I never read the content of the request if the name ends with .jpg. Instead, I reject it immediately. This is a problem with Flask's development server, which does not empty the input stream in such cases (issue here).
So, if you want to deal with this while keeping the development server, you should consume the input. In my case, I added a custom error handler, like so:
class BadRequestError(ValueError):
"""Raised when a request does not conform to the protocol"""
pass
#app.errorhandler(BadRequestError)
def bad_request_handler(error):
# throw away the request data to avoid closing the connection before receiving all of it
# http://flask.pocoo.org/snippets/47/
_ = request.data
_ = request.form
response = jsonify(str(error))
response.status_code = 400
return response
and then, in the code, I always raise BadRequestError('...'), instead of just returning a 400-response.
I am probably overseeing something obvious but I can't seem to figure it. I am trying a simple verification to start with using the following url.
http://myanimelist.net/api/account/verify_credentials.xml
http://myanimelist.net/modules.php?go=api#verifycred
(Here's the full documentation regarding this URL).
This is the code used for testing it out.
class Foobar():
def __init__(self):
pass
def bar(self):
client = requests.get('http://myanimelist.net/api/account/verify_credentials.xml',
auth=(username, password))
if client.status_code == 200:
print "Succesfull authentication. %i"%client.status_code
else:
print "Authentication failed %i"%client.status_code
print client.text
Foo = Foobar()
Foo.bar()
I got a correct response once and assumed this was the right way of going. However from this part on I only receive responses like this.
Every request send regarding the user credentials being correct or not.
I've tried various encoding and neither have affected the response in any way.
EDIT: I seem to have solved the issue. After wiping my cookies and clearing my cache it returned a valid response by status code 401.
The issue causing it was cooking placed by the site itself. I am unsure which cookies specifically caused this problem but once found I will add it.
EDIT: They have a bot checking thrid party connections which bans you upon trying to connect rending their API useless.
I'm trying to use urlfetch to make a request to my application (the same application which is sending the request) however, it doesn't work.
My code is as follows;
uploadurl = 'http://myapp.appspot.com/posturl'
result = urlfetch.fetch(
url=uploadurl,
payload=data,
method=urlfetch.POST,
headers={'Content-Type': 'application/x-www-form-urlencoded'})
There is no error at all when I call this, and everything seems to work correctly, however the request never arrives. For debugging purposes, I changed the uploadurl to a different application which I own and it worked fine. Any ideas why I can't send requests using urlfetch to the same application?
The full (real) url that I would call is made by
session = str(os.urandom(16).encode('hex'))
uploadurl = blobstore.create_upload_url('/process?session=' + session)
So I can't understand how that could be incorrect as the url is made for me.
Thanks.
I don't know how you're verifying that the request "never arrives". The blobstore URLs are not handled by your application's actual code, but by the App Engine runtime itself, so if you're looking in the logs you won't see that request there.
I think it is not possible, to prevent endless loop. From the urlfetch api documentation page:
To prevent an app from causing an endless recursion of requests, a
request handler is not allowed to fetch its own URL. It is still
possible to cause an endless recursion with other means, so exercise
caution if your app can be made to fetch requests for URLs supplied by
the user.
I am using urllib2 to do an http post request using Python 2.7.3. My request is returning an HTTPError exception (HTTP Error 502: Proxy Error).
Looking at the messages traffic with Charles, I see the following is happening:
I send the HTTP request (POST /index.asp?action=login HTTP/1.1) using urllib2
The remote server replies with status 303 and a location header of ../index.asp?action=news
urllib2 retries sending a get request: (GET /../index.asp?action=news HTTP/1.1)
The remote server replies with status 502 (Proxy error)
The 502 reply includes this in the response body: "DNS lookup failure for: 10.0.0.30:80index.asp" (Notice the malformed URL)
So I take this to mean that a proxy server on the remote server's network sees the "/../index.asp" URL in the request and misinterprets it, sending my request on with a bad URL.
When I make the same request with my browser (Chrome), the retry is sent to GET /index.asp?action=news. So Chrome takes off the leading "/.." from the URL, and the remote server replies with a valid response.
Is this a urllib2 bug? Is there something I can do so the retry ignores the "/.." in the URL? Or is there some other way to solve this problem? Thinking it might be a urllib2 bug, I swapped out urllib2 with requests but requests produced the same result. Of course, that may be because requests is built on urllib2.
Thanks for any help.
The Location being sent with that 302 is wrong in multiple ways.
First, if you read RFC2616 (HTTP/1.1 Header Field Definitions) 14.30 Location, the Location must be an absoluteURI, not a relative one. And section 10.3.3 makes it clear that this is the relevant definition.
Second, even if a relative URI were allowed, RFC 1808, Relative Uniform Resource Locators, 4. Resolving Relative URLs, step 6, only specifies special handling for .. in the pattern <segment>/../. That means that a relative URL shouldn't start with ... So, even if the base URL is http://example.com/foo/bar/ and the relative URL is ../baz/, the resolved URL is not http://example.com/foo/baz/, but http://example.com/foo/bar/../baz. (Of course most servers will treat these the same way, but that's up to each server.)
Finally, even if you did combine the relative and base URLs before resolving .., an absolute URI with a path starting with .. is invalid.
So, the bug is in the server's configuration.
Now, it just so happens that many user-agents will work around this bug. In particular, they turn /../foo into /foo to block users (or arbitrary JS running on their behalf without their knowledge) from trying to do "escape from webroot" attacks.
But that doesn't mean that urllib2 should do so, or that it's buggy for not doing so. Of course urllib2 should detect the error earlier so it can tell you "invalid path" or something, instead of running together an illegal absolute URI that's going to confuse the server into sending you back nonsense errors. But it is right to fail.
It's all well and good to say that the server configuration is wrong, but unless you're the one in charge of the server, you'll probably face an uphill battle trying to convince them that their site is broken and needs to be fixed when it works with every web browser they care about. Which means you may need to write your own workaround to deal with their site.
The way to do that with urllib2 is to supply your own HTTPRedirectHandler with an implementation of redirect_request method that recognizes this case and returns a different Request than the default code would (in particular, http://example.com/index.asp?action=news instead of http://example.com/../index.asp?action=news).