I have a Django app that's serving up a RESTful API using tasty-pie.
I'm using Django's development runserver to test.
When I access it via a browser it works fine, and using Curl also works fine:
curl "http://localhost:8000/api/v1/host/?name__regex=&format=json"
On the console with runserver, I see:
[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&format=json HTTP/1.1" 200 2845
However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404 as the output:
>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>
or:
>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>
Also, on the Django runserver console, I see:
[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072
For some reason, when I use requests, it prints out the whole request URL, including localhost - but not when I use the browser, or curl.
I'm assuming this is something to do with the encoding, user-agent or request type it's sending?
I'm not very familiar with Requests, but I think your encoding idea might be sound. That is Requests might process the URL somehow? Perhaps instead of passing everything in the URL directly, try doing what Requests docs suggest:
request_params = { 'name_regex' : '', 'format' : 'json' }
r = requests.get( 'http://localhost:8000/api/v1/host/', params = request_params )
Related
Using the following code, the server responds with a 301 redirect, but the client changes POST to GET, which is useless behavior because that GET endpoint does not exist. Using CURL -L -X POST works properly. This behavior is the same using python2 and python3 and on several versions of Raspbian.
>>> import requests
>>> url = "https://registry.micronets.in/mud/v1/register-
device/DAWG/AgoNDQcDDgg/aabbccddeeffgg"
>>> response = requests.post(url)
>>> response
<Response [404]>
# Server Log: (Note - both endpoints, are on the same server using virtual hosts)
redirecting to: https://hotdawg.micronets.in/registry/devices/register- device/AgoNDQcDDgg/aabbccddeeffgg
POST /registry/v1/register-device/DAWG/AgoNDQcDDgg/aabbccddeeffgg 301 16.563 ms - 122
{
"status": 404
}
GET /vendors//register-device/AgoNDQcDDgg/aabbccddeeffgg 404 0.604 ms - 14
# CURL version (succeeds)
curl -L -X POST "https://registry.micronets.in/mud/v1/register-
device/DAWG/AgoNDQcDDgg/aabbccddeeffgg"
Device registered (insert): {
"model": "AgoNDQcDDgg",
"pubkey": "aabbccddeeffgg",
"timestamp": "2019-12-27 15:44:14 UTC",
"_id": "HBlQzXfBnoB3N4fN"
}
# Server Log: (from CURL)
redirecting to: https://hotdawg.micronets.in/registry/devices/register-
device/AgoNDQcDDgg/aabbccddeeffgg
POST /registry/v1/register-device/DAWG/AgoNDQcDDgg/aabbccddeeffgg 301 0.364 ms - 122
POST /vendors//register-device/AgoNDQcDDgg/aabbccddeeffgg 200 1.745 ms - 157
I'd rather accept a better answer, but otherwise I plan to work around the problem as follows:
response = requests.post(url, allow_redirects=False)
if response.status_code == 301:
response = requests.post(response.headers['Location'])
or
response = requests.post(url, allow_redirects=False)
i=10
while i > 0 and response.status_code == 301:
response = requests.post(response.headers['Location'], allow_redirects=False)
i -= 1
Check your nginx configuration: Reference
You've already sent a POST request which is already reached the server successfully.
Now the server is handling the POST request and handling it according to your assigned rules.
Here's the issue occurred as the redirection assignment on your configuration is calling the end url by GET request.
You are blaming requests that it's not handling the redirect but it's already handled it correctly. but you compare it to curl which you used with it -L which is force the server the server by default to push the request. (etc, curl --list-only -X POST "https://registry.micronets.in/mud/v1/register-device/DAWG/AgoNDQcDDgg/aabbccddeeffgg") where you are using -X which is for HTTP over an HTTPS url.
-X, --request will be used for all requests, which if you for example use -L, --location may cause unintended side-effects when curl doesn't change request method according to the HTTP 30x response codes - and similar. <<< Pay attention to the part of curl doesn't change request method which is POST in your case and already forced the server with it. but for requestsit's completely different thing.
import requests
with requests.Session() as ses:
r = ses.post(
"https://registry.micronets.in/mud/v1/register-device/DAWG/AgoNDQcDDgg/aabbccddeeffgg", allow_redirects=True)
print(r.history[0].headers['Location'])
Output:
https://hotdawg.micronets.in/registry/devices/register-device/AgoNDQcDDgg/aabbccddeeffgg
at the end, i do believe this nginx server which is behind linode servers, and that's a common issue.
the knowledge is here:
import requests
with requests.Session() as ses:
r = ses.post(
"https://registry.micronets.in/mud/v1/register-device/DAWG/AgoNDQcDDgg/aabbccddeeffgg", allow_redirects=True)
print(r.history, r.history[0].reason)
print(r.status_code, r.reason)
Output:
[<Response [301]>] Moved Permanently
404 Not Found
But POST To it !
r = requests.post(
"https://hotdawg.micronets.in/registry/devices/register-device/AgoNDQcDDgg/aabbccddeeffgg")
print(r)
Output:
<Response [200]>
Which confirm that you sent POST request and it's completely switched to GET.
with that said, as i explained above that curl -X is set to force the server with the method used which is POST till the end point.
I'm attempting to perform a get request, however my instance of idle is consistently freezing/crashing when attempting to do so.
import requests
url = 'https://zkillboard.com/api/history/20161217/'
req = requests.get(url)
print(req.status_code)
print(req.text)
Ouput:
>>>
============= RESTART: D:\Google Drive\eveSoloPredictor\test.py =============
200
*then crashes
I am able to request this url and receive a JSON response both in my browser and using postman with no issues. Also, I am able to request other URL's in this script absolutely fine.
I am trying to get a response from an internal url which I can access through my laptop using a web-browser.
s = requests.Session()
r = s.get(url_1, auth=auth, verify=False)
print r.text
the reply i get is: 401 - unauthorized.
It's obviously going to be difficult to debug an HTTP 401 Unauthorized as we don't have access to the internal URL. Your code looks correct to me so I'm assuming this is a real 401 Unauthorized which means the request has incorrect authentication credentials. My advice would be to make sure you have reviewed the Python Requests docs on authentication and consider that your request is likely going through a proxy so the Requests docs on proxy config might be helpful.
I'm working with an external API that unfortunately doesn't have that great error logging.
I use django 1.9.5 and requests 2.11.1.
When I make the following request with the built-in python server (python manage.py runserver) on my local machine, I get back a 200 status code, so this works fine.
r = requests.post(
'https://plazaapi.bol.com/offers/v1/%s' % product.ean, data=xml_to_send,
headers=headers)
headers are a dictionary of the date, an authorization code and the content-type
.
But as there is a problem with requests on GAE according to other answers on this site, I have tried to use the requests_toolbelt monkeypatch and urlfetch, but I always get back the following error then:
Request contains invalid authentication headers
Code with the monkeypatch:
import requests_toolbelt.adapters.appengine
requests_toolbelt.adapters.appengine.monkeypatch()
r = requests.post(
'https://plazaapi.bol.com/offers/v1/%s' % product.ean, data=xml_to_send,
headers=headers)
and
from google.appengine.api import urlfetch
r = urlfetch.fetch(
url='https://plazaapi.bol.com/offers/v1/%s' % product.ean,
payload=xml_to_send,
method=urlfetch.POST,
headers=headers,
follow_redirects=False) # tried this, but has no effect.
The headers I'm setting are:
headers = {'Content-Type': 'application/xml',
'X-BOL-Date': date,
'X-BOL-Authorization': signature}
Is GAE changing my request and adding headers? If so, can I stop it
from doing so?
I'm learning to use proxies in making requests, but I've run into a big issue, which is primarily that it seems requests doesn't care if a provided proxy is valid or not. This makes it almost impossible to tell if something is actually working or not and I'm honestly at a loss for what to do. The documentation on proxies provided by requests is very minimal.
My code grabs a User-Agent string and a proxy from a list like so:
proxy = {"https": "https://%s:%s#%s" % (USERNAME, PASSWORD, random.choice(PROXY_LIST))}
headers = {"User-Agent": random.choice(USER_AGENT_LIST)}
return partial(requests.get, proxies=proxy, headers=headers)
an example of a PROXY_LIST entry: 185.46.87.199:8080
The issue is that I can change the username, change the password, etc... and requests doesn't seem to notice/care. A large portion of all the requests being sent aren't going through a proxy at all. Is there any way to test proxies? See if a request is actually going through a provided proxy? Really any tools for debugging and/or fixing this would be immensely appreciated.
After suggestion by larsks, changed the logging level to DEBUG and got the following output:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): mobile.twitter.com
DEBUG:requests.packages.urllib3.connectionpool:"GET /motivesbylorenr HTTP/1.1" 404 1318
unchanged whether auth is correct or incorrect, and no mention of proxy in the debug information. Again, requests are going through my local IP.
Requests logs debugging information at the DEBUG priority, so if you enable debug logging via the logging module you can see a variety of diagnostics. For example:
>>> import logging
>>> logging.basicConfig(level='DEBUG')
With that in place, I can set run:
>>> import requests
>>> s = requests.Session()
>>> s.headers={'user-agent': 'my-test-script'}
>>> s.proxies={'http': 'http://localhost:8123',
... 'https': 'http://localhost:8123'}
>>> s.get('http://mentos.com')
And see:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:requests.packages.urllib3.connectionpool:"GET http://mentos.com/ HTTP/1.1" 301 0
DEBUG:requests.packages.urllib3.connectionpool:"GET http://us.mentos.com HTTP/1.1" 200 32160
<Response [200]>
That clearly shows the connection to the proxy.
This is hopefully enough to get you started. I'm using a Session
here, but your solution using partial would behave similarly.
Compare the above output to the log message when requests is not using a proxy:
>>> requests.get('http://mentos.com')
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mentos.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 301 0
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): us.mentos.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 200 10566
<Response [200]>
Here, we see the initial connection opened to the remote site, rather
than the proxy, and the GET requests do not include the hostname.
Update
The above, with HTTPS URLs:
>>> response = s.get('https://google.com')
>>> response
<Response [200]>
Note that I am setting both the http and https keys in the proxies dictionary.