I am using Python-requests to pull data from a website. I am doing this currently :
params = {'A':'something', 'B':'something'}
response = requests.get(url, params = params)
which gives me: https://someurl?query=&A=something&B=something
This is all perfectly fine and great.
However, the website doesn't accept my API call. After some meddling around, I discovered that my target url is actually this:
https://someurl?query=%26A=something%26B=something
Hence my question : Is there a workaround for this problem? I have combed through requests' documentation and found nothing. I really don't feel like working with the url directly because I really like Python-requests.
The URL https://someurl?query=&A=something&B=something is very different than the URL https://someurl?query=%26A=something%26B=something.
URL1 https://someurl?query=&A=something&B=something is interpreted by the HTTP server as a request with 3 parameters: {
"query": "",
"A": "something",
"B": "something"
}
URL2 https://someurl?query=%26A=something%26B=something is interpreted by the HTTP server as a request with 1 parameter: {
"query": "&A=something%26B=something"
}
where "%26" is decoded as "&" character so value is decoded as &A=something&B=something.
A HTTP query with a single parameter "query" with value of &A=something&B=something needs to be properly encoded otherwise it will be converted into the wrong value. If using params options in requests API then the encoding is automatically done for you.
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
print(response.status_code)
If you want to debug requests under the covers then add this before calling requests.get().
import requests
import logging
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
Output:
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8000
DEBUG:urllib3.connectionpool:http://localhost:8000 "GET /?query=%26A%3Dsomething%26B%3Dsomething HTTP/1.1" 200 5
Notice the "=" in the URL is also enocded to avoid any confusion since "&" and "=" are special characters in URL string.
Try using urllib.parse.unquote() before sending requests to the server :
from urllib.parse import unquote
url="https://someurl?query=%26A=something%26B=something"
print(unquote(url))
# https://someurl?query=&A=something&B=something
And now you can send requests normally.
Related
Best wishes (first things first!)
I want to enable/disable a PoE port on my UniFi switch. For this I aim using Python 3.9.1 (first time) with the following code:
import requests
import json
import sys
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
gateway = {"ip": "MYSERVER.COM", "port": "8443"}
headers = {"Accept": "application/json", "Content-Type": "application/json"}
login_url = f"https://{gateway['ip']}:{gateway['port']}/api/login"
login_data = {
"username": "MYUSERNAME",
"password": "MYPASSWORD"
}
session = requests.Session()
login_response = session.post(login_url, headers=headers, data=json.dumps(login_data), verify=False)
if (login_response.status_code == 200):
api_url_portoverrides = 'api/s/default/rest/device/MYDEVICEID'
poe_url = f"https://{gateway['ip']}:{gateway['port']}/{api_url_portoverrides}"
# build json for port overrides
json_poe_state_on = '{"port_overrides": [{"port_idx": 6, "portconf_id": "MYPROFILE1"}]}'
json_poe_state_off = '{"port_overrides": [{"port_idx": 6, "portconf_id": "MYPROFILE2"}]}'
post_response = session.put(poe_url, headers=headers, data=json.dumps(json_poe_state_off))
print('Response HTTP Request {request}'.format(request=post_response.request ))
else:
print("Login failed")
The login works (I get the 2 security cookies and tried them in Paw (a macOS REST API client) to see if these were ok)) but the second call, the. PUT, returns OK but noting happens.
Before I've done this in Python, I tried all my calls in Paw first and there it works. I tried everything in bash with curl and there it works too. So I am a bit at a loss here.
Anyone has an idea?
Thank you for your time!
Best regards!
Peter
Solved it! By looking into what was posted with Wireshark I saw that the payload was different. The culprit was the json.dumps function which encoded the string by putting a backslash in front of each double quote.
It works now!
I'm having a problem building a Twitter random quotes generator API. I'm following this tutorial:
https://www.twilio.com/blog/build-deploy-twitter-bots-python-tweepy-pythonanywhere
But I get an error that he doesn't have. This is the code:
import requests
api_key = '*****'
api_url = 'https://andruxnet-random-famous-quotes.p.rapidapi.com'
headers = {'afd9cbe77emshf06f5cb2f889689p1ca1c3jsne6e79ad808cc' :
api_key, 'http://andruxnet-random-famous-quotes.p.rapidapi.com' :
api_url}
# The get method is called when we
# want to GET json data from an API endpoint
quotes = requests.get(quotes = requests.get(api_url,
headers=headers)
print(quotes.json())
And this is the error:
File "twitter_bot.py", line 12
print(quotes.json())
SyntaxError: invalid syntax
What am I doing wrong?? (I put *** on the key on purpose, I know the proper key is supposed to go there)
Thank you!
You have a copy-and-paste error; somehow you've put quotes = requests.get( twice.
It should just be:
# The get method is called when we
# want to GET json data from an API endpoint
quotes = requests.get(api_url, headers=headers)
print(quotes.json())
Tutorial is not so old but it seems it is already out of date.
Using example from RapidAPI documentation (for Random Famous Quotes API) I created Python's code which gives some information from server (but still not quotes)
import requests
url = "https://andruxnet-random-famous-quotes.p.rapidapi.com/?count=10&cat=famous"
headers={
"X-RapidAPI-Host": "andruxnet-random-famous-quotes.p.rapidapi.com",
"X-RapidAPI-Key": "afd9cbe77emshf06f5cb2f889689p1ca1c3jsne6e79ad808cc",
}
quotes = requests.get(url, headers=headers)
print(quotes.text)
#print(quotes.json())
Result:
{"message":"You are not subscribed to this API."}
The same for POST
import requests
url = "https://andruxnet-random-famous-quotes.p.rapidapi.com/?count=10&cat=famous"
headers={
"X-RapidAPI-Host": "andruxnet-random-famous-quotes.p.rapidapi.com",
"X-RapidAPI-Key": "afd9cbe77emshf06f5cb2f889689p1ca1c3jsne6e79ad808cc",
"Content-Type": "application/x-www-form-urlencoded"
}
quotes = requests.post(url, headers=headers)
print(quotes.text)
#print(quotes.json())
Result:
{"message":"You are not subscribed to this API."}
It still need some work to get quotes.
I can't figure out how to call this api correctly using python urllib or requests.
Let me give you the code I have now:
import requests
url = "http://api.cortical.io:80/rest/expressions/similar_terms?retina_name=en_associative&start_index=0&max_results=1&sparsity=1.0&get_fingerprint=false"
params = {"positions":[0,6,7,29]}
headers = { "api-key" : key,
"Content-Type" : "application/json"}
# Make a get request with the parameters.
response = requests.get(url, params=params, headers=headers)
# Print the content of the response
print(response.content)
I've even added in the rest of the parameters to the params variable:
url = 'http://api.cortical.io:80/rest/expressions/similar_terms?'
params = {
"retina_name":"en_associative",
"start_index":0,
"max_results":1,
"sparsity":1.0,
"get_fingerprint":False,
"positions":[0,6,7,29]}
I get this message back:
An internal server error has been logged # Sun Apr 01 00:03:02 UTC
2018
So I'm not sure what I'm doing wrong. You can test out their api here, but even with testing I can't figure it out. If I go out to http://api.cortical.io/, click on the Expression tab, click on the POST /expressions/similar_terms option then paste {"positions":[0,6,7,29]} in the body textbox and hit the button, it'll give you a valid response, so nothing is wrong with their API.
I don't know what I'm doing wrong. can you help me?
The problem is that you're mixing query string parameters and post data in your params dictionary.
Instead, you should use the params parameter for your query string data, and the json parameter (since the content type is json) for your post body data.
When using the json parameter, the Content-Type header is set to 'application/json' by default. Also, when the response is json you can use the .json() method to get a dictionary.
An example,
import requests
url = 'http://api.cortical.io:80/rest/expressions/similar_terms?'
params = {
"retina_name":"en_associative",
"start_index":0,
"max_results":1,
"sparsity":1.0,
"get_fingerprint":False
}
data = {"positions":[0,6,7,29]}
r = requests.post(url, params=params, json=data)
print(r.status_code)
print(r.json())
200
[{'term': 'headphones', 'df': 8.991197733061748e-05, 'score': 4.0, 'pos_types': ['NOUN'], 'fingerprint': {'positions': []}}]
So, I can't speak to why there's a server error in a third-party API, but I followed your suggestion to try using the API UI directly, and noticed you're using a totally different endpoint than the one you're trying to call in your code. In your code you GET from http://api.cortical.io:80/rest/expressions/similar_terms but in the UI you POST to http://api.cortical.io/rest/expressions/similar_terms/bulk. It's apples and oranges.
Calling the endpoint you mention in the UI call works for me, using the following variation on your code, which requires using requests.post, and as was also pointed out by t.m. adam, the json parameter for the payload, which also needs to be wrapped in a list:
import requests
url = "http://api.cortical.io/rest/expressions/similar_terms/bulk?retina_name=en_associative&start_index=0&max_results=1&sparsity=1.0&get_fingerprint=false"
params = [{"positions":[0,6,7,29]}]
headers = { "api-key" : key,
"Content-Type" : "application/json"}
# Make a get request with the parameters.
response = requests.post(url, json=params, headers=headers)
# Print the content of the response
print(response.content)
Gives:
b'[[{"term":"headphones","df":8.991197733061748E-5,"score":4.0,"pos_types":["NOUN"],"fingerprint":{"positions":[]}}]]'
I'm using the starter kit by #ErikRas
With the following code, I'm having trouble authenticating my python program.
Here's my python:
import requests
URL="http://localhost"
PORT="3030"
Session = requests.Session()
Request = Session.post(URL+':'+PORT+'/login', data={'name':'AuthedUserName'})
# (Password to follow proof of concept obviously)
In my api.js file i just had:
import express from 'express';
import session from 'express-session';
import bodyParser from 'body-parser';
import config from '../src/config';
import * as actions from './actions/index';
import {mapUrl} from 'utils/url.js';
import http from 'http';
const app = express();
const server = new http.Server(app);
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(session({
secret: 'react and redux rule!!!!',
resave: false,
saveUninitialized: false,
cookie: { maxAge: 60000 }
}));
app.use((req, res) => {
/* There's heaps here, but all that is relevant is: */
console.log(req.body)
In the console i'm just getting {}
I found this article:
req.body empty on posts
and
Python Post Request Body appears empty in Node server when sent
but as you can see i'm already using bodyparser.json and bodyparser.urlencoded.extended = true
Okay, so i compared my pythons request against my web-app's request by printing the request to the console in node.
I found that the web app had more in its header than the python's requests' request. WebApp:
referer: 'http://localhost:3001/login'
origin: 'http://localhost:3001'
host: 'http://localhost:3001'
connection: 'close'
So I included this in my header, it worked!
I wanted to see which header property was 'necessary', so i gradually pulled everything out to see if this broke the POST request.
Turns out i managed to pull everything out! So what i'm using now is this:
r = Session.post(URL+':'+PORT+'/login',headers = {}, data={'name':'AuthedUserName'})
That's it!! I'd like to understand why headers={} works, but i need to moving forward with my project!!
<<<<<<---- Edit ---->>>>>>
Above is 'half' right, since my web app is using json and i want to use json, what i needed to do was change my header to
headers = {u'content-type': u'application/json'}
Then use json.dumps on the payload only!
r = session.post('http://'+DB_URL+':3030/sinumecUpdate', headers = headers, data = json.dumps(dataObject))
I also needed to pull out
app.use(bodyParser.urlencoded({ extended: true }));
From my node API and stick with only the JSON body parser.
import requests
import json
url = 'http://127.0.0.1'
data={'name':'AuthedUserName'}
headers = {'content-type': 'application/json'}
r = requests.post(url=url, data=json.dumps(data), headers=headers)
I'm working on a simple command-line Pushbullet Python project, and have the following code:
from settings import *
import urllib
import urllib2
def pushSms(number, message):
url = 'https://api.pushbullet.com/v2/ephemerals'
values = {
"type": "push",
"push": {
"type": "messaging_extension_reply",
"package_name": "com.pushbullet.android",
"source_user_iden": settings["PUSHBULLET_USER_IDEN"],
"target_device_iden": settings["PUSHBULLET_SMS_IDEN"],
"conversation_iden": number,
"message": message
}
}
headers = {"Authorization" : "Bearer " + settings["PUSHBULLET_API_KEY"]}
data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
return response
Example usage might be pushSms("555 555 5555", "Hi there!").
This takes advantage of the Pushbullet android app access to SMS, as documented here. I've checked my settings variables and they're all valid (in fact, they're currently in use in a JavaScript version of nearly this exact code in another project of mine.
My suspicion is that this is a basic Python syntax/urllib2 misuse or error, but I've been staring/Googling for hours and can't see my error. Thoughts?
I can't tell for certain (the response from the server may contain more information), but because we accept both form encoded and json requests, you probably need to set the header "Content-Type: application/json" on the request.
If that's not the case, could you post the body of the 400 response?