Develop Reference

changing ampersands to %26

I am using Python-requests to pull data from a website. I am doing this currently :
params = {'A':'something', 'B':'something'}
response = requests.get(url, params = params)
which gives me: https://someurl?query=&A=something&B=something
This is all perfectly fine and great.
However, the website doesn't accept my API call. After some meddling around, I discovered that my target url is actually this:
https://someurl?query=%26A=something%26B=something
Hence my question : Is there a workaround for this problem? I have combed through requests' documentation and found nothing. I really don't feel like working with the url directly because I really like Python-requests.

The URL https://someurl?query=&A=something&B=something is very different than the URL https://someurl?query=%26A=something%26B=something.
URL1 https://someurl?query=&A=something&B=something is interpreted by the HTTP server as a request with 3 parameters: {
"query": "",
"A": "something",
"B": "something"
}
URL2 https://someurl?query=%26A=something%26B=something is interpreted by the HTTP server as a request with 1 parameter: {
"query": "&A=something%26B=something"
}
where "%26" is decoded as "&" character so value is decoded as &A=something&B=something.
A HTTP query with a single parameter "query" with value of &A=something&B=something needs to be properly encoded otherwise it will be converted into the wrong value. If using params options in requests API then the encoding is automatically done for you.
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
print(response.status_code)
If you want to debug requests under the covers then add this before calling requests.get().
import requests
import logging
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
Output:
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8000
DEBUG:urllib3.connectionpool:http://localhost:8000 "GET /?query=%26A%3Dsomething%26B%3Dsomething HTTP/1.1" 200 5
Notice the "=" in the URL is also enocded to avoid any confusion since "&" and "=" are special characters in URL string.

Try using urllib.parse.unquote() before sending requests to the server :
from urllib.parse import unquote
url="https://someurl?query=%26A=something%26B=something"
print(unquote(url))
# https://someurl?query=&A=something&B=something
And now you can send requests normally.

Python Request Issue - POST

I am attempting to sending JSON data to url server i.e. ("https://exampleurl.com/example?data=") and for some reason I am getting a 400 status code. Is there something wrong I am doing?
import requests
payload = {
"name":"Jane Doe"
}
r = requests.post('https://exampleurl.com/example?data=', json=payload)
print(r.text)

It looks basically okay, but I think you should apply good format for your json file.
Please add proper indentation and put a space between ":" and "Jane Doe".
And sometimes some post needs you to add headers for checking content type or password for using "POST", especially when they need to check you are authorized user. For more info, I think you need to check the official doc of the URL.
import requests
payload = {
"name": "Jane Doe"
}
headers = {'Content-type': 'application/json'}
r = requests.post('https://exampleurl.com/example?data=', headers=headers, json=payload)
print(r.text)
If this solution doesn't work, you should consider these factors:
Is your URL correct?
Is your request open for HTTPS protocol?
Do you need to apply some proxy for your development environment?
Hope this help you:)

you can try params
payload = {"name":"Jane Doe"}
r = requests.post('https://exampleurl.com/example', params=payload)

Instead of json=payload, do this:
import requests
import json
payload = {
"name":"Jane Doe"
}
r = requests.post('https://exampleurl.com/example?data=', data=json.dumps(payload))
print(r.text)

Node Express Empty body when using python API POST (requests library)

I'm using the starter kit by #ErikRas
With the following code, I'm having trouble authenticating my python program.
Here's my python:
import requests
URL="http://localhost"
PORT="3030"
Session = requests.Session()
Request = Session.post(URL+':'+PORT+'/login', data={'name':'AuthedUserName'})
# (Password to follow proof of concept obviously)
In my api.js file i just had:
import express from 'express';
import session from 'express-session';
import bodyParser from 'body-parser';
import config from '../src/config';
import * as actions from './actions/index';
import {mapUrl} from 'utils/url.js';
import http from 'http';
const app = express();
const server = new http.Server(app);
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(session({
secret: 'react and redux rule!!!!',
resave: false,
saveUninitialized: false,
cookie: { maxAge: 60000 }
}));
app.use((req, res) => {
/* There's heaps here, but all that is relevant is: */
console.log(req.body)
In the console i'm just getting {}
I found this article:
req.body empty on posts
and
Python Post Request Body appears empty in Node server when sent
but as you can see i'm already using bodyparser.json and bodyparser.urlencoded.extended = true

Okay, so i compared my pythons request against my web-app's request by printing the request to the console in node.
I found that the web app had more in its header than the python's requests' request. WebApp:
referer: 'http://localhost:3001/login'
origin: 'http://localhost:3001'
host: 'http://localhost:3001'
connection: 'close'
So I included this in my header, it worked!
I wanted to see which header property was 'necessary', so i gradually pulled everything out to see if this broke the POST request.
Turns out i managed to pull everything out! So what i'm using now is this:
r = Session.post(URL+':'+PORT+'/login',headers = {}, data={'name':'AuthedUserName'})
That's it!! I'd like to understand why headers={} works, but i need to moving forward with my project!!
<<<<<<---- Edit ---->>>>>>
Above is 'half' right, since my web app is using json and i want to use json, what i needed to do was change my header to
headers = {u'content-type': u'application/json'}
Then use json.dumps on the payload only!
r = session.post('http://'+DB_URL+':3030/sinumecUpdate', headers = headers, data = json.dumps(dataObject))
I also needed to pull out
app.use(bodyParser.urlencoded({ extended: true }));
From my node API and stick with only the JSON body parser.

import requests
import json
url = 'http://127.0.0.1'
data={'name':'AuthedUserName'}
headers = {'content-type': 'application/json'}
r = requests.post(url=url, data=json.dumps(data), headers=headers)

400 (Bad Request) error when using Pushbullet Ephemerals

I'm working on a simple command-line Pushbullet Python project, and have the following code:
from settings import *
import urllib
import urllib2
def pushSms(number, message):
url = 'https://api.pushbullet.com/v2/ephemerals'
values = {
"type": "push",
"push": {
"type": "messaging_extension_reply",
"package_name": "com.pushbullet.android",
"source_user_iden": settings["PUSHBULLET_USER_IDEN"],
"target_device_iden": settings["PUSHBULLET_SMS_IDEN"],
"conversation_iden": number,
"message": message
}
}
headers = {"Authorization" : "Bearer " + settings["PUSHBULLET_API_KEY"]}
data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
return response
Example usage might be pushSms("555 555 5555", "Hi there!").
This takes advantage of the Pushbullet android app access to SMS, as documented here. I've checked my settings variables and they're all valid (in fact, they're currently in use in a JavaScript version of nearly this exact code in another project of mine.
My suspicion is that this is a basic Python syntax/urllib2 misuse or error, but I've been staring/Googling for hours and can't see my error. Thoughts?

I can't tell for certain (the response from the server may contain more information), but because we accept both form encoded and json requests, you probably need to set the header "Content-Type: application/json" on the request.
If that's not the case, could you post the body of the 400 response?

Python request with authentication (access_token)

I am trying to use an API query in Python. From the command line I can use curl like so:
curl --header "Authorization:access_token myToken" https://website.example/id
This gives some JSON output. myToken is a hexadecimal variable that remains constant throughout.
I would like to make this call from python so that I can loop through different ids and analyze the output. Before authentication was needed I had done that with urllib2. I have also taken a look at the requests module but couldn't figure out how to authenticate with it.

The requests package has a very nice API for HTTP requests, adding a custom header works like this (source: official docs):
>>> import requests
>>> response = requests.get(
... 'https://website.example/id', headers={'Authorization': 'access_token myToken'})
If you don't want to use an external dependency, the same thing using urllib2 of the standard library looks like this (source: the missing manual):
>>> import urllib2
>>> response = urllib2.urlopen(
... urllib2.Request('https://website.example/id', headers={'Authorization': 'access_token myToken'})

I had the same problem when trying to use a token with Github.
The only syntax that has worked for me with Python 3 is:
import requests
myToken = '<token>'
myUrl = '<website>'
head = {'Authorization': 'token {}'.format(myToken)}
response = requests.get(myUrl, headers=head)

>>> import requests
>>> response = requests.get('https://website.com/id', headers={'Authorization': 'access_token myToken'})
If the above doesnt work , try this:
>>> import requests
>>> response = requests.get('https://api.buildkite.com/v2/organizations/orgName/pipelines/pipelineName/builds/1230', headers={ 'Authorization': 'Bearer <your_token>' })
>>> print response.json()

import requests
BASE_URL = 'http://localhost:8080/v3/getPlan'
token = "eyJhbGciOiJSUzI1NiIsImtpZCI6ImR"
headers = {'Authorization': "Bearer {}".format(token)}
auth_response = requests.get(BASE_URL, headers=headers)
print(auth_response.json())
Output :
{
"plans": [
{
"field": false,
"description": "plan 12",
"enabled": true
}
]
}

A lot of good answers already, but I didn't see this option yet:
If you're using requests, you could also specify a custom authentication class, similar to HTTPBasicAuth. For example:
from requests.auth import AuthBase
class TokenAuth(AuthBase):
def __init__(self, token, auth_scheme='Bearer'):
self.token = token
self.auth_scheme = auth_scheme
def __call__(self, request):
request.headers['Authorization'] = f'{self.auth_scheme} {self.token}'
return request
This could be used as follows (using the custom auth_scheme from the example):
response = requests.get(
url='https://example.com',
auth=TokenAuth(token='abcde', auth_scheme='access_token'),
)
This may look like a more complicated way to set the Request.headers attribute, but it can be advantageous if you want to support multiple types of authentication. Note this allows us to use the auth argument instead of the headers argument.

Have you tried the uncurl package (https://github.com/spulec/uncurl)? You can install it via pip, pip install uncurl. Your curl request returns:
>>> uncurl "curl --header \"Authorization:access_token myToken\" https://website.com/id"
requests.get("https://website.com/id",
headers={
"Authorization": "access_token myToken"
},
cookies={},
)

I'll add a bit hint: it seems what you pass as the key value of a header depends on your authorization type, in my case that was PRIVATE-TOKEN
header = {'PRIVATE-TOKEN': 'my_token'}
response = requests.get(myUrl, headers=header)

One of the option used in python to retrieve below:
import requests
token="abcd" < retrieved based>
headers = {'Authorization': "Bearer {}".format(token)}
response = requests.get(
'https://<url api>',
headers=headers,
verify="root ca certificate"
)
print(response.content)
If you get hostname mismatch error then additional SANs need to be configured in the server with the hostnames.
Hope this helps.