Develop Reference

Request data from Linkedin

I have Problem with scraping data from LinkedIn.
I think the documentation is too complicated ...
here the problem, I want to make a request (GET) and get for example data of my feed/posts/chats or whatever.
here is my code:
import json
import requests
# URL = "https://www.linkedin.com/voyager/api/voyagerMessagingDashConversationNudges"
URL = "https://www.linkedin.com/voyager/api/identity/dash/profiles"
cookies = {
#Cookies are here
}
params = {
'decorationId': 'com.linkedin.voyager.dash.deco.identity.profile.WebTopCardCore-6',
'memberIdentity': 'maria-albert-137632240',
'q': 'memberIdentity',
}
def get_group(url: str, cookies: dict, data:dict, header: dict):
response = requests.get(url=url, cookies=cookies, data=json.dumps(data), headers=header)
response.raise_for_status()
return response.json()
if __name__ == "__main__":
print("sending request to Server:\n")
get_group(url=URL, cookies=cookies, data=params, header=headers)
but I couldn't do it, the error --> raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: INKApi Error for url: https://www.linkedin.com/voyager/api/identity/dash/profiles
Thanks for your help.

You just need to set header like this:
header = {
"accept": 'application/vnd.linkedin.normalized+json+2.1',
"cookie": 'JSESSIONID="ajax:abcd"; li_at=abcdabcd',
}
and so fill the payload as the endpoint needed.
You don't have to Create App to access token or anything else.

How to pass a file from a URL via Python to an API?

I'm trying to use an API where you give it a CSV file and it validates it. For this API you pass it a file.
What I'm trying to do is this:
import requests
import urllib3
api_url="https://validation.openbridge.io/dryrun"
file_url = 'http://winterolympicsmedals.com/medals.csv'
http = urllib3.PoolManager()
response = http.request('GET', file_url)
data = response.data.decode('utf-8')
headers = {
# requests won't add a boundary if this header is set when you pass files=
'Content-Type': 'multipart/form-data',
}
response = requests.post(api_url, headers=headers, files=data)
I'm getting an error I don't quite understand:
'cannot encode objects that are not 2-tuples'
I'm fairly new to python, I'm mostly experienced in other languages but what I think the issue is that I'm probably passing the API the data in the file instead of giving it the file.
Any suggestions on what I"m doing wrong and how I can pass that file to the API?

For anyone who stumbles across this off google in the future. I figured out how to do it.
import requests
import urllib3
api_url="https://validation.openbridge.io/dryrun"
file_url="http://winterolympicsmedals.com/medals.csv"
remote_file = urllib3.PoolManager().request('GET', file_url)
response = requests.post(url=api_url,
json={'data': {'attributes': {'is_async': True }}},
files={ "file": remote_file},
allow_redirects=False)
if response.ok:
print("Upload completed successfully!")
print()
print("Status Code: ")
print(response.status_code)
print()
print(response.headers['Location'])
print()
else:
print("Something went wrong!")
print()
print("Status Code: ")
print(response.status_code)
print()
print(response.text)
print()

Basic Authentication returns 401 Client Error but works in postman

I have gone through number of similar posts related to firing GET requests with Basic Auth (eg: Python, HTTPS GET with basic authentication), still can't figure out the problem. I keep getting the error requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url
With the same credentials, headers tried the same in postman it works as expected. Verified that base64encoded value for the api_key, password is exactly same as the value used in postman, so I don't think its encoding or resource access permission problem.
python -V
Python 3.6.4 :: Anaconda, Inc.
Approach 1
api_key = 'some_api_key'
password = 'some_password'
headers = {'accept': 'application/json'}
url = 'https://test.access.com/this/url'
api_key_password = "%s:%s" % (api_key, password)
b64_encoded = b64encode(bytes(api_key_password, 'utf-8')).decode("ascii")
headers['authorization'] = 'Basic %s' % b64_encoded
response = requests.get(url,
headers=headers)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Approach 2
api_key = 'some_api_key'
password = 'some_password'
url = 'https://test.access.com/this/url'
headers = {
'accept': 'application/json',
}
response = requests.get(url, headers=headers, auth=(api_key, password))
print (response.ok)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Can you please provide some pointers?

I had a similar issue (although in .NET Framework).
In my case the reason was that I was using the url without a forward slash in the end and the API apparently does not support that.
So https://test.access.com/this/url
Throws 401 error Unauthorized
but
https://test.access.com/this/url/
Returns 200 OK.

Older post but I had a similar issue. Postman will cache your JSESSIONID. Be sure you are clearing out that cookie while testing. If you are hitting an API that requires a login API call to establish a session before you can make subsequent API calls, this Postman behavior can produce a false sense of security.
In this situation with Python requests, it can be handled with code similar to what I've provided below:
import requests,json
loginAPI = "https://myapi.myco.comv/someuri/someuri/users/login"
someHTTPGetAPI = "https://myapi.myco.com/someuri/someuri/someservice"
username = "myuser"
password = "mypass"
headers = {
"Content-Type": "application/json",
"login": username,
"password": password
}
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
verify=False
session = requests.Session()
sessionResponse = session.get(url=loginURL,headers=headers, verify=verify)
if sessionResponse.status_code == 200:
getResponse = session.get(url=someHTTPGetAPI)
if getResponse.status_code == 200:
responseJSON = agentStatus.json()

Getting HTTP POST Error : {"reason":null,"error":"Request JSON object for insert cannot be null."}

I am getting HTTP POST error when I am trying to connect to a Service Now Instance for Change Request Automation using Python. Here is the script I am using with Python 3.4.4
# SNOW CR AUTOMATION SCRIPT
import requests
import json
# put the ip address or dns of your SNOW API in this url
url = 'http://<>/change_request.do?JSONv2&sysparm_action=insert'
data= {
'short_description': '<value>',
'priority': '<value>',
'reason': '<value>',
'u_reason_for_change': '<value>',
'u_business_driver': '<value>',
'u_plan_of_record_id': '<value>'
}
print ("Data Inserted :")
print (data)
#Content type must be included in the header
header = {"Authorization":"Basic V1NfRVRPX1ROOkBiY2RlNTQzMjE=","Content- Type":"application/json"}
#Performs a POST on the specified url.
response = requests.request('POST', url, auth=("<value>","<value>"), json=data, headers=header)
print ( " Header is : ")
print (response.headers)
print (" ")
print ( "HTTP Response is :" )
print (response)
print (" ")
print ("***********************")
print (" Output : ")
print ( response.text)
I am getting an error as below while running the above script.
Output :
{"reason":null,"error":"Request JSON object for insert cannot be null."}
I am not sure why this error is thrown. Can anybody please help on this ?

This is a working example I tested on my instance. I am using REST Table API to insert a change request. It's not true that it can not be http. It's whatever protocol your instance allows to connect, say from browser.
#Need to install requests package for python
#easy_install requests
import requests
# Set the request parameters
url = '<yourinstance base url>/api/now/table/change_request'
user = <username>
pwd = <password>
# Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
# Do the HTTP request
response = requests.post(url, auth=(user, pwd), headers=headers ,data="{\"short_description\":\"test in python\"}")
# Check for HTTP codes other than 201
if response.status_code != 201:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)

I think you should use SSL, so no http!
First error I see in your script is how you pass your payload, you need to transform your dictionary into a JSON Object/String. And you don't need to authenticate twice, you have the basic http authentication handled by requests.post so no need for it in the header.
With this script it should work:
import json
import requests
url = 'https://instancename.service-now.com/change_request.do?JSONv2'
user = 'admin'
pwd = 'admin'
# Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
payload = {
'sysparm_action': 'insert',
'short_description': 'test_jsonv2',
'priority': '1'
}
# Do the HTTP request
response = requests.post(url, auth=(user, pwd), headers=headers, data=json.dumps(payload))
# Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)

"Error parsing JSON" when using Spotify API

I am learning Python and I am trying to create a playlist using the Spotify web api but get a http 400 error: Error parsing json. I guess it has to do with an incorrect variable type in the token but I am having a really hard time debugging it as I can't figure out a way to see the post request in raw format.
Posting through the API requires authorizing and this is the script I've created for that:
import requests
import base64
requests.packages.urllib3.disable_warnings()
client_id = 'ID'
client_secret = 'SECRET'
redirect_uri = 'http://spotify.com/'
scope = 'playlist-modify-private playlist-read-private'
def request_token():
# 1. Your application requests authorization
auth_url = 'https://accounts.spotify.com/authorize'
payload = {'client_id': client_id, 'response_type':'code','redirect_uri':redirect_uri}
auth = requests.get(auth_url,params = payload)
print '\nPlease go to this url to authorize ', auth.url
# 2. The user is asked to authorize access within the scopes
# 3. The user is redirected back to your specified URI
resp_url = raw_input('\nThen please copy-paste the url you where redirected to: ')
resp_code= resp_url.split("?code=")[1].split("&")[0]
# 4. Your application requests refresh and access tokens
token_url = 'https://accounts.spotify.com/api/token'
payload = {'redirect_uri': redirect_uri,'code': resp_code, 'grant_type': 'authorization_code','scope':scope}
auth_header = base64.b64encode(client_id + ':' + client_secret)
headers = {'Authorization': 'Basic %s' % auth_header}
req = requests.post(token_url, data=payload, headers=headers, verify=True)
response = req.json()
return response
This is the function actually trying to create the playlist using the authorization token (import authorizer is the function above):
import requests
import authorizer
def create_playlist(username, list_name):
token = authorizer.request_token()
access_token = token['access_token']
auth_header = {'Authorization': 'Bearer {token}'.format(token=access_token), 'Content-Type': 'application/json'}
api_url = 'https://api.spotify.com/v1/users/%s/playlists' % username
payload = {'name': list_name, 'public': 'false'}
r = requests.post(api_url, params=payload, headers=auth_header)
But whatever I try it only leads to a 400 error. Can anyone please point out my error here?

Solved by adding a json.dumps for the input: json.dumps(payload) and changing the payload to be 'data' and not 'params' in the request.
So the new functioning request equals:
r = requests.post(api_url, data=json.dumps(payload), headers=auth_header)