I have Problem with scraping data from LinkedIn.
I think the documentation is too complicated ...
here the problem, I want to make a request (GET) and get for example data of my feed/posts/chats or whatever.
here is my code:
import json
import requests
# URL = "https://www.linkedin.com/voyager/api/voyagerMessagingDashConversationNudges"
URL = "https://www.linkedin.com/voyager/api/identity/dash/profiles"
cookies = {
#Cookies are here
}
params = {
'decorationId': 'com.linkedin.voyager.dash.deco.identity.profile.WebTopCardCore-6',
'memberIdentity': 'maria-albert-137632240',
'q': 'memberIdentity',
}
def get_group(url: str, cookies: dict, data:dict, header: dict):
response = requests.get(url=url, cookies=cookies, data=json.dumps(data), headers=header)
response.raise_for_status()
return response.json()
if __name__ == "__main__":
print("sending request to Server:\n")
get_group(url=URL, cookies=cookies, data=params, header=headers)
but I couldn't do it, the error --> raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: INKApi Error for url: https://www.linkedin.com/voyager/api/identity/dash/profiles
Thanks for your help.
You just need to set header like this:
header = {
"accept": 'application/vnd.linkedin.normalized+json+2.1',
"cookie": 'JSESSIONID="ajax:abcd"; li_at=abcdabcd',
}
and so fill the payload as the endpoint needed.
You don't have to Create App to access token or anything else.
Related
I hope someone could help me in this.
Following this official doc from FTX:
https://docs.ftx.com/#request-historical-balances-and-positions-snapshot
I'm continue to get this error:
{'success': False, 'error': 'Missing parameter accounts', 'errorCode': 'parameter_missing'}
The code was ok and getting all the response from the other API call until I tested the the one above and I had to add the parameters, those are not working :-(
Here is my code:
import requests
from requests import Request, Session
import time
import hmac
import json
s = requests.Session()
url = "https://ftx.com/api/historical_balances/requests"
ts = int(time.time() * 1000)
tsN = int(time.time())
params = {
"accounts": ["main","subaccounts"],
"endTime": tsN,
}
request = requests.Request("POST", url, params=params)
prepared = request.prepare()
signature_payload = f'{ts}{prepared.method}{prepared.path_url}'.encode()
if prepared.body:
signature_payload += prepared.body
signature = hmac.new('MYSECRET'.encode(), signature_payload, 'sha256').hexdigest()
request.headers = {
'FTX-KEY': 'MYKEY',
'FTX-SIGN': signature,
'FTX-TS': str(ts),
}
r = s.send(request.prepare())
r.json()
print('Output Json :',r.json())
Do you have any suggestion please? I'm getting crazy with this... Thanks!!
Try switching request = requests.Request("POST", url, params=params) to request = requests.Request("POST", url, data=params) or request = requests.Request("POST", url, json=params), because I believe they want you to send them accounts and endTime as body of request, not as url parameters.
I'm trying to send a post request to a SREST API, which is currently on the localhost and keep getting various errors.
The error I'm getting at the moment is:
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The code I'm executing:
import requests
import json
def create_bucket():
url = "http://127.0.0.1:3000/api/buckets"
headers = {
"content-type": "application/json"
}
params = {
"bucket_name": "test_bucket"
}
response = requests.post(url=url, headers=headers, params=params)
print(response.json())
create_bucket()
What am I doing wrong?
Edit:
Tried printing response.content as per request and got an error:
b'<!doctype html>\n<html lang=en>\n<title>400 Bad Request</title>\n<h1>Bad Request</h1>\n<p>Failed to decode JSON object: Expecting value: line 1 column 1 (char 0)</p>\n'
Edit 2:
Solved the problem. Correct approach:
response = requests.post(url=url, headers=headers, data=json.dumps(params))
Per the error, need to encode json properly, try:
response = requests.post(url=url, headers=headers, json=params)
requests will do the right thing
or
response = requests.post(url=url, headers=headers, params=json.dumps(params))
I'm trying to get data from a public api from schiphol (airport in Amsterdam).
Im getting this api from https://api.schiphol.nl/public-flights/flights.
I'm using python to get the flight data. In my code I get the error that the "app_id" is none while this is filled in....
the full error from the console:
Usage: flight_info_api.py [options]
flight_info_api.py: error: Please provide an app id (-i, --app_id)
Can anybody see what is going wrong?
My code:
import requests
import sys
import optparse
def callPublicFlightAPI(options):
url = 'https://api.schiphol.nl/public-flights/flights'
headers = {
'resourceversion': 'v4',
'app_id': 'b209eb7f',
'app_key': '0b6c58b5ae4595dd39785b55f438fc70'
}
try:
response = requests.request('GET', url, headers=headers)
except requests.exceptions.ConnectionError as error:
print(error)
sys.exit()
if response.status_code == 200:
flightList = response.json()
print('found {} flights.'.format(len(flightList['flights'])))
for flight in flightList['flights']:
print('Found flight with name: {} scheduled on: {} at {}'.format(flight['flightName'],
flight['scheduleDate'],
flight['scheduleTime']))
else:
print('''Oops something went wrong Http response code: {}{}'''.format(response.status_code, response.text))
if __name__ == '__main__':
parser = optparse.OptionParser()
parser.add_option('-i', '--app_id', dest='app_id',
help='App id used to call the API')
parser.add_option('-k', '--app_key', dest='app_key',
help='App key used to call the API')
(options, args) = parser.parse_args()
if options.app_id is None:
parser.error('Please provide an app id (-i, --app_id)')
if options.app_key is None:
parser.error('Please provide an app key (-key, --app_key)')
callPublicFlightAPI(options)
You need to add this to your headers:
'Accept': 'application/json'
Good luck.
EDIT:
Basically, Since you would like to receive your data as json, You will have to add 'Accept': 'application/json' to your headers. In this case, your headers will look like this:
headers = {
'Accept': 'application/json',
'resourceversion': 'v4',
'app_id': YOUR_APP_ID,
'app_key': YOUR_APP_KEY
}
And when you are going to make a request, you have to add the headers in the parameter. Your request is going to look like this:
response = requests.get(URL, headers=headers)
I hope this helps!
Below is my code to your view:
import warnings
import contextlib
import json
import requests
from urllib3.exceptions import InsecureRequestWarning
old_merge_environment_settings = requests.Session.merge_environment_settings
#contextlib.contextmanager
def no_ssl_verification():
opened_adapters = set()
def merge_environment_settings(self, url, proxies, stream, verify, cert):
# Verification happens only once per connection so we need to close
# all the opened adapters once we're done. Otherwise, the effects of
# verify=False persist beyond the end of this context manager.
opened_adapters.add(self.get_adapter(url))
settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert)
settings['verify'] = False
return settings
requests.Session.merge_environment_settings = merge_environment_settings
try:
with warnings.catch_warnings():
warnings.simplefilter('ignore', InsecureRequestWarning)
yield
finally:
requests.Session.merge_environment_settings = old_merge_environment_settings
for adapter in opened_adapters:
try:
adapter.close()
except:
pass
with no_ssl_verification():
##350014,166545
payload = {'key1': '350014', 'key2': '166545'}
resp = requests.get('https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/params', params=payload, verify=False, headers={'Authorization': 'Token +++++private++++', 'content-type': 'application/json'})
print(resp.status_code)
print(resp.status_code)
j = resp.json()
##print(j)
jprint(resp.json())
how can I do a while or a for to send a list of personal id numbers and get a JSON result to witch one?
I tried pasting some parametres but it does not work, produce some errors...
I got this follow error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
if i put:
resp = requests.get('https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/350014',
with a single number, it works.
here the follow json:
200
[
{
"DT_INI_VIG_invalidez": null,
"DT_fim_VIG_invalidez": null,
"MODULO": "APOIO",
"chapa": 350014,
}
]
You have to add number to url manually
"https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/" + str(params)
or
"https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/{}".format(params)
or using f-string in Python 3.6+
f"https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/{params}"
Using params=params will not add numer to url this way but ?key1=350014&key2=166545
You can see url used by request using
print(resp.request.url)
Now you can run in loop
all_results = []
for number in [350014, 166545]:
url = 'https://rhconnect.marcopolo.com.br/api/workers/data_employee/company/1/badge/{}'.format(number)
resp = requests.get(url, verify=False, headers={'Authorization': 'Token +++++private++++', 'content-type': 'application/json'})
#print(resp.request.url)
print(resp.status_code)
print(resp.json())
# keep result on list
all_results.append(resp.json())
BTW: If you get error then you should check what you get
print(resp.text)
Maybe you get HTML with information or warning
I'm trying POST a check-in request in Google Places API. The way they described it, I have to request this -
POST https://maps.googleapis.com/maps/api/place/check-in/json?sensor=true_or_false&key=AddYourOwnKeyHere HTTP/1.1
Host: maps.googleapis.com
{
"reference": "place_reference"
}
My Current code looks like this -
def checkin(self, reference="", sensor="true"):
"""
"""
base_url = "https://maps.googleapis.com/maps/api/place/check-in/json"
params = urllib.urlencode(
{
'key': self.API_KEY,
'sensor': sensor,
}
)
post_url = base_url + "?" + params
headers = { 'Host': "maps.googleapis.com" }
data = urllib.urlencode({ 'reference': reference })
req = Request(post_url, data, headers)
response = urllib2.urlopen(req)
resp = response.read()
But I keep getting the error -
urllib2.HTTPError: HTTP Error 400: Bad Request
What am I doing wrong?
Your problem is that the API is expecting JSON when you are sending it the literal reference: xyz
You need to send it the JSON representation.
Try:
data = json.dumps({'reference': reference})