I've created a push streaming dataset (history on) and I've managed to post data to it from a Python script using the "Push URL" which I got from the API Info tab for the dataset in question. What I also need to do is to delete the historic data so as to clear out my test data and/or be able to reset the dataset and re-populate from scratch as and when necessary.
The Push Url is of the form https://api.powerbi.com/beta/xxxxxxxx/datasets/xxxxxxxxxxxx/rows?key=xxxxxxxxxxxxxxx
The following code works fine and the data is posted;
import requests
import pyodbc as db
import pandas as pd
API_ENDPOINT = "https://api.powerbi.com/beta/xxxxxxxx/datasets/xxxxxxxxxxxx/rows?key=xxxxxxxxxxxxxxx"
dbcon = db.connect('DRIVER={SQL Server};SERVER=tcp:fxdb.database.windows.net;DATABASE=FXDatabase;UID=xxxx;PWD=xxxx')
df = pd.read_sql("select statement etc...", dbcon)
data = df.to_dict(orient='records')
response = requests.post(API_ENDPOINT, json=data)
But adding this:
response = requests.delete(API_ENDPOINT)
gives me:
404
{
"error":{
"code":"","message":"No HTTP resource was found that matches the request URI 'http://api.powerbi.com/beta/...
I couldn't figure this out so I started looking into OAuth2 authentication thinking that perhaps the Auth URL is only for posting data. After registering the app at https://dev.powerbi.com/apps my code now looks like this:
import requests
import pyodbc as db
import pandas as pd
API_ENDPOINT = "https://api.powerbi.com/beta/xxxxxxxxxxxxxx/datasets/xxxxxxxxxxxxxxx/rows"
data = {
'grant_type': 'password',
'scope': 'openid',
'resource': r'https://analysis.windows.net/powerbi/api',
'client_id': 'xxxxxxxxx',
'username': 'xxxxxxxxx',
'password': 'xxxxxxxx'
}
response = requests.post('https://login.microsoftonline.com/common/oauth2/token', data=data)
access_token = response.json().get('access_token')
headers = {'Authorization': 'Bearer ' + access_token}
dbcon = db.connect('DRIVER={SQL Server};SERVER=tcp:fxdb.database.windows.net;DATABASE=FXDatabase;UID=xxxx;PWD=xxxx')
df = pd.read_sql("select statement etc...", dbcon)
data = df.to_dict(orient='records')
response = requests.post(API_ENDPOINT, json=data, headers=headers)
response = requests.delete(API_ENDPOINT, headers=headers)
The authentication works, returning status code 200. The POST returns 401 (this worked with the previous method) and the DELETE still returns 404.
Thanks to jonrsharpe who pointed me in the right direction.
Revisiting the API documentation I discovered a call to get the table names;
GET https://api.powerbi.com/v1.0/myorg/datasets/{datasetKey}/tables
so after authenticating I ran;
response = requests.get("https://api.powerbi.com/v1.0/myorg/datasets/xxxxxxxx/tables", headers=headers)
The content of the response told me that there was a table called "RealTimeData" inside my dataset, must be a default name because I haven't knowingly created this table.
I have now updated the endpoint to;
API_ENDPOINT = "https://api.powerbi.com/v1.0/myorg/datasets/xxxxxxxxx/tables/RealTimeData/rows"
and all works perfectly.
Thanks Jon!
Related
I am trying to Retrieve Statistics for Specific UGC Posts using the official documentations
My request looks like this:
requests.get(f'https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A00000&ugcPosts[0]=urn%3Ali%3AugcPost%3A111111&ugcPosts[1]=urn%3Ali%3AugcPost%3A222222', headers = headers)
"00000" is the company ID
"111111" and "222222" - are ugcPosts URNs
The headers look like this:
def headers(access_token):
'''
Make the headers to attach to the API call.
'''
headers = {
'Authorization': f'Bearer {access_token}',
'cache-control': 'no-cache',
'X-Restli-Protocol-Version': '2.0.0'
}
I have also passed the scope as one of the parameters, when authorizing:
def authorize(api_url,client_id,client_secret,redirect_uri):
# Request authentication URL
csrf_token = create_CSRF_token()
params = {
'response_type': 'code',
'client_id': client_id,
'redirect_uri': redirect_uri,
'state': csrf_token,
'scope': 'r_liteprofile,r_emailaddress,w_member_social,r_organization_social,r_1st_connections_size,r_ads_reporting,rw_organization_admin,r_basicprofile,r_ads,rw_ads,w_organization_social'
}
response = requests.get(f'{api_url}/authorization',params=params)
Unfortunately this particular request doesn't give me the response I was expecting:
{'serviceErrorCode': 100, 'message': 'Unpermitted fields present in PARAMETER: Data Processing Exception while processing fields [/ugcPosts[1], /ugcPosts[0]]', 'status': 403}
It works ok, when requesting a list of all ugcPosts
requests.get('https://api.linkedin.com/v2/ugcPosts?q=authors&authors=List(urn%3Ali%3Aorganization%3A00000)&sortBy=CREATED&count=100', headers = headers)
I have no clue what am I don't wrong. Can you please help me with my predicament?
The full request code is here
import requests
import json
from liapiauth import auth, headers
def organization_info(headers):
response = requests.get(f'https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A00000&ugcPosts[0]=urn%3Ali%3AugcPost%3A1111111&ugcPosts[1]=urn%3Ali%3AugcPost%3A2222222', headers = headers)
organization_info = response.json()
return organization_info
if __name__ == '__main__':
credentials = 'credentials.json'
access_token = auth(credentials)
headers = headers(access_token)
organization_info = organization_info(headers)
with open('lisharelist.json', 'w') as outfile:
json.dump(organization_info, outfile)
So it turns out, that the problem is in request formatting. The documentation is outdated.
The sample request looks like this:
GET https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn:li:organization:2414183&ugcPosts[0]=urn:li:ugcPost:1000000&ugcPosts[1]=urn:li:ugcPost:1000001
Apart from changing ":" to "%3A" we also have to edit the list of ugcPosts. Here is the correct looking sample request:
GET https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A2414183&ugcPosts=List(urn%3Ali%3AugcPost%3A1000000,urn%3Ali%3AugcPost%3A1000001)
I'm new to getting data using API and Python. I want to pull data from my trading platform. They've provided the following instructions:
http://www.questrade.com/api/documentation/getting-started
I'm ok up to step 4 and have an access token. I need help with step 5. How do I translate this request:
GET /v1/accounts HTTP/1.1
Host: https://api01.iq.questrade.com
Authorization: Bearer C3lTUKuNQrAAmSD/TPjuV/HI7aNrAwDp
into Python code? I've tried
import requests
r = requests.get('https://api01.iq.questrade.com/v1/accounts', headers={'Authorization': 'access_token myToken'})
I tried that after reading this: python request with authentication (access_token)
Any help would be appreciated. Thanks.
As you point out, after step 4 you should have received an access token as follows:
{
“access_token”: ”C3lTUKuNQrAAmSD/TPjuV/HI7aNrAwDp”,
“token_type”: ”Bearer”,
“expires_in”: 300,
“refresh_token”: ”aSBe7wAAdx88QTbwut0tiu3SYic3ox8F”,
“api_server”: ”https://api01.iq.questrade.com”
}
To make subsequent API calls, you will need to construct your URI as follows:
uri = [api_server]/v1/[rest_operation]
e.g.
uri = "https://api01.iq.questrade.com/v1/time"
Note: Make sure you use the same [api_server] that you received in your json object from step 4, otherwise your calls will not work with the given access_token
Next, construct your headers as follows:
headers = {'Authorization': [token_type] + ' ' + [access_token]}
e.g.
headers = {'Authorization': 'Bearer C3lTUKuNQrAAmSD/TPjuV/HI7aNrAwDp'}
Finally, make your requests call as follows
r = requests.get(uri, headers=headers)
response = r.json()
Hope this helps!
Note: You can find a Questrade API Python wrapper on GitHub which handles all of the above for you.
https://github.com/pcinat/QuestradeAPI_PythonWrapper
Improving a bit on Peter's reply (Thank you Peter!)
start by using the token you got from the QT website to obtain an access_token and get an api_server assigned to handle your requests.
# replace XXXXXXXX with the token given to you in your questrade account
import requests
r = requests.get('https://login.questrade.com/oauth2/token?grant_type=refresh_token&refresh_token=XXXXXXXX')
access_token = str(r.json()['access_token'])
refresh_token= str(r.json()['refresh_token']) # you will need this refresh_token to obtain another access_token when it expires
api_server= str(r.json()['api_server'])
token_type= str(r.json()['token_type'])
api_server= str(r.json()['api_server'])
expires_in = str(r.json()['expires_in'])
# uri = api_server+'v1/'+[action] - let's try checking the server's time:
uri = api_server+'v1/'+'time'
headers = {'Authorization': token_type +' '+access_token}
# will look sth like this
# headers will look sth like {'Authorization': 'Bearer ix7rAhcXx83judEVUa8egpK2JqhPD2_z0'}
# uri will look sth like 'https://api05.iq.questrade.com/v1/time'
# you can test now with
r = requests.get(uri, headers=headers)
response = r.json()
print(response)
I am trying to request an authorization code as documented here.
I am using Python requests package to do this and have the following example code:
import requests
auth_endpoint = 'https://login.microsoftonline.com/%s/oauth2/authorize?api-version=1.0' % TENANT_ID
payload = {
'client_id': CLIENT_ID,
'response_type': 'code',
'resource': APP_ID_URI,
'redirect_uri': REPLY_URL
}
response = requests.get(url=auth_endpoint, data=payload)
However, when I run the code above, I get back HTML in the body and not the response I'm expecting. It seems like the HTML code is for a login page.
When I take the formatted endpoint URI and plug it into a browser, I am able to get the auth code from the redirect URI. But, is there a way to get this from the body of the response while still using the requests package?
Please use session class of requests module to implement your requirement. Please refer to the following code sample:
import requests
s = requests.Session()
USERNAME = '<username_email>'
PASSWORD = '<userpassword>'
s.auth = (USERNAME, PASSWORD)
TENANT_ID = '<tenant_id>'
# Authorize URL
authorize_url = 'https://login.microsoftonline.com/%s/oauth2/authorize' % TENANT_ID
# Token endpoint.
token_url = 'https://login.microsoftonline.com/%s/oauth2/token' % TENANT_ID
payload = { 'response_type': 'code',
'client_id': '<tenant_id>',
'redirect_uri': 'http://localhost',
'authority' :'authority'
}
response = s.get(authorize_url, params=payload ,allow_redirects=True)
print response
print response.url
Any further concern, please feel free to let me know.
I'm fairly new to Python programming and I don't know all the libraries needed for the following.
I would like to use Python to test some HTTP APIs. Mainly I want to use OAuth and make a few JSON calls. The APIs in question can be found on: https://developers.trustpilot.com/authentication and the generate product review link (I can only use one link)
I want to authenticate myself and then generate a product review link in one step. So far I've been using the Advanced REST client (ARC) to make these calls individually. I could also use .arc files if you think it's easier.
The idea would be make these calls successively in one go. So it would be something along the lines:
1) Make the authentication call.
The HTTP Method looks like this:
https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken
Method Post:
Header
Authorization: Basic Base64encode(APIkey:Secret)
Content-Type: application/x-www-form-urlencoded
Payload:
grant_type=password&username=user#mail.com&password=SomePass
Translate this bit into Python basically.
1.a) Add a header to the call
Header Authorization: base64encode hash Content-Type: application/x-www-form-urlencoded
1.b) Add a payload to the call
Payload: grant_type=password&username
4) Receive the token from call made in step 1) (Result is format)
"access token": Auth_token
5) Take the token and use it in creating a product review.
5.a) Add the token in the header
Header: Authorization: Bearer Auth_token
6.a) Add a JSON payload to the call made in step 5.
Here's the code I have so far:
Import requests
header = {'Authorization: Basic NnNrQUprTWRHTU5VSXJGYXBVRGxack1oT01oTUFRZHI6QTFvOGJjRUNDdUxBTmVqUQ==}','Content-Type: application/x-www-form-urlencoded'}
payload = {'grant_type=password&username=email#address.com&password=SomePassword'}
r = requests.post('https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken', headers=header, params=payload )
Ideally I want to create the requests.post(url, header, payload) and then return what the server answers in JSON format. I think that print r.text would do the last part.
So this is the code I have writtent (that works now):
import requests
import getpass
import json
from requests.auth import HTTPBasicAuth
header = {'grant_type':'password' , 'username':'mail#maildomain.com', 'password':'YourPassword'}
username= "YOURAPIKEY" #APIKey
password= "YOURSECRET" #Secret
res = requests.post(
'URL/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
data=header)
#print(res.content) #See content of the call result.
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access_token')
requests can do all what you ask without any work from your part.
See the doc for authentication, parameters, json output, json input
Make the authentication call.
import requests
import getpass
from requests.auth import HTTPBasicAuth
username = raw_input('Username: ')
password = getpass.getpass('Password: ')
res = requests.post(
'https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
params={ # url parameters
'grant_type': 'password',
'username': 'email#address.com',
'password': 'SomePassword'
})
Receive the token from call made in step 1) (Result is format)
# res = requests.post.....
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access token')
Take the token and use it in creating a product review.
request.post(
'.../product_review',
headers={
'Authorization': 'Bearer ' + auth_token
},
json={'my': 'payload'}) # send data as json
I am trying to send a POST request with a Neo4j transaction query. Although I get a response 200 the node is not created. This is my Python script:
import requests
import json
import csv
headers = {'content-type': 'application/json'}
url = "http://localhost:7474/db/data/transaction/commit"
checkNode = {"query" : '{"statements": [{"statement":"CREATE (n:test) RETURN n"}]}'}
mkr =requests.post(url, data=json.dumps(checkNode), headers=headers)
print(mkr)
I haven't used transactions before and nver tried to create one through the Rest Api. What am I doing wrong here?
It seems unlikely to me that you're receiving a response code of 200; you should be getting a 500 as the transactional endpoint doesn't accept a query parameter. Try this:
import requests
import json
import csv
headers = {'content-type': 'application/json'}
url = "http://localhost:7474/db/data/transaction/commit"
checkNode = {"statements":[{"statement":"CREATE n RETURN n"}]}
mkr = requests.post(url, data=json.dumps(checkNode), headers=headers)
print(mkr.text)