I have a RESTful API that I have exposed using an implementation of Elasticsearch on an EC2 instance to index a corpus of content. I can query the search by running the following from my terminal (MacOSX):
curl -XGET 'http://ES_search_demo.com/document/record/_search?pretty=true' -d '{
"query": {
"bool": {
"must": [
{
"text": {
"record.document": "SOME_JOURNAL"
}
},
{
"text": {
"record.articleTitle": "farmers"
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 50,
"sort": [],
"facets": {}
}'
How do I turn above into a API request using python/requests or python/urllib2 (not sure which one to go for - have been using urllib2, but hear that requests is better...)? Do I pass as a header or otherwise?
Using requests:
import requests
url = 'http://ES_search_demo.com/document/record/_search?pretty=true'
data = '''{
"query": {
"bool": {
"must": [
{
"text": {
"record.document": "SOME_JOURNAL"
}
},
{
"text": {
"record.articleTitle": "farmers"
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 50,
"sort": [],
"facets": {}
}'''
response = requests.post(url, data=data)
Depending on what kind of response your API returns, you will then probably want to look at response.text or response.json() (or possibly inspect response.status_code first). See the quickstart docs here, especially this section.
Using requests and json makes it simple.
Call the API
Assuming the API returns a JSON, parse the JSON object into a
Python dict using json.loads function
Loop through the dict to extract information.
Requests module provides you useful function to loop for success and failure.
if(Response.ok): will help help you determine if your API call is successful (Response code - 200)
Response.raise_for_status() will help you fetch the http code that is returned from the API.
Below is a sample code for making such API calls. Also can be found in github. The code assumes that the API makes use of digest authentication. You can either skip this or use other appropriate authentication modules to authenticate the client invoking the API.
#Python 2.7.6
#RestfulClient.py
import requests
from requests.auth import HTTPDigestAuth
import json
# Replace with the correct URL
url = "http://api_url"
# It is a good practice not to hardcode the credentials. So ask the user to enter credentials at runtime
myResponse = requests.get(url,auth=HTTPDigestAuth(raw_input("username: "), raw_input("Password: ")), verify=True)
#print (myResponse.status_code)
# For successful API call, response code will be 200 (OK)
if(myResponse.ok):
# Loading the response data into a dict variable
# json.loads takes in only binary or string variables so using content to fetch binary content
# Loads (Load String) takes a Json file and converts into python data structure (dict or list, depending on JSON)
jData = json.loads(myResponse.content)
print("The response contains {0} properties".format(len(jData)))
print("\n")
for key in jData:
print key + " : " + jData[key]
else:
# If response code is not ok (200), print the resulting http error code with description
myResponse.raise_for_status()
Below is the program to execute the rest api in python-
import requests
url = 'https://url'
data = '{ "platform": { "login": { "userName": "name", "password": "pwd" } } }'
response = requests.post(url, data=data,headers={"Content-Type": "application/json"})
print(response)
sid=response.json()['platform']['login']['sessionId'] //to extract the detail from response
print(response.text)
print(sid)
So you want to pass data in body of a GET request, better would be to do it in POST call. You can achieve this by using both Requests.
Raw Request
GET http://ES_search_demo.com/document/record/_search?pretty=true HTTP/1.1
Host: ES_search_demo.com
Content-Length: 183
User-Agent: python-requests/2.9.0
Connection: keep-alive
Accept: */*
Accept-Encoding: gzip, deflate
{
"query": {
"bool": {
"must": [
{
"text": {
"record.document": "SOME_JOURNAL"
}
},
{
"text": {
"record.articleTitle": "farmers"
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 50,
"sort": [],
"facets": {}
}
Sample call with Requests
import requests
def consumeGETRequestSync():
data = '{
"query": {
"bool": {
"must": [
{
"text": {
"record.document": "SOME_JOURNAL"
}
},
{
"text": {
"record.articleTitle": "farmers"
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 50,
"sort": [],
"facets": {}
}'
url = 'http://ES_search_demo.com/document/record/_search?pretty=true'
headers = {"Accept": "application/json"}
# call get service with headers and params
response = requests.get(url,data = data)
print "code:"+ str(response.status_code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "content:"+ str(response.text)
consumeGETRequestSync()
Related
headers = {
"User-Agent": "Mozilla/5.0",
'accept': 'application/json',
'Content-Type': 'application/json',
}
with open("jsonattempt.txt","r") as f:
data = f.read()
json_data = "'" + data + "'"
response = requests.post('https://www.pathofexile.com/api/trade/search/Standard', headers=headers, data=json_data)
print(response)
Generally, there is a curl request like this:
curl -X 'POST' \
'https://www.pathofexile.com/api/trade/search/Standard' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"query": {
"status": {
"option": "online"
},
"type": "Turquoise Amulet",
"stats": [
{
"type": "and",
"filters": [
{
"id": "pseudo.pseudo_total_mana",
"value": {
"min": 47,
"max": 49
},
"disabled": false
}
]
}
]
},
"sort": {
"price": "asc"
}
}'
Which returns a bunch of unnecessary things.
My json_data variable and jsonattempt.txt is the same as -d parameter, I add ' ' to start and to end:
{
"query": {
"status": {
"option": "online"
},
"type": "Turquoise Amulet",
"stats": [
{
"type": "and",
"filters": [
{
"id": "pseudo.pseudo_total_mana",
"value": {
"min": 47,
"max": 49
},
"disabled": false
}
]
}
]
},
"sort": {
"price": "asc"
}
}
I convert curl request to python which is the code on the top, I add the data as json_data and yeet the post request but keep getting 400 Bad Request.
Request 401 is Unauthorized AFAIK, so I dont need an OAuth2 key for this. How can I insert my json file appropiately to the request?
(Same exact json works on https://app.swaggerhub.com/apis-docs/Chuanhsing/poe/1.0.0#/Trade/get_api_trade_fetch__items_ I am just asking how to insert my json file to requests.post correctly.)
Why are you adding quotes around the JSON content? That doesn't make any sense. Those quotes aren't part of your curl request. If you just write...
import requests
headers = {
"User-Agent": "Mozilla/5.0",
"accept": "application/json",
"Content-Type": "application/json",
}
with open("jsonattempt.txt", "r") as f:
data = f.read()
response = requests.post(
"https://www.pathofexile.com/api/trade/search/Standard",
headers=headers,
data=data,
)
print(response)
...it works as expected.
And in fact you can further simplify that; you don't need to read in the file data yourself, you can let requests do that:
import requests
headers = {
"User-Agent": "Mozilla/5.0",
"accept": "application/json",
"Content-Type": "application/json",
}
with open("jsonattempt.txt", "r") as f:
response = requests.post(
"https://www.pathofexile.com/api/trade/search/Standard",
headers=headers,
data=f,
)
print(response)
import json
# some codes here
with open("jsonattempt.txt","r") as f:
data = f.read()
json_data = json.loads(data)
# rest of codes here
Your request requires json type data, while you are passing a string. The json.loads method converts string to json. Try it out.
I'm fairly new to Python and I was able to run a python request and grabbed the session token and saved it to a variable, Now I'm trying to pass that session to a new request but I'm not sure how can I API request sequentially right after one another?
this is my request ..
url = "https://1.1.1.1/jsonrpc"
payload = json.dumps(
{
"session": 1,
"id": 1,
"method": "exec",
"params": [
{
"url": "sys/login/user",
"data": [
{
"user": "admin",
"passwd": "password"
}
]
}
]
}
)
response = requests.request("POST", url, data=payload, verify=False)
s = (response.json())
print (s['session'])
now I want to pass 's' variable to a new API request in the same .py file but not sure how to run them right after each other.
url = "https://1.1.1.1/jsonrpc"
payload = json.dumps({
"session": s
"id": 1,
"method": "set",
"params": [
{
"url": "/dvmdb/adom",
"data": [
{
"name": "NEW_ADOM"
}
]
}
]
})
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
If you are using s in two separated functions, you can declare s variable outside those two functions and change that via global keyword
s = None
def get_session():
...
global s
s = response.json()['session']
Then you can use s in another function
I try to use Firebase cloud messaging HTTP v1
Here's the Python code which run under Windows
import json
import requests
from google.oauth2.service_account import Credentials
import google.auth.transport.requests
def _get_access_token():
credentials = Credentials.from_service_account_file(
'C:/yocto/wenote-notification/send_notification/wenote-206215-firebase-adminsdk-9fpls-a3fdd2934c.json',
scopes=['https://www.googleapis.com/auth/firebase.messaging']
)
request = google.auth.transport.requests.Request()
credentials.refresh(request)
access_token = credentials.token
return access_token
def generate_header():
return {'Authorization' : 'Bearer ' + _get_access_token(), 'Content-type' : 'application/json'}
# Header
headers = generate_header()
print(headers)
# Data
string = '{"message":{"token": "cSW75-6OF-w:APA91bE-E6vWucBe8rRijpDLlZGHhyfwoaLJr3R1TZEvieBM-__ZXwXlBS34kUticUN_eSbwvQGTymbmtd7sHT5U9O_v9HePyVn7jnUD9IBdoZZYSQ1CrgxXS1sz9wjAd5pKHIddoKj8", "data": {"sync": false, "sync_device_count": 2}, "fcm_options": {"analytics_label": "wenote_analytics_label"}}}'
request_data = json.loads(string)
print(request_data)
r = requests.post('https://fcm.googleapis.com/v1/projects/wenote-206215/messages:send', headers=headers, json=request_data)
if r.status_code != 200:
r.raise_for_status()
The message I send to Firebase server is
{
"message":{
"token":"cSW75-6OF-w:APA91bE-E6vWucBe8rRijpDLlZGHhyfwoaLJr3R1TZEvieBM-__ZXwXlBS34kUticUN_eSbwvQGTymbmtd7sHT5U9O_v9HePyVn7jnUD9IBdoZZYSQ1CrgxXS1sz9wjAd5pKHIddoKj8",
"data":{
"sync":false,
"sync_device_count":2
},
"fcm_options":{
"analytics_label":"wenote_analytics_label"
}
}
}
However, I'm getting the following error
c:\yocto\wenote-notification\send_notification>python a.py
{'Authorization': 'Bearer ya29.c.Ko8BvQdCd-8US-DzQ-AcuTOURlGTQEvJt4mtofaeM08LRIhXBfC4RFN3QzkAcCdnaqI6uJLQlk1YJg67KQOhF8CtNux9t743nq2NN9uoW2mbQiB2y_2fjRUhiIIM7Wz2nt9rDOM4zFIFwKlHtLPXRLa4IGo0Ho-dq8zzVxQH1qPNGqy_ja8WowQCY5ReAQkmAmE', 'Content-type': 'application/json'}
{'message': {'token': 'cSW75-6OF-w:APA91bE-E6vWucBe8rRijpDLlZGHhyfwoaLJr3R1TZEvieBM-__ZXwXlBS34kUticUN_eSbwvQGTymbmtd7sHT5U9O_v9HePyVn7jnUD9IBdoZZYSQ1CrgxXS1sz9wjAd5pKHIddoKj8', 'data': {'sync': False, 'sync_device_count': 2}, 'fcm_options': {'analytics_label': 'wenote_analytics_label'}}}
Traceback (most recent call last):
File "a.py", line 38, in <module>
r.raise_for_status()
File "C:\Python36\lib\site-packages\requests\models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://fcm.googleapis.com/v1/projects/wenote-206215/messages:send
Any idea why it is so, and how I can resolve this?
I have check my Firebase cloud messaging console. It is enabled.
You need to slightly adapt your request_data object, because the schema changed.
Before (legacy http protocol):
{
"to": "/topics/news",
"notification": {
"title": "Breaking News",
"body": "New news story available."
},
"data": {
"story_id": "story_12345"
}
}
After (HTTP v1 API):
{
"message": {
"topic": "news",
"notification": {
"title": "Breaking News",
"body": "New news story available."
},
"data": {
"story_id": "story_12345"
}
}
}
Notice the change in the topic parameters, as well the wrap by a message object.
Check out this guide: https://firebase.google.com/docs/cloud-messaging/migrate-v1
You can check out this very clean and structure python based minimal example for FCM: https://github.com/firebase/quickstart-python/blob/688fcfa8068dcac67978a171df828c9e77cd320e/messaging/messaging.py#L57
This is the way to debug the above problem.
Use
if r.status_code != 200:
print(r.text)
r.raise_for_status()
instead of
if r.status_code != 200:
r.raise_for_status()
We will get the following error
{
"error": {
"code": 400,
"message": "Invalid value at 'message.data[0].value' (TYPE_STRING), false\nInvalid value at 'message.data[1].value' (TYPE_STRING), 2",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"field": "message.data[0].value",
"description": "Invalid value at 'message.data[0].value' (TYPE_STRING), false"
},
{
"field": "message.data[1].value",
"description": "Invalid value at 'message.data[1].value' (TYPE_STRING), 2"
}
]
}
]
}
}
Look like HTTP v1 is pretty strict on the value type (It need to be String)
Hence, instead of using
string = '{"message":{"token": "cSW75-6OF-w:APA91bE-E6vWucBe8rRijpDLlZGHhyfwoaLJr3R1TZEvieBM-__ZXwXlBS34kUticUN_eSbwvQGTymbmtd7sHT5U9O_v9HePyVn7jnUD9IBdoZZYSQ1CrgxXS1sz9wjAd5pKHIddoKj8", "data": {"sync": false, "sync_device_count": 2}, "fcm_options": {"analytics_label": "wenote_analytics_label"}}}'
We need to use
string = '{"message":{"token": "cSW75-6OF-w:APA91bE-E6vWucBe8rRijpDLlZGHhyfwoaLJr3R1TZEvieBM-__ZXwXlBS34kUticUN_eSbwvQGTymbmtd7sHT5U9O_v9HePyVn7jnUD9IBdoZZYSQ1CrgxXS1sz9wjAd5pKHIddoKj8", "data": {"sync": "false", "sync_device_count": "2"}, "fcm_options": {"analytics_label": "wenote_analytics_label"}}}'
I have used the below GitHub api and i'am able to get the files details of the path.
https://github.***.com/api/v3/repos/exampleowner-Management/examplerepo/contents/Compile/Teradata/Tables?access_token=*****
The result of this API call is :
[
{
"name": ".DS_Store",
"path": "Compile/Tables/test",
"sha": "1cef8efa8694678e3b7ab230a6a891afa1a1996d",
"size": 8196,
"url": "***",
"html_url": "***",
"git_url": "***",
"download_url": "***",
"type": "file",
"_links": {
"self": "***",
"git": "***",
"html": "***"
}
}]
I need to get the commit date details for the sha in this response.
"sha": "1cef8efa8694678e3b7ab230a6a891afa1a1996d"
I have tried using another API ,which is :
https://github.***.com/api/v3/repos/exampleowner-Management/examplerepo/commits/1cef8efa8694678e3b7ab230a6a891afa1a1996d?access_token=*****
but the response of this API for this sha is:
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/enterprise/2.14/v3/repos/commits/#get-a-single-commit"}
How can we get commit date details along with GitHub content details by using API calls?
finally got the expected result by using Graphql.here is the complete code
def run_query(query): # A simple function to use requests.post to make the API call. Note the json= section.
try:
request = requests.post('https://api.github.***.com/graphql', json={'query': query}, headers=headers)
return request.json()
except e:
returnVal = '404'
query = """
{
repository(owner: \""""+ownerVal+"""\", name: \""""+repoVal+"""\") {
object(expression: \""""+branchVal+"""\") {
... on Commit {
blame(path: \""""+folderVal+"/"+data['name']+"""\") {
ranges {
commit {
committedDate
}
}
}
}
}
}
}
"""
headers = {"Authorization": "Bearer "+access_token}
result = run_query(query)
commit_date = result["data"]["repository"]["object"]["blame"]["ranges"][0]["commit"]["committedDate"]
I am experimenting with the Rapaport Technet API, and want to hit an endpoint which expects the following JSON:
{
"request": {
"header": {
"username": "my_username",
"password": "my_password"
},
"body": {}
}
}
Code:
url = 'https://technet.rapaport.com:449/HTTP/JSON/Prices/GetPriceChanges.aspx'
headers = {'username': 'my_username', 'password': 'my_password'}
r = requests.post(url, headers)
I get this response:
{
"response": {
"header": {
"error_code": 1001,
"error_message": "Invalid format"
},
"body": {}
}
}
Any idea what the problem could be?
According to this example from Rapaport Technet API docs, that whole JSON is sent to be as data in the POST request. So simply do the same as given here in Requests docs.
json_data = {
"request": {
"header": {
"username": "my_username",
"password": "my_password"
},
"body": {}
}
}
r = requests.post(url, json=json_data)