I am accessing data on futures prices from the Chicago Mercantile Exchange via API calls.
I am using an URL which, when I post it in a browser, returns a csv-file (gzip compressed). I do however need to write a program in python that can take in several thousand URLs and handle the data it receives. When using this URL in a requests.get() command in python, I get a response which I do not know how to read.
import requests
url='https://datamine.cmegroup.com/cme/api/v1/download?fid=20181211-
EOD_xcbt_ff_fut_0-eth_p'
user = '***'
password = '***'
r = requests.get(url, auth=(user, password))
print(r.headers)
Which, among other things, returns:
{
'Date': 'Sun, 30 Dec 2018 13:01:07 GMT',
'Content-Type': 'application/x-gzip',
'Content-Length': '1287',
'Content-disposition': 'attachment; filename="xcbt-eodp-ff-fut-20181211.csv.gz"'
}
I assume that the prices I am trying to obtain are contained in the element 'attachment; filename="xcbt-eodp-ff-fut-20181211.csv.gz"'.
My question:
How do I obtain the prices from the response I am getting?
Then, how do I import the data into a DataFrame?
Simply using:
print(r.content)
does not return numeric values.
You should do something like:
resp = requests.get(url)
# for now it's a Response object. Make a json out of it
json_resp = resp.json()
print(json_resp.get('some_key')) # or what ever you need
Then use a simple dict functionality to extract what ever you need..
Related
I am using python request library to interact with API.
import requests
url = "https://amazon24.p.rapidapi.com/api/product"
querystring = {"keyword":"iphone","country":"US","page":"1"}
headers = {
'x-rapidapi-host': "amazon24.p.rapidapi.com",
'x-rapidapi-key': "key"
}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text)
Response I am getting is something like this. I have only included first few lines of response as it is too long.
{"docs":[{"isBestSeller":false,"product_title":"Apple iPhone XR (64GB, (PRODUCT)RED) [Locked] + Carrier Subscription","product_main_image_url":"https://m.media-amazon.com/images/I/51YXG1bDM5L._AC_UY218_.jpg","app_sale_price":"499.00","app_sale_price_currency":"$","isPrime":true,"product_detail_url":"https://www.amazon.com/dp/B07VSWLY55","product_id":"B07VSWLY55","evaluate_rate":"4.2 out of 5 stars","original_price":null},{"isBestSeller":false,"product_title":"New Apple iPhone 12 Mini (64GB, Black) Unlocked","product_main_image_url":"https://m.media-amazon.com/images/I/71uuDYxn3XL._AC_UY218_.jpg","app_sale_price":"729.00","app_sale_price_currency":"$","isPrime":true,"product_detail_url":"https://www.amazon.com/dp/B096R76SFK","product_id":"B096R76SFK","evaluate_rate":"4.2 out of 5 stars","original_price":null}}}]}}}
My question is how can I extract attributes like "product_title" or "app_sale_price" from above response. Desperately need help. Thanks
You can convert the response to a dictionary using response.json().
Following that, you can access the various attributes such as data["docs"][0]["product_title"]
An example is as follows:
data = response.json()
for doc in data["docs"]:
print(doc["product_title"], doc["app_sale_price"])
You can learn more about dictionaries here (official python docs) or here (w3schools)
Your response is JSON, requests has handy method for such situation, just do
response = requests.request("GET", url, headers=headers, params=querystring)
response_data = response.json()
then response_data is python object, dict in your case and you might do for example
titles = [i.get("product_title") for i in response_data["docs"]]
print(titles)
to get titles from list of dicts present under key "docs"
I am very new to python (and coding in general) so I hope I articulate my issues well enough. I've managed to cobble together an application that connects to the API of one of my companies software.
I've been able to auth, get the token, pass it to other functions to GET and POST to some API functions. I'm working on a new API call that posts a file. I am STUMPED. All the examples I find show only passing the file but I need to pass the file, and data, and an auth header. I have tried so many variations of my code, and nothing has gotten me any closer.
First, this function is what works, but with a different API (groups) for posting a new group. It doesn't include any files.
def apiPost(token):
if not token == "":
status.run_set_text('Running API POST Call', 3)
headers = { 'Content-Type':'application/json', 'Authorization':'Bearer '+str(token) }
data = {"data":{ 'id':0, 'customerId':33, 'name':'567tuhj', 'description':'sdfgsdfg'}}
r = requests.post(api_url+"v1.1/groups/", headers=headers, data=json.dumps(data))
***other code removed updating status bars***
return r
else:
***other code removed updating status bars***
My dev environment only has access to one customer, but it still requires that I post the customerId to POST.
I've tried hundreds of variations of converting this to posting a file from what I read on the requests site tutorials and some other stackoverflow questions. This posts to the API packageFiles. According to the Swagger page I have access to, it says I need to include the ID and customer ID when uploading the file still.
def apiPost(token):
if not token == "":
status.run_set_text('Running API POST Call', 3)
headers = {'Authorization':'Bearer '+str(token)}
files = {'file': open('log.txt', 'rb')}
data = {"data":{ 'id':0, 'customerId':33}}
r = requests.post(api_url+"v1.1/package_files/"+set_api_pair_value, headers=headers, data=json.dumps(data), file=files)
***other code removed updating status bars***
return r
else:
***other code removed updating status bars***
First, you should format your code to be clean, nice and readable
I am trying to solve your issue, but I think you should attach to your question some kind of expected data, file, request parameters.
def apiPost(token):
if not token == "":
status.run_set_text('Running API POST Call', 3)
headers = { 'Content-Type':'application/json',
'Authorization':'Bearer '+str(token) }
data = {"data":
{ 'id':0,
'customerId':33,
'name':'567tuhj',
'description':'sdfgsdfg'
}
}
#The part you are looking for probably
files = {'name_of_file_field' : open(filename, "rb")}
r = requests.post(api_url+"v1.1/groups/", headers=headers, data=json.dumps(data), files = files)
***other code removed updating status bars***
return r
else:
***other code removed updating status bars***
Comment:I get back a 400 (bad request) with that accepted answer..
{"status":"FAILURE",
"error":[{"code":"5004",
"name":"General Error","severity":"3","message":"Error Occurred During the operation",
"details":{"5004":"General Error null"}}]}
{'Connection': 'keep-alive', 'X-Powered-By': 'Undertow/1', 'Server': 'WildFly/9', 'Content-Length': '172', 'Content-Type': 'application/json;charset=UTF-8', 'X-Application-Context': 'application:9090', 'Date': 'Fri, 28 Sep 2018 17:57:57 GMT'}
Please edit your Question and add your Python and requests Version!
Tried the following, using a Text file instead of Image:
import requests
url = 'http://httpbin.org/anything'
files = {'file': ('helloworld.txt', open('../test/helloworld.txt', 'rb'), 'text/text')}
data = dict(name='barca', country='spain')
r = requests.post(url, files=files, data=data)
# Print my requests.post header, files and data.
r_dict = r.json()
for key in r_dict:
print('{}:{}'.format(key, r_dict[key]))
Response from http://httpbin.org/anything:
<Response [200]>
My requests.post header, files and data, send back from host.
method:POST
files:{'file': 'Hello World'}
url:http://httpbin.org/anything
form:{'name': 'barca', 'country': 'spain'}
origin:xx.xx.xx.xx
args:{}
headers:{'Content-Length': '369',
'Accept': '*/*',
'Content-Type': 'multipart/form-data; boundary=bc3c1927f542430f8166e8f3f27f3c72',
'Host': 'httpbin.org', 'Connection': 'close',
'User-Agent': 'python-requests/2.11.1',
'Accept-Encoding': 'gzip, deflate'}
json:None
data:
Tested with Python:3.4.2 - requests:2.11.1
I can't figure out how to call this api correctly using python urllib or requests.
Let me give you the code I have now:
import requests
url = "http://api.cortical.io:80/rest/expressions/similar_terms?retina_name=en_associative&start_index=0&max_results=1&sparsity=1.0&get_fingerprint=false"
params = {"positions":[0,6,7,29]}
headers = { "api-key" : key,
"Content-Type" : "application/json"}
# Make a get request with the parameters.
response = requests.get(url, params=params, headers=headers)
# Print the content of the response
print(response.content)
I've even added in the rest of the parameters to the params variable:
url = 'http://api.cortical.io:80/rest/expressions/similar_terms?'
params = {
"retina_name":"en_associative",
"start_index":0,
"max_results":1,
"sparsity":1.0,
"get_fingerprint":False,
"positions":[0,6,7,29]}
I get this message back:
An internal server error has been logged # Sun Apr 01 00:03:02 UTC
2018
So I'm not sure what I'm doing wrong. You can test out their api here, but even with testing I can't figure it out. If I go out to http://api.cortical.io/, click on the Expression tab, click on the POST /expressions/similar_terms option then paste {"positions":[0,6,7,29]} in the body textbox and hit the button, it'll give you a valid response, so nothing is wrong with their API.
I don't know what I'm doing wrong. can you help me?
The problem is that you're mixing query string parameters and post data in your params dictionary.
Instead, you should use the params parameter for your query string data, and the json parameter (since the content type is json) for your post body data.
When using the json parameter, the Content-Type header is set to 'application/json' by default. Also, when the response is json you can use the .json() method to get a dictionary.
An example,
import requests
url = 'http://api.cortical.io:80/rest/expressions/similar_terms?'
params = {
"retina_name":"en_associative",
"start_index":0,
"max_results":1,
"sparsity":1.0,
"get_fingerprint":False
}
data = {"positions":[0,6,7,29]}
r = requests.post(url, params=params, json=data)
print(r.status_code)
print(r.json())
200
[{'term': 'headphones', 'df': 8.991197733061748e-05, 'score': 4.0, 'pos_types': ['NOUN'], 'fingerprint': {'positions': []}}]
So, I can't speak to why there's a server error in a third-party API, but I followed your suggestion to try using the API UI directly, and noticed you're using a totally different endpoint than the one you're trying to call in your code. In your code you GET from http://api.cortical.io:80/rest/expressions/similar_terms but in the UI you POST to http://api.cortical.io/rest/expressions/similar_terms/bulk. It's apples and oranges.
Calling the endpoint you mention in the UI call works for me, using the following variation on your code, which requires using requests.post, and as was also pointed out by t.m. adam, the json parameter for the payload, which also needs to be wrapped in a list:
import requests
url = "http://api.cortical.io/rest/expressions/similar_terms/bulk?retina_name=en_associative&start_index=0&max_results=1&sparsity=1.0&get_fingerprint=false"
params = [{"positions":[0,6,7,29]}]
headers = { "api-key" : key,
"Content-Type" : "application/json"}
# Make a get request with the parameters.
response = requests.post(url, json=params, headers=headers)
# Print the content of the response
print(response.content)
Gives:
b'[[{"term":"headphones","df":8.991197733061748E-5,"score":4.0,"pos_types":["NOUN"],"fingerprint":{"positions":[]}}]]'
I am using the requests library to make a POST request in order to obtain an access token. My request works properly, but, I'm not sure how to extract it and then use it in a GET request.
url = 'https://login.insideview.com/Auth/login/v1/token'
payload = {'clientId' : '****', 'clientSecret' : '****','grantType':'cred'}
headers = { 'Accept' : 'application/json'}
r = requests.post(url, headers=headers, params=payload)
solution:
data = json.loads(r.text)
data['accessTokenDetails']['accessToken']
Returns:
{"accessTokenDetails":{"accessToken":"the_access_token","tokenType":"bearer","expirationTime":"Fri, Mar 25, 2016 09:59:53 PM GMT","userInfo":{"userId":null,"firstName":null,"lastName":null,"userName":null,"companyName":null,"accountId":null,"role":null}}}
If it returns a dict, why not just access its contents as usual?
token = r['accessTokenDetails']['accessToken']
#michael-queue the response from the request to a JSON endpoint is a JSON encoded string. To load it into a dictionary and access inner properties it's needed to json.loads(json_string) into Python. For the opposite operation, to dump a dictionary into a JSON string is json.dumps(dictionary) instead.
Using WheniWork's api, I need to use a token for authentication, and I also need to send data to create a new user. Does the order or name of arguments I send with requests.post() matter?
If I'm just using GET to pull information, I can have the url contain the thing I'm looking for, and then send a payload that is the token. For example:
url = 'https://api.wheniwork.com/2/users/2450964'
payload = {"W-Token": "ilovemyboss"}
r = requests.get(url, params=payload)
print r.text
When I try to add a new user however, I'm either not able to authenticate or not passing the data correctly. The api reference shows this format for using cURL:
curl https://api.wheniwork.com/2/users --data '{"first_name":"FirstName", "last_name": "LastName", "email": "user#email.com"}' -H "W-Token: ilovemyboss"
Here's what I've written out in python (2.7.10) using Requests:
url = 'https://api.wheniwork.com/2/users'
data={'first_name':'TestFirst', 'last_name': 'TestLast','email':'test#aol.com'}
params={"W-Token": "ilovemyboss"}
r = requests.post(url, data=data, params=params)
print r.text
Can someone explain if/how data(the user) gets sent separately from authentication(the token)?
I found the issue!
The data (user dict) needs to be in quotes. I'm not sure if their API is expecting a string, or if that's how requests works, or what. But here's the solution:
url = 'https://api.wheniwork.com/2/users'
data = "{'first_name':'TestFirst', 'last_name': 'TestLast','email':'test#aol.com'}"
params = {"W-Token": "ilovemyboss"}
r = requests.post(url, data=data, params=params)
print r.text
We can solve the above problem by converting the data dictionary to JSON string by using json.dumps.
data={'first_name':'TestFirst', 'last_name': 'TestLast','email':'test#aol.com'}
r = requests.post(url, data=json.dumps(data), params=params)
print r.text