I'm trying to get the last data from the bitmex API
Base URI: https://www.bitmex.com/api/v1
I don't really understand how to get the last data (from today) using filters : https://www.bitmex.com/app/restAPI
here is my code:
from datetime import date
import requests
import json
import pandas as pd
today = date.today()
d1 = today.strftime("%Y-%m-%d")
#print("d1 =", d1)
def parser():
today = date.today()
# yy/dd/mm
d1 = today.strftime("%Y-%m-%d")
# print("d1 =", d1)
return f'https://www.bitmex.com/api/v1/trade?symbol=.BVOL24H&startTime={d1}×tamp.time=12:00:00.000&columns=price'
# Making a get request
response = requests.get(parser()).json()
# print(response)
for elem in response:
print(elem)
and the response is :
...
{'symbol': '.BVOL24H', 'timestamp': '2021-12-27T08:05:00.000Z', 'price': 2.02}
{'symbol': '.BVOL24H', 'timestamp': '2021-12-27T08:10:00.000Z', 'price': 2.02}
{'symbol': '.BVOL24H', 'timestamp': '2021-12-27T08:15:00.000Z', 'price': 2.02}
it's missing a few hours, I tried using endTime, StartTime and Count without success..
I think I need to pass another filter like endtime = now and timestamp.time = now but I don't know how to send a payload or how to url-encode it.
As Filtering part tells
Many table endpoints take a filter parameter. This is expected to be JSON
These parameters are not keys in the query string, but keys in a dictionary given in filter key
url = "https://www.bitmex.com/api/v1/trade"
filters = {
'startTime': date(2021, 12, 20).strftime("%Y-%m-%d"),
'timestamp.time': '12:00:00.000'
}
params = {
'symbol': '.BVOL24H',
'filter': json.dumps(filters),
}
response = requests.get(url, params=params)
for elem in response.json():
print(elem)
Example
/trade?symbol=.BVOL24H&filter={%22startTime%22:%222021-12-20%22,%22timestamp.time%22:%2212:00:00.000%22}
You can add additional parameters to the url with & like below.
'https://www.bitmex.com/api/v1/trade?symbol=.BVOL24H&startTime={d1}×tamp.time=12:00:00.000&columns=price&endTime={date.today()}×tamp.time={date.today()}'
Related
I'm a python beginner. I would like to ask for help regarding the retrieve the response data. Here's my script:
import pandas as pd
import re
import time
import requests as re
import json
response = re.get(url, headers=headers, auth=auth)
data = response.json()
Here's a part of json response:
{'result': [{'display': '',
'closure_code': '',
'service_offer': 'Integration Platforms',
'updated_on': '2022-04-23 09:05:53',
'urgency': '2',
'business_service': 'Operations',
'updated_by': 'serviceaccount45',
'description': 'ALERT returned 400 but expected 200',
'sys_created_on': '2022-04-23 09:05:53',
'sys_created_by': 'serviceaccount45',
'subcategory': 'Integration',
'contact_type': 'Email',
'problem_type': 'Design: Availability',
'caller_id': '',
'action': 'create',
'company': 'aaaa',
'priority': '3',
'status': '1',
'opened': 'smith.j',
'assigned_to': 'doe.j',
'number': '123456',
'group': 'blabla',
'impact': '2',
'category': 'Business Application & Databases',
'caused_by_change': '',
'location': 'All Locations',
'configuration_item': 'Monitor',
},
I would like to extract the data only for one group = 'blablabla'. Then I would like to extract fields such as:
number = data['number']
group = data['group']
service_offer = data['service_offer']
updated = data['updated_on']
urgency = data['urgency']
username = data['created_by']
short_desc = data['description']
How it should be done?
I know that to check the first value I should use:
service_offer = data['result'][0]['service_offer']
I've tried to create a dictionary, but, I'm getting an error:
data_result = response.json()['result']
payload ={
number = data_result['number']
group = data_result['group']
service_offer = data_result['service_offer']
updated = data_result['updated_on']
urgency = data_result['urgency']
username = data_result['created_by']
short_desc = data_result['description']
}
TypeError: list indices must be integers or slices, not str:
So, I've started to create something like below., but I'm stuck:
get_data = []
if len(data) > 0:
for item in range(len(data)):
get_data.append(data[item])
May I ask for help?
If data is your decoded json response from the question then you can do:
# find group `blabla` in result:
g = next(d for d in data["result"] if d["group"] == "blabla")
# get data from the `blabla` group:
number = g["number"]
group = g["group"]
service_offer = g["service_offer"]
updated = g["updated_on"]
urgency = g["urgency"]
username = g["sys_created_by"]
short_desc = g["description"]
print(number, group, service_offer, updated, urgency, username, short_desc)
Prints:
123456 blabla Integration Platforms 2022-04-23 09:05:53 2 serviceaccount45 ALERT returned 400 but expected 200
I am trying to interact with the coinbase api and keep getting "TypeError: Object of type method is not JSON serializable" when trying to print out this json data, I know the get request is correct as it returns a 200 when I remove the json.dumps().
import requests
import json
response = requests.get('https://api.coinbase.com/v2/prices/BTC-USD/buy')
data = json.dumps(response.json)
print(data)
Try this:
response = requests.get('https://api.coinbase.com/v2/prices/BTC-USD/buy')
data = json.loads(response.text)
print(data) # {'data': {'base': 'BTC', 'currency': 'USD', 'amount': '40935.10'}}
Your return is:
{'data': {'base': 'BTC', 'currency': 'USD', 'amount': '40880.98'}}
It has a JSON syntax error
Try this
response = requests.get('https://api.coinbase.com/v2/prices/BTC-USD/buy')
data = response.json()
dataJson = json.dumps(data['data'])
print(dataJson)
I am trying to retrieve data from the first available date to present date from an API. I've tried using min and max in parameter.
def getcomplete(cid, pid, tag, type):
api_endpoint = ''
headers = {'token': get_token()['access_token'], 'Content-Type': 'application/json'}
params = {'cid': str(cid),
'from-date': datetime.datetime.min,
'to-date': datetime.datetime.max,
'tag': str(tag),
'type': str(type),
'pid': str(pid)
}
r = requests.post(url=api_endpoint, headers=headers, params=params)
return r.json()
getcomplete(10,12,'x','y')
This returns {'status': 'success', 'message': 'success', 'data': []}.
Is there anything wrong with the written function.
Thanks
Pythons min() and max() have an optional default parameter. This will prevent them from throwing errors
min("", default="")
I am searching a string of text which contains dictionaries that look like so:
soup_string = """{"loadType":"","shiftId":"ROVR-DUMMY-SHIFTID","carbonFriendly":"no","cost":"£2.00","initialSlotPrice":"","timeSlotISO":"2019-06-13T12:00+01:00/13:00+01:00","isSameDayPremium":"false","stopId":"10446315588190612134701380","availability":"full","slotDiscountedByDP":"false","slotId":"1hr-12-13-20190613","time":"12:00pm - 1:00pm","rawSlotPrice":"","slotDiscounted":"false"},
{"loadType":"","shiftId":"ROVR-DUMMY-SHIFTID","carbonFriendly":"no","cost":"£2.00","initialSlotPrice":"","timeSlotISO":"2019-06-13T12:30+01:00/13:30+01:00","isSameDayPremium":"false","stopId":"10446315588190612134701380","availability":"available","slotDiscountedByDP":"false","slotId":"1hr-12:30-13:30-20190613","time":"12:30pm - 1:30pm","rawSlotPrice":"","slotDiscounted":"false"}"""
I am looking to return the string which follows each key in the 'dictionaries'.
I have decided an appropriate method is to use Regex expressions. I can return each times and costs using
Costs = re.findall(r"\£[0-9]\.[0-9][0-9]", soup_string)
times = re.findall(r'\"(time)\"\:\"(.{14,16})\"\,', soup_string)
Essentially I would like to be able to look for each key in the dictionary, and search for a specific string then return the value.
The end goal is to create a dictionary with the 'Cost', 'Availability' and 'time'.
Full code:
import requests
from bs4 import BeautifulSoup
import json
postcode = "L4 0TH"
ASDA_url = "https://groceries.asda.com/api/user/checkpostcode?postcode="+ postcode + "&requestorigin=gi"
ASDA_url2 = "https://groceries.asda.com/api/slot/view?startdate=12%2F06%2F2019&deliveryoption=homedelivery&requestorigin=gi&_="
client = requests.Session()
r = client.get(ASDA_url)
r2 = client.get(ASDA_url2)
soup = BeautifulSoup(r2.text, 'html.parser')
soup_string = str(soup)
soup_dicts = json.loads('[' + soup_string + ']')
keep_keys = ('cost', 'availability', 'time')
filtered = [{k:soup_dict[k] for k in keep_keys} for soup_dict in soup_dicts]```
Given that you have multiple dictionaries, I'm not exactly sure what you're trying to obtain, but from my understanding this should help:
import json
soup_string = ''' ... ''' # As it is in the question
soup_dicts = json.loads('[' + soup_string + ']')
keep_keys = ('cost', 'availability', 'time')
filtered = [{k:soup_dict[k] for k in keep_keys} for soup_dict in soup_dicts]
It treats your string of dictionaries as a list of JSON dictionaries, and uses the json module to parse it. Then it filters out everything except the key/value pairs you need. The result is a list of the filtered dictionaries.
Output (i.e. value of filtered):
[
{'cost': '£2.00', 'availability': 'full', 'time': '12:00pm - 1:00pm'},
{'cost': '£2.00', 'availability': 'available', 'time': '12:30pm - 1:30pm'}
]
EDIT:
In response to you providing your code, I can see that you're calling str on the results from BeautifulSoup. Rather than doing that, you can just process the client.get() results directly:
import json
import requests
postcode = "L4 0TH"
ASDA_url = "https://groceries.asda.com/api/user/checkpostcode?postcode="+ postcode + "&requestorigin=gi"
ASDA_url2 = "https://groceries.asda.com/api/slot/view?startdate=12%2F06%2F2019&deliveryoption=homedelivery&requestorigin=gi&_="
client = requests.Session()
r = client.get(ASDA_url)
r2 = client.get(ASDA_url2)
dicts = r2.json()['slotHeader'][0]['slots']
keep_keys = ('cost', 'availability', 'time')
filtered = [{k:d[k] for k in keep_keys} for d in dicts]
First you need to put your data into a list and create a dictionary with key: data. (see my example below). Then use json to convert it as a dictionary of dictionaries. Then extract cost, availability and time per dictionary on a loop.
import json
soup_string = """{"data": [{"loadType":"","shiftId":"ROVR-DUMMY-SHIFTID","carbonFriendly":"no","cost":"£2.00","initialSlotPrice":"","timeSlotISO":"2019-06-13T12:00+01:00/13:00+01:00","isSameDayPremium":"false","stopId":"10446315588190612134701380","availability":"full","slotDiscountedByDP":"false","slotId":"1hr-12-13-20190613","time":"12:00pm - 1:00pm","rawSlotPrice":"","slotDiscounted":"false"}, {"loadType":"","shiftId":"ROVR-DUMMY-SHIFTID","carbonFriendly":"no","cost":"£2.00","initialSlotPrice":"","timeSlotISO":"2019-06-13T12:30+01:00/13:30+01:00","isSameDayPremium":"false","stopId":"10446315588190612134701380","availability":"available","slotDiscountedByDP":"false","slotId":"1hr-12:30-13:30-20190613","time":"12:30pm - 1:30pm","rawSlotPrice":"","slotDiscounted":"false"}]}"""
d = json.loads(soup_string)
result = []
cost, avail, time = [], [], []
for data in d['data']:
tmp = {}
tmp['Cost'] = data['cost']
tmp['Availability'] = data['availability']
tmp['Time'] = data['time']
result.append(tmp)
result
Output:
[{'Cost': '£2.00', 'Availability': 'full', 'Time': '12:00pm - 1:00pm'},
{'Cost': '£2.00', 'Availability': 'available', 'Time': '12:30pm - 1:30pm'}]
I'm using the Google APIs python client to download some data from Google Analytics. I basically copied one of their exampels and modified it to do exactly what I need.
I took this piece of code from the examples:
request = service.data().ga().get(
ids=ids,
start_date=str(start_date),
end_date=str(end_date),
dimensions=','.join(dimensions),
filters=filters,
sort="ga:date",
metrics=','.join(metrics)
)
Then add it to the batch object, and execute it once it has collected 10 requests. This all works well, but the problem is, some of those requests return a nextLink. Now I could just create a new request object (with the above code) with a different start-index, but isn't there a better way?
Is there a way to just parse the nextLink into a new request object?
I'm using this approach:
firstRun = True
params = {'ids':'ga:00000001',
'start_date':'2013-07-01',
'end_date':'2013-07-31',
'metrics':'ga:visits',
'dimensions':'ga:source',
'sort':'-ga:visits',
'start_index':1,
'max_results':10000}
while firstRun == True or result.get('nextLink'):
if firstRun == False:
params['start_index'] = int(params['start_index']) + int(params['max_results'])
result = service.data().ga().get(**params).execute()
firstRun = False
I could not found a way to parse the nextLink object and do a request with it but this was my solution and works fine:
max_results = 10000
params = {
'ids': 'ga:' + profile_id,
'start_date': start_date,
'end_date': end_date,
'metrics': ','.join(metrics),
'dimensions': ','.join(dimensions),
'start_index': 1,
'max_results': max_results
}
has_more = True
while has_more:
results = service.data().ga().get(**params).execute()
#do something with results
params['start_index'] = int(params['start_index']) + int(params['max_results'])
has_more = results.get('nextLink')
Why we can't just do:
params = {'ids': profile_id,
'start_date': start_date,
'end_date': end_date,
'metrics': metrics,
'dimensions': dimensions,
'sort': sort,
'start_index': 1,
'max_results': 1}
dummy_call = service.data().ga().get(**params).execute() # just to find out the totalResults number
params['max_results'] = dummy_call[u'totalResults'] # replace max_result with the totalResults number
all_rows = service.data().ga().get(**params).execute()[u'rows']
???