I'm collecting some market data from Binance's API. My goal is to collect the list of all markets and use the 'status' key included in each row to detect if the market is active or not. If it's not active, I must search the last trade to collect the date of the market's shutdown.
I wrote this code
import requests
import pandas as pd
import json
import csv
url = 'https://api.binance.com/api/v3/exchangeInfo'
trade_url = 'https://api.binance.com/api/v3/trades?symbol='
response = requests.get(url)
data = response.json()
df = data['symbols'] #list of dict
json_data=[]
with open(r'C:\Users\Utilisateur\Desktop\json.csv', 'a' , encoding='utf-8', newline='') as j :
wr=csv.writer(j)
wr.writerow(["symbol","last_trade"])
for i in data['symbols'] :
if data[i]['status'] != "TRADING" :
trades_req = requests.get(trade_url + i)
print(trades_req)
but I got this error
TypeError: unhashable type: 'dict'
How can I avoid it?
That's because i is a dictionary. If data['symbols'] is a list of dictionaries, when you do in the loop:
for i in data['symbols']:
if data[i]['status'] ...
you are trying to hash i to use it as a key of data. I think you want to know the status of each dictionary on the list. That is:
for i in data['symbols']:
if i['status'] ...
In such a case, it would be better to use more declarative variable names, e.g., d, s, symbol instead of i.
Related
I have the following piece of python code, which is supposed to access a JSON file stored in an s3 bucket, extract the organisations for each JSON object and then place them in a list. There is a for loop then which should go through the list and apply the organisation name to the ending /bitgear/IO-Air to create the appropriate IoT topic.
for uuid_index, uuid in enumerate(uuid_list):
s3 = boto3.resource('s3')
client = boto3.client('s3')
bucket = '3deo-sensor-data'
key = 'simulated/config/IoT-sim-config.json'
obj = s3.Object(bucket, key)
data = obj.get()['Body'].read().decode('utf-8')
json_data = json.loads(data)
dframe = pd.DataFrame(json_data, columns= ['organisation'])
org = dframe.values.tolist()
for orgs in org():
TOPIC = org[orgs] + '/bitgear/IO-Air'
However, I am getting the error 'list' object is not callable
Here is a snippet of how the JSON file looks:
[
{
"uuid": "1597c163-6fbf-4f46-8ff6-1e9eb4f07e34",
"organisation": "port_36",
"device_vendor": "bitgear",
"device_type": "IO-Air",
"client_id": "AQ_2"
},
{
"uuid": "cde2107e-8736-47de-9e87-2033c3063589",
"organisation": "hchjffsd2immvavb7jiqtedp",
"device_vendor": "bitgear",
"device_type": "IO-Air",
"client_id": "IoT_Sim_1"
}
]
Can anyone advise how to best form my desired IoT topic using the organisation name?
You don't need to use pandas for this, json.loads() deserialises a string to a Python object. Therefore the json loaded in becomes a list of dictionaries, so you can use:
import json
topics = []
for uuid_index, uuid in enumerate(uuid_list):
s3 = boto3.resource('s3')
client = boto3.client('s3')
bucket = '3deo-sensor-data'
key = 'simulated/config/IoT-sim-config.json'
obj = s3.Object(bucket, key)
data = obj.get()['Body'].read().decode('utf-8')
json_data = json.loads(data)
for data in json_data:
topics.append(data["organisation"] + '/bitgear/IO-Air')
Which leaves topics containing:
['port_36/bitgear/IO-Air', 'hchjffsd2immvavb7jiqtedp/bitgear/IO-Air']
You can then iterate through each of the topics in the topics list.
Change this part of you code:
for orgs in org:
TOPIC = orgs + '/bitgear/IO-Air'
Looking at you code, I think a much cleaner code would be to first convert your whole json to dataframe with all the columns using:
df = pd.DataFrame.from_records(json_data)
This will give you a dataframe with all the required fields and then you can simply append a string to all the column values of "organisation" column as below:
df['organisation'] = df['col'].astype(str) + '/bitgear/IO-Air'
To get this column values as list, you can simply use .tolist function as below:
df['organisation'].tolist()
for uuid_index, uuid in list(enumerate(uuid_list)):
try this in the first line of your code snippet. This will create a list object of elements with two items the index and the uuid which seems to be what you are trying to do.
First of all org = dframe.values.tolist() returns a list to which you can iterate but not call using (). So, for loop should be for orgs in org:.
Other thing I noticed is the following:
for orgs in org():
TOPIC = org[orgs] + '/bitgear/IO-Air'
When iterate over a list you get values directly not indices, so you can't do org[orgs] in your case, orgs is the element in the list itself.
On the other hand, if you want index you can do for idx, value in enumerate(org):
I am trying to pull out multiple ticker data from the yfinance API and save it to a csv file (in total I have 1000 tickers I need to get the data for, that data being the entire table of date, open, high, low, close, volume, etc etc), so far I am able to successfully get data for 1 ticker by using the following Python code:
import yfinance as yf
def yfinance(ticker_symbol):
ticker_data = yf.Ticker(ticker_symbol)
tickerDF = ticker_data.history(period='1d', start='2020-09-30', end='2020-10-31')
print(tickerDF)
yfinance('000001.SS')
However if I try on multiple tickers this doesn't work. Following the yfinance docs which say for multiple tickers use:
tickers = yf.Tickers('msft aapl goog')
# ^ returns a named tuple of Ticker objects
# access each ticker using (example)
tickers.tickers.MSFT.info
tickers.tickers.AAPL.history(period="1mo")
tickers.tickers.GOOG.actions
I have a couple of issue here, the docs use a string such as 'aapl' my tickers are all of digit format like '000001.SS', the ".SS" part is proving to be an issue when passing it into the code:
tickers.tickers.000001.SS.history(period="1mo")
# Clearly this wont for for a start
The next issue I am having is, even if I pass in for example 3 tickers to my function like so:
yfinance('000001.SS 000050.KS 00006.KS')
# similar to yfinance docs of tickers = yf.Tickers('msft aapl goog')
I get errors like:
AttributeError: 'Tickers' object has no attribute '000001.SS'
(I have also tried to run these into a for loop and pass each on to the Tickers object but get the same error.)
Im stuck now, I dont know how to pass in multiple tickers to yfinance and get back data that I want and the docs aren't very helpful.
Is anyone able to help me with this?
Could you not just store them in an array specifying the type as dtype object then use that pull the data from.
import yfinance as yf
import numpy as np
tickers = ['msft', 'aapl', 'goog']
totalPortfolio = np.empty([len(tickers)], dtype=object)
num = 0
for ticker in tickers:
totalPortfolio[num] = yf.download(ticker, start='2020-09-30', end='2020-10-31', interval="1d")
num = num + 1
Take a look at the code below:
test = yf.Tickers("A B C")
# creates test as a yf.tickers object
test_dict = test.tickers
# creates a dict object containing the individual tickers. Can be checked with type()
You are trying to use "tickers.tickers.MSFT.info" to retrieve the ticker data from your dictionary "tickers.tickers" but like your error message says, a dict object has no attributes named after your specific ticker names. This is in general not how you access elements in a dictionary.
Instead you should use the code as below (like with all dict objects):
#old code from above
test = yf.Tickers("A B C")
test_dict = test.tickers
#new code accessing the dict correctly
a_data = test_dict["A"]
a_data = test.tickers["A"] #does the same as the line above
b_data = test.tickers["B"] #and so on for the other tickers
In a loop this could look something like this:
ticker_list = ["A", "B", "C"] #add tickers as needed
tickers_data = {}
tickers_history = {}
for ticker in ticker_list:
tickers_data[ticker] = yf.Ticker(ticker)
tickers_history = tickers_data[ticker].history(period='1d', start='2020-09-30', end='2020-10-31')
#access the dicts as needed using tickers_data[" your ticker name "]
alternatively you can also use the "yf.Tickers" function to retrieve multiple tickers at once, but because you save the history seperately I don't think this will necessarily improve your code much.
You should pay attention however, that "yf.Ticker()" and "yf.Tickers()" are different functions from each other with differing syntax and are not interchangeable.
You did mix that up when you tried accessing multiple tickers with your custom "yfinance()" function, that has been previously defined with the "yf.Ticker()" function and thus only accepts one symbol at a time.
Here I try to calculate mean value based on the data in two list of dicts. Although I used same code before, I keep getting error. Is there any solution?
import pandas as pd
data = pd.read_csv('data3.csv',sep=';') # Reading data from csv
data = data.dropna(axis=0) # Drop rows with null values
data = data.T.to_dict().values() # Converting dataframe into list of dictionaries
newdata = pd.read_csv('newdata.csv',sep=';') # Reading data from csv
newdata = newdata.T.to_dict().values() # Converting dataframe into list of dictionaries
score = []
for item in newdata:
score.append({item['Genre_Name']:item['Ranking']})
from statistics import mean
score={k:int(v) for i in score for k,v in i.items()}
for item in data:
y= mean(map(score.get,map(str.strip,item['Recommended_Genres'].split(','))))
print(y)
Too see csv files: https://repl.it/#rmakakgn/SVE2
.get method of dict return None if given key does not exist and statistics.mean fail due to that, consider that
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
print(statistics.mean(data))
result in:
TypeError: can't convert type 'NoneType' to numerator/denominator
You need to remove Nones before feeding into statistics.mean, which you can do using list comprehension:
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
data = [i for i in data if i is not None]
print(statistics.mean(data))
or filter:
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
data = filter(lambda x:x is not None,data)
print(statistics.mean(data))
(both snippets above code will print 2)
In this particular case, you might get filter effect by replacing:
mean(map(score.get,map(str.strip,item['Recommended_Genres'].split(','))))
with:
mean([i for i in map(score.get,map(str.strip,item['Recommended_Genres'].split(','))) if i is not None])
though as with most python built-in and standard library functions accepting list as sole argument, you might decide to not build list but feed created generator directly i.e.
mean(i for i in map(score.get,map(str.strip,item['Recommended_Genres'].split(','))) if i is not None)
For further discussion see PEP 202 xor PEP 289.
I am attempting to format this Api
https://www.binance.com/api/v1/ticker/allBookTickers
Here is an abbreviated version of the Api
[{"symbol":"ETHBTC","bidPrice":"0.07200500","bidQty":"0.67800000","askPrice":"0.07203200","askQty":"7.19200000"},{"symbol":"LTCBTC","bidPrice":"0.01281100","bidQty":"10.90000000","askPrice":"0.01282500","askQty":"1.01000000"}]
Each dict is saved as an index in the list my issue is with the fact that each dict starts with 'symbol' rather then the name like 'ETHBTC'
I can call the index number but as their is hundreds of dicts in the api I need to find a method of being able to type in for instance 'ETHBTC' to call that dict?
This is what it would look like in an ideal world but I have no idea how to achieve this any help would be greatly appreciated?
> data = requests.get('https://www.binance.com/api/v1/ticker/allBookTickers')
> data = data.json()
> ltc = data['LTCBTC']
Use the following code:-
import requests
#fetched data from url using requests
data = requests.get('https://www.binance.com/api/v1/ticker/allBookTickers')
# creating json object from response
dataJson = data.json()
# creating dictionary from json object using symbol value as key
dataDictionary = {d['symbol'] : d for d in dataJson}
# accessing dictionary object using symbol value
ltc = dataDictionary['LTCBTC']
print ltc
# now you can use ltc values by given keys as, and so on for other values
print ltc['askPrice']
In this code we created python dictionary from the response returned.
The api call
from rtstock.stock import Stock
stock = Stock('AAPL')
data = stock.get_latest_price()
print(data)
yeilds
[{'LastTradeTime': '2:54pm', 'LastTradePriceOnly': '119.855'}]
I'm trying to print 119.855 with no apostrophes using
from rtstock.stock import Stock
stock = Stock('AAPL')
data = stock.get_latest_price()
print(data['LastTradePriceOnly'])
and am getting the error message
print(data['LastTradePriceOnly'])
TypeError: list indices must be integers or slices, not str
Any help is appreciated, Thanks!!
data is a list containing a single dictionary:
data = [{'LastTradeTime': '2:54pm', 'LastTradePriceOnly': '119.855'}]
print(data) # list containing a dictionary
print(data[0]) # the dictionary
print(data[0]['LastTradePriceOnly']) # 119.855