Soap API data to Google Sheets - python

Good morning,
I have been having troubles getting soap API data into google sheets. When i run the Soap request I get the data as shown in the image .
[output data][1]
Then i tried getting this data into a google sheets using different methods, unfortunately no solution so far has worked.
The solutions i have tried is pickling the data the setting it in a different file and pushing that file into google sheets.
The current solution I'm working on is setting the output_data in a pandas dataframe and pushing like that, this is the current code but this also doesn't seem to work. I will only leave out the credentials to authenticate with the API.
def pandas_to_sheets(pandas_df, sheet, clear = True):
# Updates all values in a workbook to match a pandas dataframe
if clear:
sheet.clear()
(row, col) = pandas_df.shape
cells = sheet.range("A1:{}".format(gspread.utils.rowcol_to_a1(row + 1, col)))
for cell, val in zip(cells, iter_pd(pandas_df)):
cell.value = val
sheet.update_cells(cells)
def iter_pd(df):
for val in list(df.columns):
yield val
for row in df.values:
for val in list(row):
if pd.isna(val):
yield ""
else:
yield val
optionsReportAffiliateSite = [ {'dateFrom' : '01-01-2020', }]
client = Client(wsdl)
client.service.authenticate(username, password, sandbox, locale, demo)
testReportAffiliateSite = client.service.getReportAffiliateSite(idCampaigns,optionsReportCampaign )
input_dict = zeep.helpers.serialize_object(testReportAffiliateSite)
df = pd.DataFrame(input_dict)
affliatesite = pd.DataFrame(df.values.tolist())[0]
reportdata = pd.DataFrame(df.values.tolist())[1]
pd.json_normalize(affliatesite)
pd.json_normalize(reportdata)
pd.concat([pd.json_normalize(affliatesite), pd.json_normalize(reportdata).reindex(pd.json_normalize(affliatesite).index)], axis=1)
wks = gc.open_by_key('1uPdi2w_1TajnKNN8G3uahgrSHLAPnbAHtHSPeaZN3y0').sheet1
pandas_to_sheets(pd.concat([pd.json_normalize(affliatesite), pd.json_normalize(reportdata).reindex(pd.json_normalize(affliatesite).index)], axis=1), wks)
This gives me the error "TypeError: Object of type Decimal is not JSON serializable"
Many thanks in advance.
[1]: https://i.stack.imgur.com/efOEN.png

The decimal values in the response you're getting cannot be serialized to JSON.
Because of this, you should transform this decimal values to another type which can be serialized. For example, float. So you can do the following:
Define this function, to check if an element is a decimal and return it converted to float:
def f(v):
if isinstance(v, Decimal):
return float(v)
else:
return v
Iterate through your list and call the previous function for every value in it, using this:
soapResponse = map(lambda el : {k: f(v) for k, v in el.items()}, soapResponse)
Note:
Import decimal via from decimal import Decimal.
Reference:
decimal
float
map
Python JSON serialize a Decimal object

Related

dictionary not properly called out in python

I have a made a dictionary and tried to call out the value but I am getting value of the excel cell row number also along with the cell value. I only want the values of the respective cells. Here's what I am getting while I run the code. I have attached the images. And why does it show dtype: object. Am I missing something in the code? or something is wrongly written.
def Trainreadin(actfile, actsheetname):
frame = Input.xclReadIn(actfile, actsheetname)
dict = {'resistance':
[']resistance_name(word)_A(N)_B(N/(km/h))_C(N/(km/h)2)_windT(km/h)_windB(km/h)', 'table']
outputdict = {key: framehandle.value_readin(value) for (key, value) in dict.items()}
outputdict["resistance_A"] = outputdict["resistance"][1]
outputdict["resistance_B"] = outputdict["resistance"][2] * 3.6
print(outputdict["resistance_A"])
return outputdict

How to extract the data from a list of dictionaries?

I'm collecting some market data from Binance's API. My goal is to collect the list of all markets and use the 'status' key included in each row to detect if the market is active or not. If it's not active, I must search the last trade to collect the date of the market's shutdown.
I wrote this code
import requests
import pandas as pd
import json
import csv
url = 'https://api.binance.com/api/v3/exchangeInfo'
trade_url = 'https://api.binance.com/api/v3/trades?symbol='
response = requests.get(url)
data = response.json()
df = data['symbols'] #list of dict
json_data=[]
with open(r'C:\Users\Utilisateur\Desktop\json.csv', 'a' , encoding='utf-8', newline='') as j :
wr=csv.writer(j)
wr.writerow(["symbol","last_trade"])
for i in data['symbols'] :
if data[i]['status'] != "TRADING" :
trades_req = requests.get(trade_url + i)
print(trades_req)
but I got this error
TypeError: unhashable type: 'dict'
How can I avoid it?
That's because i is a dictionary. If data['symbols'] is a list of dictionaries, when you do in the loop:
for i in data['symbols']:
if data[i]['status'] ...
you are trying to hash i to use it as a key of data. I think you want to know the status of each dictionary on the list. That is:
for i in data['symbols']:
if i['status'] ...
In such a case, it would be better to use more declarative variable names, e.g., d, s, symbol instead of i.

TypeError: can't convert type 'NoneType' to numerator/denominator

Here I try to calculate mean value based on the data in two list of dicts. Although I used same code before, I keep getting error. Is there any solution?
import pandas as pd
data = pd.read_csv('data3.csv',sep=';') # Reading data from csv
data = data.dropna(axis=0) # Drop rows with null values
data = data.T.to_dict().values() # Converting dataframe into list of dictionaries
newdata = pd.read_csv('newdata.csv',sep=';') # Reading data from csv
newdata = newdata.T.to_dict().values() # Converting dataframe into list of dictionaries
score = []
for item in newdata:
score.append({item['Genre_Name']:item['Ranking']})
from statistics import mean
score={k:int(v) for i in score for k,v in i.items()}
for item in data:
y= mean(map(score.get,map(str.strip,item['Recommended_Genres'].split(','))))
print(y)
Too see csv files: https://repl.it/#rmakakgn/SVE2
.get method of dict return None if given key does not exist and statistics.mean fail due to that, consider that
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
print(statistics.mean(data))
result in:
TypeError: can't convert type 'NoneType' to numerator/denominator
You need to remove Nones before feeding into statistics.mean, which you can do using list comprehension:
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
data = [i for i in data if i is not None]
print(statistics.mean(data))
or filter:
import statistics
d = {"a":1,"c":3}
data = [d.get(x) for x in ("a","b","c")]
data = filter(lambda x:x is not None,data)
print(statistics.mean(data))
(both snippets above code will print 2)
In this particular case, you might get filter effect by replacing:
mean(map(score.get,map(str.strip,item['Recommended_Genres'].split(','))))
with:
mean([i for i in map(score.get,map(str.strip,item['Recommended_Genres'].split(','))) if i is not None])
though as with most python built-in and standard library functions accepting list as sole argument, you might decide to not build list but feed created generator directly i.e.
mean(i for i in map(score.get,map(str.strip,item['Recommended_Genres'].split(','))) if i is not None)
For further discussion see PEP 202 xor PEP 289.

Python Loop Addition

No matter what I do I don't seem to be able to add all the base volumes and quote volumes together easily! I want to end up with a total base volume and a total quote volume of all the data in the data frame. Can someone help me on how you can do this easily?
I have tried summing and saving the data in a dictionary first and then adding it but I just don't seem to be able to make this work!
import urllib
import pandas as pd
import json
def call_data(): # Call data from Poloniex
global df
datalink = 'https://poloniex.com/public?command=returnTicker'
df = urllib.request.urlopen(datalink)
df = df.read().decode('utf-8')
df = json.loads(df)
global current_eth_price
for k, v in df.items():
if 'ETH' in k:
if 'USDT_ETH' in k:
current_eth_price = round(float(v['last']),2)
print("Current ETH Price $:",current_eth_price)
def calc_volumes(): # Calculate the base & quote volumes
global volume_totals
for k, v in df.items():
if 'ETH' in k:
basevolume = float(v['baseVolume'])*current_eth_price
quotevolume = float(v['quoteVolume'])*float(v['last'])*current_eth_price
if quotevolume > 0:
percentages = (quotevolume - basevolume) / basevolume * 100
volume_totals = {'key':[k],
'basevolume':[basevolume],
'quotevolume':[quotevolume],
'percentages':[percentages]}
print("volume totals:",volume_totals)
print("#"*8)
call_data()
calc_volumes()
A few notes:
For the next 2 years don't use the keyword globals for anything.
put function documentation under the function in quotes
using the requests library will be much easier than urllib. However ...
pandas can fetch the JSON and parse it all in one step
ok it doesn't have to be as split up as this, I'm just showing you how to properly pass variables around instead of globals.
I could not find "ETH" by itself. In the data they sent they have these 3 ['BTC_ETH', 'USDT_ETH', 'USDC_ETH']. So I used "USDT_ETH" I hope the substitution is ok.
calc_volumes is seeming to do the calculation and being some sort of filter (it's picky as to what it prints). This function needs to be broken up in to it's two separate jobs. printing and calculating. (maybe there was a filter step but I leave that for homework)
.
import pandas as pd
eth_price_url = 'https://poloniex.com/public?command=returnTicker'
def get_data(url=''):
""" Call data from Poloniex and put it in a dataframe"""
data = pd.read_json(url)
return data
def get_current_eth_price(data = None):
""" grab the price out of the dataframe """
current_eth_price = data['USDT_ETH']['last'].round(2)
return current_eth_price
def calc_volumes(data=None, current_eth_price=None):
""" Calculate the base & quote volumes """
data = df[df.columns[df.columns.str.contains('ETH')]].loc[['baseVolume', 'quoteVolume', 'last']]
data = data.transpose()
data[['baseVolume','quoteVolume']]*= current_eth_price
data['quoteVolume']*=data['last']
data['percentages']=(data['quoteVolume'] - data['baseVolume']) / data['quoteVolume'] * 100
return data
df = get_data(url = eth_price_url)
the_price = get_current_eth_price(data = df)
print(f'the current eth price is: {the_price}')
volumes = calc_volumes(data=df, current_eth_price=the_price)
print(volumes)
This code seems kind of odd and inconsistent... for example, you're importing pandas and calling your variable df but you're not actually using dataframes. If you used df = pd.read_json('https://poloniex.com/public?command=returnTicker', 'index')* to get a dataframe, most of your data manipulation here would become much easier, and wouldn't require any loops either.
For example, the first function's code would become as simple as current_eth_price = df.loc['USDT_ETH','last'].
The second function's code would basically be
eth_rows = df[df.index.str.contains('ETH')]
total_base_volume = (eth_rows.baseVolume * current_eth_price).sum()
total_quote_volume = (eth_rows.quoteVolume * eth_rows['last'] * current_eth_price).sum()
(*The 'index' argument tells pandas to read the JSON dictionary indexed by rows, then columns, rather than columns, then rows.)

DataFrame constructor not properly called! error

I am new to Python and I am facing problem in creating the Dataframe in the format of key and value i.e.
data = [{'key':'\[GlobalProgramSizeInThousands\]','value':'1000'},]
Here is my code:
columnsss = ['key','value'];
query = "select * from bparst_tags where tag_type = 1 ";
result = database.cursor(db.cursors.DictCursor);
result.execute(query);
result_set = result.fetchall();
data = "[";
for row in result_set:
`row["tag_expression"]`)
data += "{'value': %s , 'key': %s }," % ( `row["tag_expression"]`, `row["tag_name"]` )
data += "]" ;
df = DataFrame(data , columns=columnsss);
But when I pass the data in DataFrame it shows me
pandas.core.common.PandasError: DataFrame constructor not properly called!
while if I print the data and assign the same value to data variable then it works.
You are providing a string representation of a dict to the DataFrame constructor, and not a dict itself. So this is the reason you get that error.
So if you want to use your code, you could do:
df = DataFrame(eval(data))
But better would be to not create the string in the first place, but directly putting it in a dict. Something roughly like:
data = []
for row in result_set:
data.append({'value': row["tag_expression"], 'key': row["tag_name"]})
But probably even this is not needed, as depending on what is exactly in your result_set you could probably:
provide this directly to a DataFrame: DataFrame(result_set)
or use the pandas read_sql_query function to do this for you (see docs on this)
Just ran into the same error, but the above answer could not help me.
My code worked fine on my computer which was like this:
test_dict = {'x': '123', 'y': '456', 'z': '456'}
df=pd.DataFrame(test_dict.items(),columns=['col1','col2'])
However, it did not work on another platform. It gave me the same error as mentioned in the original question. I tried below code by simply adding the list() around the dictionary items, and it worked smoothly after:
df=pd.DataFrame(list(test_dict.items()),columns=['col1','col2'])
Hopefully, this answer can help whoever ran into a similar situation like me.
import json
# Opening JSON file
f = open('data.json')
# returns JSON object as
# a dictionary
data1 = json.load(f)
#converting it into dataframe
df = pd.read_json(data1, orient ='index')

Categories

Resources