Develop Reference

How to convert independent output lists to a dataframe

Hope you are having a great weekend. My problem is as follows:
For my designed model i am getting the following predictions:
[0.3182012736797333, 0.6817986965179443, 0.5067878365516663, 0.49321213364601135, 0.4795221984386444, 0.520477831363678, 0.532780110836029, 0.46721988916397095, 0.3282901346683502, 0.6717098355293274]
[0.362120658159256, 0.6378793120384216, 0.5134761929512024, 0.4865237772464752, 0.46048662066459656, 0.539513349533081, 0.5342788100242615, 0.4657211899757385, 0.34932515025138855, 0.6506748199462891]
[0.3647380471229553, 0.6352618932723999, 0.5087167620658875, 0.49128326773643494, 0.4709164798259735, 0.5290834903717041, 0.5408024787902832, 0.4591975510120392, 0.37024226784706116, 0.6297577023506165]
[0.43765324354171753, 0.5623468160629272, 0.505147397518158, 0.49485257267951965, 0.45281311869621277, 0.5471869111061096, 0.5416161417961121, 0.45838382840156555, 0.3789178133010864, 0.6210821866989136]
[0.44772887229919434, 0.5522711277008057, 0.5119441151618958, 0.48805591464042664, 0.46322566270828247, 0.5367743372917175, 0.5402485132217407, 0.45975151658058167, 0.4145151972770691, 0.5854847431182861]
[0.35674020648002625, 0.6432597637176514, 0.48104971647262573, 0.5189502835273743, 0.4554695188999176, 0.54453045129776, 0.5409557223320007, 0.45904430747032166, 0.3258989453315735, 0.6741010546684265]
[0.3909384310245514, 0.609061598777771, 0.4915180504322052, 0.5084819793701172, 0.45033228397369385, 0.5496677160263062, 0.5267384052276611, 0.47326159477233887, 0.34493446350097656, 0.6550655364990234]
[0.32971733808517456, 0.6702827215194702, 0.5224012732505798, 0.47759872674942017, 0.4692566692829132, 0.5307433605194092, 0.5360044836997986, 0.4639955163002014, 0.41811054944992065, 0.5818894505500793]
[0.37096619606018066, 0.6290338039398193, 0.5165190100669861, 0.4834809899330139, 0.4739859998226166, 0.526013970375061, 0.5340168476104736, 0.46598318219184875, 0.3438771069049835, 0.6561229228973389]
[0.4189890921115875, 0.5810109376907349, 0.52749103307724, 0.47250890731811523, 0.44485437870025635, 0.5551456212997437, 0.5398098230361938, 0.46019014716148376, 0.3739124536514282, 0.6260875463485718]
[0.3979812562465668, 0.6020187139511108, 0.5050275325775146, 0.49497246742248535, 0.4653399884700775, 0.5346599817276001, 0.537341833114624, 0.4626581072807312, 0.33742010593414307, 0.6625799536705017]
[0.368088960647583, 0.631911039352417, 0.49925288558006287, 0.5007471442222595, 0.4547160863876343, 0.545283854007721, 0.5408452749252319, 0.45915472507476807, 0.4053747355937958, 0.5946252346038818]
As you can see they are independent lists. I want to convert these lists into a dataframe. Although they are independent, they are coming out of a for loop, so i cannot append them because they are not coming at once.

Use:
data = [[0.3182012736797333, 0.6817986965179443, 0.5067878365516663, 0.49321213364601135, 0.4795221984386444, 0.520477831363678, 0.532780110836029, 0.46721988916397095, 0.3282901346683502, 0.6717098355293274],
[0.362120658159256, 0.6378793120384216, 0.5134761929512024, 0.4865237772464752, 0.46048662066459656, 0.539513349533081, 0.5342788100242615, 0.4657211899757385, 0.34932515025138855, 0.6506748199462891],
[0.3647380471229553, 0.6352618932723999, 0.5087167620658875, 0.49128326773643494, 0.4709164798259735, 0.5290834903717041, 0.5408024787902832, 0.4591975510120392, 0.37024226784706116, 0.6297577023506165],
[0.43765324354171753, 0.5623468160629272, 0.505147397518158, 0.49485257267951965, 0.45281311869621277, 0.5471869111061096, 0.5416161417961121, 0.45838382840156555, 0.3789178133010864, 0.6210821866989136],
[0.44772887229919434, 0.5522711277008057, 0.5119441151618958, 0.48805591464042664, 0.46322566270828247, 0.5367743372917175, 0.5402485132217407, 0.45975151658058167, 0.4145151972770691, 0.5854847431182861],
[0.35674020648002625, 0.6432597637176514, 0.48104971647262573, 0.5189502835273743, 0.4554695188999176, 0.54453045129776, 0.5409557223320007, 0.45904430747032166, 0.3258989453315735, 0.6741010546684265],
[0.3909384310245514, 0.609061598777771, 0.4915180504322052, 0.5084819793701172, 0.45033228397369385, 0.5496677160263062, 0.5267384052276611, 0.47326159477233887, 0.34493446350097656, 0.6550655364990234],
[0.32971733808517456, 0.6702827215194702, 0.5224012732505798, 0.47759872674942017, 0.4692566692829132, 0.5307433605194092, 0.5360044836997986, 0.4639955163002014, 0.41811054944992065, 0.5818894505500793],
[0.37096619606018066, 0.6290338039398193, 0.5165190100669861, 0.4834809899330139, 0.4739859998226166, 0.526013970375061, 0.5340168476104736, 0.46598318219184875, 0.3438771069049835, 0.6561229228973389],
[0.4189890921115875, 0.5810109376907349, 0.52749103307724, 0.47250890731811523, 0.44485437870025635, 0.5551456212997437, 0.5398098230361938, 0.46019014716148376, 0.3739124536514282, 0.6260875463485718],
[0.3979812562465668, 0.6020187139511108, 0.5050275325775146, 0.49497246742248535, 0.4653399884700775, 0.5346599817276001, 0.537341833114624, 0.4626581072807312, 0.33742010593414307, 0.6625799536705017],
[0.368088960647583, 0.631911039352417, 0.49925288558006287, 0.5007471442222595, 0.4547160863876343, 0.545283854007721, 0.5408452749252319, 0.45915472507476807, 0.4053747355937958, 0.5946252346038818]]
# Create this before your for loop
df = pd.DataFrame(columns = range(10))
for pred_list in data:
#Add this within your for loop
df = df.append(pd.Series(pred_list), ignore_index=True)
output:

Failing while passing dataframe column seperated by comma into a API

I have to pass locations to API to retrieve values.
Working Code
dfs = []
locations = ['ZRH','SIN']
for loc in locations:
response = requests.get(f'https://risk.dev.tyche.eu-central-1.aws.int.kn/il/risk/location/{loc}', headers=headers, verify=False)
if 'items' in data:
df = pd.json_normalize(data, 'items', 'totalItems')
df1 = pd.concat([pd.DataFrame(x) for x in df.pop('relatedEntities')], keys=df.index).add_prefix('relatedEntities.')
df3 = df.join((df1).reset_index(level=1, drop=True))
dfs.append(df3)
df = pd.concat(dfs, ignore_index=True)
Failing Code ( while passing as parameter)
When I try to pass location as parameter which is created another dataframe column it fails.
Unique_Location = data['LOCATION'].unique()
Unique_Location = pd.DataFrame( list(zip(Unique_Location)), columns =['Unique_Location'])
t= ','.join(map(repr,Unique_Location['Unique_Location'] ))
locations = [t]
for loc in locations:
response = requests.get(f'https://risk.dev.logindex.com/il/risk/location/{loc}', headers=headers)
data = json.loads(response.text)
df = pd.json_normalize(data, 'items', 'totalItems')
What is wrong in my code?
Error
`c:\users\ashok.eapen\pycharmprojects\rs-components\venv\lib\site-packages\pandas\io\json\_normalize.py in _pull_records(js, spec)
246 if has non iterable value.
247 """
--> 248 result = _pull_field(js, spec)
249
250 # GH 31507 GH 30145, GH 26284 if result is not list, raise TypeError if not
c:\users\ashok.eapen\pycharmprojects\rs-components\venv\lib\site-packages\pandas\io\json\_normalize.py in _pull_field(js, spec)
237 result = result[field]
238 else:
--> 239 result = result[spec]
240 return result
241
KeyError: 'items'
`

You can test if items exist in json like:
dfs = []
locations = ['NZAKL', 'NZ23-USBCH', 'DEBAD', 'ARBUE', 'AR02_GSTI', 'AEJEA', 'UYMVD', 'UY03', 'AE01_GSTI', 'TH02_GSTI', 'JO01_GSTI', 'ITSIM', 'GB75_GSTI', 'DEAMA', 'DE273_GSTI', 'ITPRO', 'AT07_GSTI', 'FR05', 'FRHAU', 'FR01_GSTI', 'FRHER', 'ES70X-FRLBM', 'THNEO']
for loc in locations:
response = requests.get(f'https://risk.dev.logindex.com/il/risk/location/{loc}', headers=headers)
data = json.loads(response.text)
if 'items' in data:
if len(data['items']) > 0:
df = pd.json_normalize(data, 'items', 'totalItems')
#NaN in column, so failed - replace NaN to empty list
f = lambda x: x if isinstance(x, list) else []
df['raw.identifiers'] = df['raw.identifiers'].apply(f)
df['raw.relationships'] = df['raw.relationships'].apply(f)
df1 = pd.concat([pd.DataFrame(x) for x in df.pop('raw.identifiers')], keys=df.index).add_prefix('raw.identifiers.')
df2 = pd.concat([pd.DataFrame(x) for x in df.pop('raw.relationships')], keys=df.index).add_prefix('raw.relationships.')
df3 = df.join(df1.join(df2).reset_index(level=1, drop=True))
dfs.append(df3)
df = pd.concat(dfs, ignore_index=True)

Convert the results of a dictironary to a dataframe

From this commands
from stackapi import StackAPI
lst = ['11786778','12370060']
df = pd.DataFrame(lst)
SITE = StackAPI('stackoverflow', key="xxxx")
results = []
for i in range(1,len(df)):
SITE.max_pages=10000000
SITE.page_size=100
post = SITE.fetch('/users/{ids}/reputation-history', ids=lst[i])
results.append(post)
The results variable prints the results of the json format
How is it possible to converts the results variable to a dataframe with five columns?
reputation_history_type, reputation_change, post_id, creation_date,
user_id

Here try this :
from stackapi import StackAPI
import pandas as pd
lst = ['11786778','12370060']
SITE = StackAPI('stackoverflow')
results = []
SITE.max_pages=10000000
SITE.page_size=100
for i in lst:
post = SITE.fetch('/users/{ids}/reputation-history', ids=[i]).get('items')
results.extend([list(j.values()) for j in post])
df = pd.DataFrame(results, columns = ['reputation_history_type', 'reputation_change', 'post_id', 'creation_date', 'user_id'])
Output :
print(df.head()) gives :
reputation_history_type reputation_change post_id creation_date user_id
0 asker_accepts_answer 2 59126012 1575207944 11786778.0
1 post_undownvoted 2 59118819 1575139301 11786778.0
2 post_upvoted 10 59118819 1575139301 11786778.0
3 post_downvoted -2 59118819 1575139299 11786778.0
4 post_upvoted 10 59110166 1575094452 11786778.0
print(df.tail()) gives :
reputation_history_type reputation_change post_id creation_date user_id
170 post_upvoted 10 58906292 1574036540 12370060.0
171 answer_accepted 15 58896536 1573990105 12370060.0
172 post_upvoted 10 58896044 1573972834 12370060.0
173 post_downvoted 0 58896299 1573948372 12370060.0
174 post_downvoted 0 58896158 1573947435 12370060.0
NOTE :
You can just create a dataframe direct from the result which will be list of lists.
You don't need to declare SITE.max_page and SIZE.page_size every time you loop through the lst.

from stackapi import StackAPI
import pandas as pd
lst = ['11786778', '12370060']
df = pd.DataFrame(lst)
SITE = StackAPI('stackoverflow', key="xxxx")
results = []
for i in range(1, len(df)):
SITE.max_pages = 10000000
SITE.page_size = 100
post = SITE.fetch('/users/{ids}/reputation-history', ids=lst[i])
results.append(post)
data = []
for item in results:
data.append(item)
df = pd.DataFrame(data, columns=['reputation_history_type', 'reputation_change', 'post_id', 'creation_date', 'user_id']
print(df)

Kinda flying in the blind since I maxed out my StackOverflow API limit, but this should work:
from stackapi import StackAPI
from pandas.io.json import json_normalize
lst = ['11786778','12370060']
SITE = StackAPI('stackoverflow', key="xxx")
results = []
for ids in lst:
SITE.max_pages=10000000
SITE.page_size=100
post = SITE.fetch('/users/{ids}/reputation-history', ids=ids)
results.append(json_normalize(post, 'items'))
df = pd.concat(results, ignore_index=True)
json_normalize converts the JSON to dataframe
pd.concat concatenates the dataframes together to make a single frame

How to reshape data in Python?

I have a data set as given below-
Timestamp = 22-05-2019 08:40 :Light = 64.00 :Temp_Soil = 20.5625 :Temp_Air = 23.1875 :Soil_Moisture_1 = 756 :Soil_Moisture_2 = 780 :Soil_Moisture_3 = 1002
Timestamp = 22-05-2019 08:42 :Light = 64.00 :Temp_Soil = 20.5625 :Temp_Air = 23.125 :Soil_Moisture_1 = 755 :Soil_Moisture_2 = 782 :Soil_Moisture_3 = 1002
And I want to Reshape(rearrange) the dataset to orient header columns like [Timestamp, Light, Temp_Soil, Temp_Air, Soil_Moisture_1, Soil_Moisture_2, Soil_Moisture_3] and their values as the row entry in Python.

One of possible solutions:
Instead of a "true" input file, I used a string:
inp="""Timestamp = 22-05-2019 08:40 :Light = 64.00 :TempSoil = 20.5625 :TempAir = 23.1875 :SoilMoist1 = 756 :SoilMoist2 = 780 :SoilMoist3 = 1002
Timestamp = 22-05-2019 08:42 :Light = 64.00 :TempSoil = 20.5625 :TempAir = 23.125 :SoilMoist1 = 755 :SoilMoist2 = 782 :SoilMoist3 = 1002"""
buf = pd.compat.StringIO(inp)
To avoid "folding" of output lines, I shortened field names.
Then let's create the result DataFrame and a list of "rows" to append to it.
For now - both of them are empty.
df = pd.DataFrame(columns=['Timestamp', 'Light', 'TempSoil', 'TempAir',
'SoilMoist1', 'SoilMoist2', 'SoilMoist3'])
src = []
Below is a loop processing input rows:
while True:
line = buf.readline()
if not(line): # EOF
break
lst = re.split(r' :', line.rstrip()) # Field list
if len(lst) < 2: # Skip empty source lines
continue
dct = {} # Source "row" (dictionary)
for elem in lst: # Process fields
k, v = re.split(r' = ', elem)
dct[k] = v # Add field : value to "row"
src.append(dct)
And the last step is to append rows from src to df :
df = df.append(src, ignore_index =True, sort=False)
When you print(df), for my test data, you will get:
Timestamp Light TempSoil TempAir SoilMoist1 SoilMoist2 SoilMoist3
0 22-05-2019 08:40 64.00 20.5625 23.1875 756 780 1002
1 22-05-2019 08:42 64.00 20.5625 23.125 755 782 1002
For now all columns are of string type, so you can change the required
columns to either float or int:
df.Light = pd.to_numeric(df.Light)
df.TempSoil = pd.to_numeric(df.TempSoil)
df.TempAir = pd.to_numeric(df.TempAir)
df.SoilMoist1 = pd.to_numeric(df.SoilMoist1)
df.SoilMoist2 = pd.to_numeric(df.SoilMoist2)
df.SoilMoist3 = pd.to_numeric(df.SoilMoist3)
Note that to_numeric() function is clever enough to recognize the possible
type to convert to, so first 3 columns changed their type to float64
and the next 3 to int64.
You can check it executing df.info().
One more possible conversion is to change Timestamp column
to DateTime type:
df.Timestamp = pd.to_datetime(df.Timestamp)

Python , how to get value from a loop and assign to another loop?

I have two loops below the first one is the timand and the second one is shared. What i want to know how can i assign each result to shared["score"] ? cause what i had try below i assign shared["score"] = timang["score"] just return 1 1 1 .... And also how can we return multiple response in python for example
return Response(shared_data, tomon_dat, status=status.HTTP_200_OK) is this possible?
#result of timang
Result: 0
Result: 1
Result: 0
Result: 0
Result: 1
Result: 1
for timang in tomon_dat:
tm_ins = QuestionaireAnswerModel.objects.get(id=timang["id"])
timang["score"] = tm_ins.score
timang["id"] = tm_ins.id
datatest = timang["score"]
for shared in shared_data:
questionaire_ins = QuestionaireModel.objects.get(random_code=shared["random_code"])
shared["title"] = questionaire_ins.title
shared["sub_title"] = questionaire_ins.sub_title
shared["idddd"] = questionaire_ins.id
answer_ins = SharedQuestionaire.objects.get(id=shared["id"])
shared["is_answered"] = (QuestionaireAnswerModel.objects.filter(shared_questionaire=answer_ins).count()) > 0
shared["score"] = timang["score"]