Develop Reference

Error in loading json with os and pandas package [duplicate]

I have some difficulty in importing a JSON file with pandas.
import pandas as pd
map_index_to_word = pd.read_json('people_wiki_map_index_to_word.json')
This is the error that I get:
ValueError: If using all scalar values, you must pass an index
The file structure is simplified like this:
{"biennials": 522004, "lb915": 116290, "shatzky": 127647, "woode": 174106, "damfunk": 133206, "nualart": 153444, "hatefillot": 164111, "missionborn": 261765, "yeardescribed": 161075, "theoryhe": 521685}
It is from the machine learning course of University of Washington on Coursera. You can find the file here.

Try
ser = pd.read_json('people_wiki_map_index_to_word.json', typ='series')
That file only contains key value pairs where values are scalars. You can convert it to a dataframe with ser.to_frame('count').
You can also do something like this:
import json
with open('people_wiki_map_index_to_word.json', 'r') as f:
data = json.load(f)
Now data is a dictionary. You can pass it to a dataframe constructor like this:
df = pd.DataFrame({'count': data})

You can do as #ayhan mention which will give you a column base format
Or you can enclose the object in [ ] (source) as shown below to give you a row format that will be convenient if you are loading multiple values and planing on using matrix for your machine learning models.
df = pd.DataFrame([data])

I think what is happening is that the data in
map_index_to_word = pd.read_json('people_wiki_map_index_to_word.json')
is being read as a string instead of a json
{"biennials": 522004, "lb915": 116290, "shatzky": 127647, "woode": 174106, "damfunk": 133206, "nualart": 153444, "hatefillot": 164111, "missionborn": 261765, "yeardescribed": 161075, "theoryhe": 521685}
is actually
'{"biennials": 522004, "lb915": 116290, "shatzky": 127647, "woode": 174106, "damfunk": 133206, "nualart": 153444, "hatefillot": 164111, "missionborn": 261765, "yeardescribed": 161075, "theoryhe": 521685}'
Since a string is a scalar, it wants you to load it as a json, you have to convert it to a dict which is exactly what the other response is doing
The best way is to do a json loads on the string to convert it to a dict and load it into pandas
myfile=f.read()
jsonData=json.loads(myfile)
df=pd.DataFrame(data)

{
"biennials": 522004,
"lb915": 116290
}
df = pd.read_json('values.json')
As pd.read_json expects a list
{
"biennials": [522004],
"lb915": [116290]
}
for a particular key, it returns an error saying
If using all scalar values, you must pass an index.
So you can resolve this by specifying 'typ' arg in pd.read_json
map_index_to_word = pd.read_json('Datasets/people_wiki_map_index_to_word.json', typ='dictionary')

For newer pandas, 0.19.0 and later, use the lines parameter, set it to True.
The file is read as a json object per line.
import pandas as pd
map_index_to_word = pd.read_json('people_wiki_map_index_to_word.json', lines=True)
If fixed the following errors I encountered especially when some of the json files have only one value:
ValueError: If using all scalar values, you must pass an index
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ValueError: Trailing data

For example
cat values.json
{
name: "Snow",
age: "31"
}
df = pd.read_json('values.json')
Chances are you might end up with this
Error: if using all scalar values, you must pass an index
Pandas looks up for a list or dictionary in the value. Something like
cat values.json
{
name: ["Snow"],
age: ["31"]
}
So try doing this. Later on to convert to html tohtml()
df = pd.DataFrame([pd.read_json(report_file, typ='series')])
result = df.to_html()

I solved this by converting it into an array like so
[{"biennials": 522004, "lb915": 116290, "shatzky": 127647, "woode": 174106, "damfunk": 133206, "nualart": 153444, "hatefillot": 164111, "missionborn": 261765, "yeardescribed": 161075, "theoryhe": 521685}]

How can I convert a CSV file into a list inside a dictionary?

I want to convert a CSV file into a python Dict, this is the format I want to achieve.
1,apple
1,orange
1,pearl
1,ocean
2,apple
2,pearl
3,house
4,market
4,asia
I would like to create something like this.
{
"1":["apple","orange","pearl","ocean"],
"2":["apple","pearl"],
"3":["house"],
"4":["market","asia"]
}
I am trying to do this in Python, could someone please help me?

group by first column, and collect all results in a list
import pandas, io
df = pandas.read_csv(io.StringIO("""1,apple
1,orage
1,pearl
1,ocean
2,apple
2,pearl
3,house
4,market
4,asia"""), names=["1", "2"], dtype=str)
df = df.groupby("1")["2"].apply(list).to_dict()
df

How to work with multiple of the same key?

I have a large dictionary that contains weather data. You can take a look at it here
This weather data is for multiple days, and I want to get all of the values from one key. How would I do this?
Here is a simplified version of the dictionary:
'data': { 'day1' : {'weather_discription': 'cloudy'},
'day2' : {'weather_discription': 'clear'}
}
I tried to use this code:
import requests
r = requests.get('data website')
res = r.json()
print(res['weather_discription'])

You need a loop to get them all.
for day, data in res['data'].items():
print(f"Weather on {day} was {data['weather_description']}")

Iterating deeply nested pandas json object?

I have a pretty big json object which is of the format
[
{
"A":"value",
"TIME":1551052800000,
"C":35,
"D":36,
"E":34,
"F":35,
"G":33
},
{
"B":"value",
"TIME":1551052800000,
"C":36,
"D":56,
"E":44,
"F":75,
"G":38
}, ...
...
]
Converted to json with the help of pandas
df.to_json(orient='records')
I want to loop through the json body and update a specific key inside this json object and send it back to the client through my api
I want to do something like
for i = 0
object[i]["TIME"] = updateCaclulations
return i
I am new to python and have tried this. It helps iterate through the object but updation is not there and the time taken due to recursion is a lot.

First, pd.read_sql_query returns pd.DataFrame and not json.
As per your question:
Say you have a sample function calculate:
def update_calculation(time):
return time
You could update time so:
df["TIME"] = df["TIME"].apply(update_calculation)

Nested JSON file into Pandas Dataframe

I'm having trouble getting this nested JSON object into a pandas dataframe using python:
{
"count":275,
"calls":[
{
"connectedTo":"18885068980",
"serviceName":"",
"callGuid":"01541af0-d87c-4911-a868-f5ac573d1e31",
"origin":"+19178558701",
"stateChangedAt":"2016-04-15T18:21:23Z",
"sequence":9,
"appletName":"ACD Sales General"
}
]
}
I've tried using json_normalize and am going in circles. Any help would be very much appreciated!

I know that it includes json_normalize, but I think this is what you are trying to do.
import json
import pandas as pd
from pandas.io.json import json_normalize
from pprint import pprint
j = json.dumps( //to create the json
{'count': 275,
"calls":
[{'connectedTo': "18885068980",
"serviceName":"",
"callGuid":"01541af0-d87c-4911-a868-f5ac573d1e31",
"stateChangedAt":"2016-04-15T18:21:23Z",
"sequence":9,
"appletName":"ACD Sales General"}]})
data = json.loads(j)
pprint(json_normalize(data['calls']))
which returns
appletName callGuid connectedTo \
0 ACD Sales General 01541af0-d87c-4911-a868-f5ac573d1e31 18885068980
sequence serviceName stateChangedAt
0 9 2016-04-15T18:21:23Z