I am a newbie to the world of Python and JSON though I've managed to work my way through most problems. The latest though is stumping me. I am trying to work with the API at localbitcoins.com and the JSON file is here LBC_JSON--it's a public file.
The output is quite large. I have tried working with it pandas using this code:
from pandas.io.json import json_normalize
from pandas.io.json import read_json
pandas_json = read_json('https://localbitcoins.com/buy-bitcoins-online/alipay/.json')
print(len(pandas_json))
print(type(pandas_json))
print(pandas_json)
But the completed data is not outputted, and then, not completely.
I have tried using the requests library and generating a response.json() on the response. Even though this brings in the complete data I cannot find a way to access the data that I need. I've tried iteration through the data with no luck. All I need is the first price in the API.
I have managed to get this info by using BeautifulSoup and CSS tags but I don't feel this is the correct way to access this info since an API is provided.
Thanks in advance for your answers.
You have to iterate over ad_list, for example:
for ad in pandas_json['data']['ad_list']:
print(ad['data']['profile']['username'], ad['data']['temp_price'])
Related
Wanted to see if anyone had any insight on this.
For a side project and for Python practice, I'm loading my fantasy football draft data into PyCharm. I'm doing this by using Postman to GET the league data from the Sleeper API as a JSON file, and then I am loading the JSON files into PyCharm as dictionaries and converting them to data frames using pd.DataFrame.from_dict for about 30 leagues. I'll then concat them all into a master dataframe for analysis.
Long story short; Is this the most effective way to do this? I'm very new to JSON and my Python is rusty as heck right now. I intend to use the data frames for general trend analysis. I'd appreciate any feedback.
Full disclosure, I work for the sports betting media company The Action Network. One of our API's returns some json so I will use that since you did not provide the postman request. Below is a simple example of how a list of json can be easily converted into a pandas dataframe.
import pandas as pd
import requests
r = requests.get('https://api.actionnetwork.com/web/v1/scoreboard/nfl')
j = r.json()
games = j['games']
df = pd.io.json.json_normalize(games)
I am creating a watcher for my Videos section in youtube which will keep monitoring the stats of latest video uploaded. I am not using Selenium because it will keep the browser engaged and interrupt. But using requests_html to load the /videos page and return me the stats like how many view are now and send me a message on Telegram app.
So when I did requests_html call i retrieve a small json which has all the videos & stats but the command json.load() actually coverts it into dict of dict. The intention is to use the result of json.load() as JSON so that I can perform json parsing.
Attached the snapshot of JSON structure and highlighted the desired keys
I couldnt find any better example that tell me how to convert it into just JSON instead of dict of dict. because there are multiple levels approx 20+ to retrieve the desired key value. I have seen very simple example that do the following, but it is not possible to write those many nodes in the below statement.
aKeyValue = dictName['parentnode']['childNode']
Secondly if it is just JSON then I believe jsonpath_ng and its parse methods can be used to retrieve a desired key with having to provide the complete path from root. If this is tnot the right way, please suggest any other Py Module.
the recursive functions didnt work properly. I tried may be 10+ different functions, none worked on the JSON. Finally I found jsonpath-ng to work after I read thru its entire documentation
It was pretty simple, I dont know why I didnt figure this out earlier.
json.loads(jsonString)
parse($..gridRenderer.items[0].gridVideoRenderer.viewCountText.simpleText)
But at the same time the statement below doesnt work. Although statement below wasnt required, but i just tried. It worked on jsonpath evaluator online but not in the python code.
$..gridRenderer.items[0].gridVideoRenderer.viewCountText[?(simpleText="5927 views")]
If someone could point out why the above statement wouldnt work in python code ?
I make a call to an API and get a JSON response. For the better half of 2 days i have been trying to access the data, and send it to a CSV file, or push it to Pandas with no luck. I have gone through page after page of assistance, but something is just not clicking.
Below is the JSON data, all i need are the following elements singleLineAddress, isAgentsAdvice & contractDate
[
{
"_embedded":{
"propertySummaryList":[
{
"propertySummary":{
"address":{
"singleLineAddress":"Example St"
},
"id":47328284,
"lastSaleDetail":{
"contractDate":"Example",
"isAgentsAdvice":Yes,
"isArmsLength":0,
"isPriceWithheld":0,
The ideal end result would be a csv export that had the following column names
Address | AA | Contract_Date
So far i've tried to create a simple dataframe in python, however it seems to always print on singular lines, so i moved over to attempting to 'flatten' the file for easier processing, but no data was extracted at all in those attempts.
I attempted to follow a few github recipes, but the overall theme seemed to be just flattening the data, which as above didnt seem to work well with my json file.
I attempted the normal dataframe route but didnt have much luck, my main focus has been in both json_normalize, and flatten_json type work, but if someone wants to offer a better solution, i am all ears.
Any help is appreciated. I'd appreciate any leads on what may be 'missing' from my knowledge base around JSON and Pandas as it's been quite handy the last few weeks.
Cheers
I just wanted to ask what can I do to solve this issue I have.
Essentially I am making a stock checker for sneakers from Adidas, I know the endpoint to obtain the stock but the JSON data given back to me whilst readable and contains what I need also contains a bunch of other information that is unnecessary to what I am trying to do.
Example of a link to an endpoint:
http://production.store.adidasgroup.demandware.net/s/adidas-GB/dw/shop/v16_9/products/(BZ0221)?client_id=c1f3632f-6d3a-43f4-9987-9de920731dcb&expand=availability,variations,prices
This is a link to the JSON containing the stock of the shoe, price and availability. However, if you try to open it you'll see that it responds a bunch of useless info such as the description of the shoe and the price which I do not need.
A github repository that I was using to try and get to grips with the requests I am trying to make is:
https://github.com/yzyio/adidas-stock-checker/blob/master/assets/index.js
I can get it to give me the JSON response I am just trying to strip what I don't need and keep what I do need which I am finding very difficult especially in python.
Many Thanks!
Since you've said you can get a JSON response from the server than the first think you need to do is tell python to load it as JSON.
import json
data = json.loads(response_from_server)
After doing this you can now access the values in your JSON object the way you would access them via a Python dict.
data["artist"]["id"]
So I was reading this: Reading JSON from a file?. So the person in this question was having trouble loading the JSON file that they presumably made. But I wanted to know if it was possible to load a JSON URL such as this: http://data.consumerfinance.gov/api/views.json. If it is possible, how would you do it? Would it be as simple as json.load(http://data.consumerfinance.gov/api/views.json)? If it isn't possible what would you recommend me to read this same JSON file?
PS. Someone tell how to earn reputation so I make a comment instead of having to ask a separate question.
import requests
requests.get('http://data.consumerfinance.gov/api/views.json').json()
You'll need to have the requests library installed.