I like to periodically import json data into a Django. What's the best way to import the data below to a Django model.
As far as I've seen is the best approach to use Django fixtures. Since my Model looks like this and I can't change the given JSON, what's the best way to import the data or restructure the JSON to use it as a fixture.
Django Model
class Item(models.Model):
item_name = models.CharField(max_length=250)
timestamp = models.DateTimeField()
value01 = models.DecimalField(max_digits=7, decimal_places=2)
value02 = models.DecimalField(max_digits=7, decimal_places=2)
Data I like to store:
Item1 | 2017-04-18 09:24:46 | 15.00 | 12.68
Item1 | 2017-04-18 09:25:44 | 14.92 | 12.42
Given JSON:
[
{
"id": 1,
"timestamp": "2017-04-18 09:24:46",
"details": {
"id": 843,
"color": "white"
},
"item": {
"id": 1691,
"category": "1",
"item_type": {
"id": 14,
"name": "Item1"
}
},
"itemdatavalues": [
{
"id": 227,
"value": "15.00"
},
{
"id": 228,
"value": "12.68"
}
]
},
{
"id": 2,
"timestamp": "2017-04-18 09:25:44",
"details": {
"id": 843,
"color": "white"
},
"item": {
"id": 161,
"category": "1",
"item_type": {
"id": 14,
"name": "Item1"
}
},
"itemdatavalues": [
{
"id": 283,
"value": "14.92"
},
{
"id": 284,
"value": "12.42"
}
]
}
]
I wrote a simple script to parse the input file and generate a fixture:
import json
import copy
with open('/tmp/data.json') as dataf, open('/tmp/output.json', 'w') as out:
data = json.load(dataf)
newdata = []
for i, block in enumerate(data):
new = dict(model="appname.Item", pk=block['id'])
new['fields'] = dict(item_name=block['item']['item_type']['name'],
timestamp=block['timestamp'],
value01=block['itemdatavalues'][0]['value'],
value02=block['itemdatavalues'][1]['value'])
newdata.append(copy.deepcopy(new))
json.dump(newdata, out, indent=2)
Using your input, this results in:
[
{
"pk": 1,
"model": "appname.Item",
"fields": {
"item_name": "Item1",
"timestamp": "2017-04-18 09:24:46",
"value01": "15.00",
"value02": "12.68"
}
},
{
"pk": 2,
"model": "appname.Item",
"fields": {
"item_name": "Item1",
"timestamp": "2017-04-18 09:25:44",
"value01": "14.92",
"value02": "12.42"
}
}
]
Which is suitable for use as a fixture. Obviously, you will have to fix the appname.
This might help: https://docs.djangoproject.com/en/1.11/ref/models/instances/. You should be able to create a new model object in whatever methods you are receiving that json in, then parse whatever json values you want into the object. There are some pretty easy json libraries for python. That is, unless you are wanting to handle something differently, or I am reading the documentation wrong. Have a good day!
Related
I just learnt django and I am getting data from api and looping through the json and appending the data into the list. but When I use .map() function in react then the data is appending in list (from for loop) like
[
{
"results": {
"id": 544,
"name": "User_1",
}
},
{
"results": {
"id": 218,
"name": "User_2",
}
},
{
"results": {
"id": 8948,
"name": "User_3",
}
},
{
"results": {
"id": 9,
"name": "User_4",
}
},
]
It is not appending like (Like I want)
[
results : [
{
"id": 544,
"name": "User_1"
},
{
"id": 218,
"name": "User_2"
},
{
"id": 8948,
"name": "User_3"
},
{
"id": 9,
"name": "User_4"
}
],
"length_of_results": 25,
]
views.py
def extract_view(request):
results_list = []
// api url for explanation only
get_response = "https://api.punkapi.com/v2/beers"
if get_response.status_code == 200:
for result in get_response.json():
results_list.append({"results": result})
results_list.append({"length_of_results": len(results_list)})
return Response({"data": results_list})
I know, In for loop it is appending every result within it with every iteration but I also want to assign all the responses within results list. Because I will add a append another field after for loop.
I have tried many times but it is still not working.
You can solve it by map function iterating over list:
dict(results=list(map(lambda x: x["results"], response)))
Full working example:
response = [
{
"results": {
"id": 544,
"name": "User_1",
}
},
{
"results": {
"id": 218,
"name": "User_2",
}
},
{
"results": {
"id": 8948,
"name": "User_3",
}
},
{
"results": {
"id": 9,
"name": "User_4",
}
},
]
results = dict(results=list(map(lambda x: x["results"], response)))
results["length_of_results"] = len(results["results"])
>> {'results': [{'id': 544, 'name': 'User_1'},
>> {'id': 218, 'name': 'User_2'},
>> {'id': 8948, 'name': 'User_3'},
>> {'id': 9, 'name': 'User_4'}],
>> 'length_of_results': 4}
By doing
results_list.append({"results": result})
you are creating a new dictionary with the value being result which I believe is a dictionary itself. So you should be able to just do this:
if get_response.status_code == 200:
for result in get_response.json():
results_list.append(result)
This is the first time I'm working with JSON, and I'm trying to pull url out of the JSON below.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer,
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"
}
I have been able to access description and _id via
data = json.loads(line)
if 'xpath' in data:
xpath = data["_id"]
description = data["sections"][0]["payload"][0]["description"]
However, I can't seem to figure out a way to access url. One other issue I have is there could be other items in sections, which makes indexing into Contact Info a non starter.
Hope this helps:
import json
with open("test.json", "r") as f:
json_out = json.load(f)
for i in json_out["sections"]:
for j in i["payload"]:
for key in j:
if "url" in key:
print(key, '->', j[key])
I think your JSON is damaged, it should be like that.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer",
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"
}
You can check it on http://json.parser.online.fr/.
And if you want to get the value of the url.
import json
j = json.load(open('yourJSONfile.json'))
print(j['sections'][1]['payload'][0]['url'])
I think it's worth to write a short function to get the url(s) and make a decision whether or not to use the first found url in the returned list, or skip processing if there's no url available in your data.
The method shall looks like this:
def extract_urls(data):
payloads = []
for section in data['sections']:
payloads += section.get('payload') or []
urls = [x['url'] for x in payloads if 'url' in x]
return urls
This should print out the URL
import json
# open json file to read
with open('test.json','r') as f:
# load json, parameter as json text (file contents)
data = json.loads(f.read())
# after observing format of JSON data, the location of the URL key
# is determined and the data variable is manipulated to extract the value
print(data['sections'][1]['payload'][0]['url'])
The exact location of the 'url' key:
1st (position) of the array which is the value of the key 'sections'
Inside the array value, there is a dict, and the key 'payload' contains an array
In the 0th (position) of the array is a dict with a key 'url'
While testing my solution, I noticed that the json provided is flawed, after fixing the json flaws(3), I ended up with this.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer",
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"}
After utilizing the JSON that was provided by Vincent55.
I made a working code with exception handling and with certain assumptions.
Working Code:
## Assuming that the target data is always under sections[i].payload
from json import loads
line = open("data.json").read()
data = loads(line)["sections"]
for x in data:
try:
# With assumption that there is only one payload
if x["payload"][0]["url"]:
print(x["payload"][0]["url"])
except KeyError:
pass
I am writing some code with Python and Spotipy and I'm relatively new to coding. I have some code that get all the info about a Spotify playlist and prints it out for me:
from spotipy.oauth2 import SpotifyClientCredentials
import spotipy
import json
client_credentials_manager = SpotifyClientCredentials()
sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager)
playlist_id = 'spotify:playlist:76CVeJDw2b90up5PgkZXyU'
results = sp.playlist(playlist_id)
#print(json.dumps(results, indent=4))
print((json.dumps(results, indent=4)))
It works well and gives me all the info. My problem is that I only need specifics from the print:
"collaborative": false,
"description": "",
"external_urls": {
"spotify": "https://open.spotify.com/playlist/76CVeJDw2b90up5PgkZXyU"
},
"followers": {
"href": null,
"total": 0
},
"href": "https://api.spotify.com/v1/playlists/76CVeJDw2b90up5PgkZXyU?additional_types=track",
"id": "76CVeJDw2b90up5PgkZXyU",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/ab67616d0000b2734a052b99c042dc15f933145b",
"width": 640
}
],
"name": "TEST",
"owner": {
"display_name": "Name",
"external_urls": {
"spotify": "https://open.spotify.com/user/myname"
},
"href": "https://api.spotify.com/v1/users/myname",
"id": "Myname",
"type": "user",
"uri": "spotify:user:kovizsombor"
},
"primary_color": null,
"public": true,
"snapshot_id": "MixmMGE0MDgxNDQ1ZGVlNmE4MThiMmQwODMwYWU0OTI3YzkyOGJhOWIz",
"tracks": {
"href": "https://api.spotify.com/v1/playlists/76CVeJDw2b90up5PgkZXyU/tracks?offset=0&limit=100&additional_types=track",
"items": [
{
"added_at": "2020-05-17T10:00:11Z",
"added_by": {
"external_urls": {
"spotify": "https://open.spotify.com/user/kovizsombor"
},
"href": "https://api.spotify.com/v1/users/kovizsombor",
"id": "kovizsombor",
"type": "user",
"uri": "spotify:user:kovizsombor"
},
"is_local": false,
"primary_color": null,
"track": {
"album": {
"album_type": "album",
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/0PFtn5NtBbbUNbU9EAmIWF"
},
"href": "https://api.spotify.com/v1/artists/0PFtn5NtBbbUNbU9EAmIWF",
"id": "0PFtn5NtBbbUNbU9EAmIWF",
"name": "TOTO",
"type": "artist",
"uri": "spotify:artist:0PFtn5NtBbbUNbU9EAmIWF"
}
],
],
"external_urls": {
"spotify": "https://open.spotify.com/album/62U7xIHcID94o20Of5ea4D"
},
"href": "https://api.spotify.com/v1/albums/62U7xIHcID94o20Of5ea4D",
"id": "62U7xIHcID94o20Of5ea4D",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/ab67616d0000b2734a052b99c042dc15f933145b",
"width": 640
},
{
"height": 300,
"url": "https://i.scdn.co/image/ab67616d00001e024a052b99c042dc15f933145b",
"width": 300
},
{
"height": 64,
"url": "https://i.scdn.co/image/ab67616d000048514a052b99c042dc15f933145b",
"width": 64
}
],
"name": "Toto IV",
"release_date": "1982-04-08",
"release_date_precision": "day",
"total_tracks": 10,
"type": "album",
"uri": "spotify:album:62U7xIHcID94o20Of5ea4D"
},
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/0PFtn5NtBbbUNbU9EAmIWF"
},
"href": "https://api.spotify.com/v1/artists/0PFtn5NtBbbUNbU9EAmIWF",
"id": "0PFtn5NtBbbUNbU9EAmIWF",
"name": "TOTO",
"type": "artist",
"uri": "spotify:artist:0PFtn5NtBbbUNbU9EAmIWF"
}
],
"available_markets": [
],
"disc_number": 1,
"duration_ms": 295893,
"episode": false,
"explicit": false,
"external_ids": {
"isrc": "USSM19801941"
},
"external_urls": {
"spotify": "https://open.spotify.com/track/2374M0fQpWi3dLnB54qaLX"
},
"href": "https://api.spotify.com/v1/tracks/2374M0fQpWi3dLnB54qaLX",
"id": "2374M0fQpWi3dLnB54qaLX",
"is_local": false,
"name": "Africa",
"popularity": 83,
"preview_url": "https://p.scdn.co/mp3-preview/dd78dafe31bb98f230372c038a126b8808f9349b?cid=d568e7073a38465bba48268ea9f10153",
"track": true,
"track_number": 10,
"type": "track",
"uri": "spotify:track:2374M0fQpWi3dLnB54qaLX"
},
"video_thumbnail": {
"url": null
}
}
],
"limit": 100,
"next": null,
"offset": 0,
"previous": null,
"total": 1
},
"type": "playlist",
"uri": "spotify:playlist:76CVeJDw2b90up5PgkZXyU"
}
From this long print I somehow need to extract the Artist and the song title and preferably make it into a variable. Also not sure how this would work if there are multiple songs in a playlist.
It's also a solution if I can print out only the Artist and the title of the song without printing out all the information.
Based on your posted example it appears that you have only 1 song named "Africa" by artist "TOTO". Copying the track to have two songs, I added another track with two artists for testing the arrays.
If that json is loaded into a variable named results then (as #xcmkz said) you have a python dictionary and can process accordingly.
Try working with the following to transverse through your dict appending artists and songs to lists:
song_dict = {}
for track in results['tracks']['items']:
song_name = track["track"]["name"]
a2 = []
for t1 in track['track']['artists']:
a2.append(t1['name'])
song_dict.update({song_name: a2})
print(f'Dictionary of Songs and Artists:')
for k, v in song_dict.items():
print(f'Song --> {k}, by --> {", ".join(v)}')
Results:
Dictionary of Songs and Artists:
Song --> Africa, by --> TOTO
Song --> Just Another Silly Song, by --> Artist 2, Artist 3
sp.playlist returns a dictionary, so you can simply access its values by their keys. For example:
>>> results['name']
'TEST'
JSON is a data serialization format, ie a standardized way of representing objects as pure text and parsing them back from the text. json.dumps therefore converts the dictionary object to a string of text. This is useful if you want to for example save the results to a file and load it back later. You don't need it to access contents from results.
(This is a playlist though—you will need to get information on each song/track to get its artist and name.)
I have some data from a project where the variables can change from motorcycle and car. I need to get the name out of them and that value is inside the variable.
This is not the data i will be using but it has the same structure, the "official" data is some persional information so i changed it to some random values. I can not change the structure of the JSON data since this is the way the serveradmins decided to structure it for some reason.
This is my python code:
import json
with open('exampleData.json') as j:
data = json.load(j)
name = 0
Vehicle = 0
for x in data:
print(data['persons'][x]['name'])
for i in data['persons'][x]['things']["Vehicles"]:
print(data['persons'][x]['things']['Vehicles'][i]['type']['name'])
print("\n")
This is my Json data i extracted from the file "ExampleData.json"(sorry for long but it is kinda complex and necessary to understand the problem):
{
"total": 2,
"persons": [
{
"name": "Sven Svensson",
"things": {
"House": "apartment",
"Vehicles": [
{
"id": "46",
"type": {
"name": "Kawasaki ER6N",
"type": "motorcyle"
},
"Motorcycle": {
"plate": "aaa111",
"fields": {
"brand": "Kawasaki",
"status": "in shop"
}
}
},
{
"id": "44",
"type": {
"name": "BMW m3",
"type": "Car"
},
"Car": {
"plate": "bbb222",
"fields": {
"brand": "BMW",
"status": "in garage"
}
}
}
]
}
},
{
"name": "Eric Vivian Matthews",
"things": {
"House": "House",
"Vehicles": [
{
"id": "44",
"type": {
"name": "Volvo XC90",
"type": "Car"
},
"Car": {
"plate": "bbb222",
"fields": {
"brand": "Volvo",
"status": "in garage"
}
}
}
]
}
}
]
}
I want it to print out something like this :
Sven Svensson
Bmw M3
Kawasaki ER6n
Eric Vivian Matthews
Volvo XC90
but i get this error:
print(data['persons'][x]['name'])
TypeError: list indices must be integers or slices, not str
Process finished with exit code 1
What you need is
for person in data["persons"]:
for vehicle in person["things"]["vehicles"]:
print(vehicle["type"]["name"])
type = vehicle["type"]["type"]
print(vehicle[type]["plate"])
Python for loop does not return the key but rather an object here:
for x in data:
Referencing an object as key
print(data['persons'][x]['name'])
Is causing the error
What you need is to use the returning json object and iterate over them like so:
for x in data['persons']:
print(x['name'])
for vehicle in x['things']['Vehicles']:
print(vehicle['type']['name'])
print('\n')
I have the following JSON data and would like to extract email value using Python:
{
"_links": {
"self": {
"href": "https://example.com/comments/9"
}
},
"_embedded": {
"customer": {
"name": "Jamie XXXX",
"email": "jamie#example.tv",
"thumbnail": {
"small": "https://secure.gravatar.com/avatar/dfd.png?d=blank&r=PG&s=100",
"medium": "https://secure.gravatar.com/avatar/dfd.png?d=blank&r=PG&s=200",
"large": "https://secure.gravatar.com/avatar/dfdfd.png?d=blank&r=PG&s=300"
}
},
"comments": []
},
"id": 9,
"video_id": null,
"content": "j/k I meant that as a reply",
"comments_count": 0,
"created_at": "2014-03-12T17:46:07Z",
"updated_at": "2014-03-12T17:46:07Z"
}
I tried something like but it's not working:
jsonresp = r.json()
for k, v in jsonresp:
print(jsonresp['_embedded']['customer']['email'])
jsonresp = r.json()
print(jsonresp['_embedded']['customer']['email'])
Using pyjq:
import json
import pyjq
with open("input.json", "r") as myfile:
data=json.load(myfile)
print pyjq.first('._embedded.customer.email', data);