Develop Reference

How do I merge and sort JSON objects using its counts?

I got two json objects that I need to combine together based on ID and do count and sort operations on it.
Here is the first object comments:
[
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
},
{
"userId": 1,
"id": 2,
"title": "qui est esse",
"body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla"
},
{
"userId": 1,
"id": 3,
"title": "ea molestias quasi exercitationem repellat qui ipsa sit aut",
"body": "et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut"
},
{
"userId": 1,
"id": 4,
"title": "eum et est occaecati",
"body": "ullam et saepe reiciendis voluptatem adipisci\nsit amet autem assumenda provident rerum culpa\nquis hic commodi nesciunt rem tenetur doloremque ipsam iure\nquis sunt voluptatem rerum illo velit"
},
]
This is second json object:
[
{
"postId": 1,
"id": 1,
"name": "id labore ex et quam laborum",
"email": "Eliseo#gardner.biz",
"body": "laudantium enim quasi est quidem magnam voluptate ipsam eos\ntempora quo necessitatibus\ndolor quam autem quasi\nreiciendis et nam sapiente accusantium"
},
{
"postId": 1,
"id": 2,
"name": "quo vero reiciendis velit similique earum",
"email": "Jayne_Kuhic#sydney.com",
"body": "est natus enim nihil est dolore omnis voluptatem numquam\net omnis occaecati quod ullam at\nvoluptatem error expedita pariatur\nnihil sint nostrum voluptatem reiciendis et"
},
{
"postId": 1,
"id": 3,
"name": "odio adipisci rerum aut animi",
"email": "Nikita#garfield.biz",
"body": "quia molestiae reprehenderit quasi aspernatur\naut expedita occaecati aliquam eveniet laudantium\nomnis quibusdam delectus saepe quia accusamus maiores nam est\ncum et ducimus et vero voluptates excepturi deleniti ratione"
},
{
"postId": 1,
"id": 4,
"name": "alias odio sit",
"email": "Lew#alysha.tv",
"body": "non et atque\noccaecati deserunt quas accusantium unde odit nobis qui voluptatem\nquia voluptas consequuntur itaque dolor\net qui rerum deleniti ut occaecati"
},
{
"postId": 2,
"id": 5,
"name": "et fugit eligendi deleniti quidem qui sint nihil autem",
"email": "Presley.Mueller#myrl.com",
"body": "doloribus at sed quis culpa deserunt consectetur qui praesentium\naccusamus fugiat dicta\nvoluptatem rerum ut voluptate autem\nvoluptatem repellendus aspernatur dolorem in"
},
{
"postId": 2,
"id": 6,
"name": "repellat consequatur praesentium vel minus molestias voluptatum",
"email": "Dallas#ole.me",
"body": "maiores sed dolores similique labore et inventore et\nquasi temporibus esse sunt id et\neos voluptatem aliquam\naliquid ratione corporis molestiae mollitia quia et magnam dolor"
},
]
Object one is basically posts with poster details and object two is comments with commenter details.
So expected that object one has one to many relationships with second object. For example one post has many comments. This relationship is based on id in object one is postId in object two. The ultimate objective is to count and sort post by number of comments.
I attempt the problem with simple for loops and creating new json object, I managed to combine them together, but I dont know how to count and sort them properly.
in the views:
for i in posts:
if (id==postId):
newobj.append(objtwo[i])
count+=1
else:
newobj.append(count)
count=0
Normally I use django ORM to sort this but I dont have access to the database and model of the table. How to count and sort the new object so it can return list of posts with most comments counts and descend to lower comments counts?

Assuming your posts and comments data structures are lists, you can use python's defaultdict to count the comments. Then, use posts.sort(key=...) to sort your posts based on the collected counts using the key parameter. Altogether, it could like like this:
import json
from collections import defaultdict
posts = [ ... ]
comments = [ ... ]
# data structure to count the to comments
# automatically initializes to 0
comments_per_post = defaultdict(int)
# iterate through the comments to increase the count for the posts
for comment in comments:
comments_per_post[comment['postId']] += 1
# add comment count to post
for post in posts:
post['number_of_comments'] = comments_per_post[post['id']]
# sort the posts based on the counts collected
posts.sort(key=lambda post: post['number_of_comments'], reverse=True)
# print them to verify
# number of comments per Post will be in the `number_of_comments` key on the post dict.
print(json.dumps(posts, indent=2))
Note: this sorts the posts array in-place. If you don't want this, you can use sorted_posts = sorted(posts, key=... instead.

My answer is very similar to Byted's answer.
I would use Counter from the built-in collections to count the number of postIds in the second object.
Then sort the first object by using these counts from the previous step as a sorting key. Counter object returns 0 if a key is not present in it, so just use it as a lookup as a sorting key. The negative sign ensures a descending order (because sorted() sorts in ascending order by default).
import json
from collections import Counter
# count the comments
counts = Counter([d['postId'] for d in objtwo])
# add the counts to each post
for d in objone:
d["number of comments"] = counts[d['id']]
# sort posts by number of comments in descending order
objone.sort(key=lambda x: -x['number of comments'])
# convert to json
json.dumps(objone, indent=4)
Intermediate output for this input:
print(counts)
# Counter({1: 4, 2: 2})

Convert nested JSON to HTML Unordered List with Python

I did a fair amount of google searching but didn't find a suitable answer for someone who would like take arbitrarily nested JSON, for example from an API response, and display it as an unordered list using HTML using python.
Ultimately, the issue was solved with a fairly short recursive function.
Here's an example of the type of input I was dealing with:
{
"_id": "61dc4e9130473a8465a11cd0",
"index": 0,
"guid": "a2a7e550-8bf4-4be5-b0e0-2b124a2ca7e1",
"isActive": false,
"balance": "$1,011.28",
"picture": "http://placehold.it/32x32",
"age": 25,
"eyeColor": "green",
"name": "Monique Dickerson",
"gender": "female",
"company": "AQUASURE",
"about": "Dolore dolor excepteur tempor excepteur nulla occaecat Lorem dolor cillum sint velit. Minim labore irure ea anim duis in enim laboris. Aute amet ut sunt ea. Do irure sint commodo ea id. Amet dolore culpa anim irure ipsum est labore nostrud irure.\r\n",
"registered": "2015-12-15T11:10:14 +05:00",
"latitude": -63.913924,
"longitude": -21.554531,
"tags": [
"eiusmod",
"dolore",
"pariatur",
"in",
"ipsum",
"Lorem",
"adipisicing"
]
}
Here was the desired output:
_id: 61dc4e9130473a8465a11cd0
index: 0
guid: a2a7e550-8bf4-4be5-b0e0-2b124a2ca7e1
isActive: False
balance: $1,011.28
picture: http://placehold.it/32x32
age: 25
eyeColor: green
name: Monique Dickerson
gender: female
company: AQUASURE
about: Dolore dolor excepteur tempor excepteur nulla occaecat Lorem dolor cillum sint velit. Minim labore irure ea anim duis in enim laboris. Aute amet ut sunt ea. Do irure sint commodo ea id. Amet dolore culpa anim irure ipsum est labore nostrud irure.\r\n
registered: 2015-12-15T11:10:14 +05:00
latitude: -63.913924
longitude: -21.554531
tags:
eiusmod
dolore
pariatur
in
ipsum
Lorem
adipisicing

I was able to solve the problem successfully with this bit of code:
items = []
def render(json_data, v=""):
if isinstance(json_data, dict):
items.append(f"<ul>")
for k2, v2 in json_data.items():
render(k2 ,v2) # <---If we have a dict, apply function again
items.append(f"</ul>")
elif isinstance(v, dict):
items.append(f"<li>{json_data}: <ul>")
for k2, v2 in v.items():
render(k2 ,v2) # <---If we have a dict, apply function again
items.append("</ul></li>")
elif isinstance(v, list):
items.append(f"<li>{json_data}:<ul>")
for i in v:
if isinstance(i, str):
items.append(f"<li>{i}</li>")
elif isinstance(i, dict):
render(i, v)
items.append("</ul></li>")
else:
items.append(f"<li>{json_data}: {v}</li>")
render(data)
html_str = "".join(items)
html_str

Django Rest Framework two Serializers for the same Model

I'm pretty sure there's a better way to do this:
class PostSerializer(serializers.ModelSerializer):
class Meta:
model = Post
fields = ('category', 'id', 'title', 'image', 'slug', 'author', 'excerpt', 'content', 'status', 'published')
class FrontendPostSerializer(serializers.ModelSerializer):
author = AuthorSerializer(many=False, read_only=True)
category = CategorySerializer(many=False, read_only=True)
class Meta:
model = Post
fields = ('category', 'id', 'title', 'image', 'slug', 'author', 'excerpt', 'content', 'status', 'published')
PostSerializer is gonna look like this
{
"category": 1,
"id": 45,
"title": "Lorem Ipsum - Lorem ipsum dolor sit amet consectetur",
"image": "http://localhost:8000/media/posts/car_SxXcUTV.jpg",
"slug": "lorem-ipsum-lorem-ipsum-dolor-sit-amet-consectetur",
"author": 4,
"excerpt": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"content": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"status": "published",
"published": "2021-10-01T14:46:34.872576Z"
}
FrontendPostSerializer is gonna look like this
{
"category": {
"name": "django"
},
"id": 45,
"title": "Lorem Ipsum - Lorem ipsum dolor sit amet consectetur",
"image": "http://localhost:8000/media/posts/car_SxXcUTV.jpg",
"slug": "lorem-ipsum-lorem-ipsum-dolor-sit-amet-consectetur",
"author": {
"username": "luigi.verdi"
},
"excerpt": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"content": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"status": "published",
"published": "2021-10-01T14:46:34.872576Z"
}
What I'm doing atm is using FrontendPostSerializer for showing data in the frontend, for example in a table with category name, author name and title. Instead, I'm using PostSerializer for the backend CRUD.
These are the viewsets I'm using in the views.py
class ManagePosts(viewsets.ModelViewSet):
serializer_class = PostSerializer
parser_classes = [MultiPartParser, FormParser]
def get_object(self, queryset=None, **kwargs):
item = self.kwargs.get('pk')
return get_object_or_404(Post, slug=item)
# Define Custom Queryset
def get_queryset(self):
return Post.objects.all()
class FrontendPosts(viewsets.ModelViewSet):
serializer_class = FrontendPostSerializer
def get_object(self, queryset=None, **kwargs):
item = self.kwargs.get('pk')
return get_object_or_404(Post, slug=item)
# Define Custom Queryset
def get_queryset(self):
return Post.objects.all()
I already tried to use only one serializer, I had this:
class PostSerializer(serializers.ModelSerializer):
author = AuthorSerializer(many=False, read_only=True)
category = CategorySerializer(many=False, read_only=True)
class Meta:
model = Post
fields = ('category', 'id', 'title', 'image', 'slug', 'author', 'excerpt', 'content', 'status', 'published')
but, for example, when I try to create a new post it doesn't work, because category and author are not numbers, but objects.
I'm also gonna put here create.js I have in my React frontend that handles the create submit.
const handleSubmit = (e) => {
e.preventDefault();
let formData = new FormData();
formData.append('category', 1);
formData.append('title', postData.title);
formData.append('slug', postData.slug);
formData.append('author', userInfo.id);
formData.append('excerpt', postData.excerpt);
formData.append('content', postData.content);
if(postImage.image !== null) {
formData.append('image', postImage.image);
}
axiosInstance.post('', formData);
history.push({
pathname: '/admin/',
});
window.location.reload();
};
Is there a better way? I'm sure I can use only one serializer, but I'm not sure how yet.
Thanks!

Actually yes. You can add specific fields you want by using the source attribute. Example:
class PostSerializer(serializers.ModelSerializer):
authorUserName = serializers.CharField(read_only=true, source="author.username")
categoryName = serializers.CharField(read_only=true, source="category.name"
class Meta:
model = Post
fields = (
'category', 'id', 'title',
'image', 'slug', 'author',
'excerpt', 'content', 'status',
'published', 'authorName', 'categoryName')
# Remember add the field that are created
And when you try to get you should get the result like this:
{
"categoryName": "django",
"category": 1,
"id": 45,
"title": "Lorem Ipsum - Lorem ipsum dolor sit amet consectetur",
"image": "http://localhost:8000/media/posts/car_SxXcUTV.jpg",
"slug": "lorem-ipsum-lorem-ipsum-dolor-sit-amet-consectetur",
"authorName": "luigi.verdi",
"author": 4,
"excerpt": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"content": "Officiis iure rerum voluptates a cumque velit \nquibusdam sed amet tempora. Sit laborum ab, eius fugit doloribus tenetur \nfugiat, temporibus enim commodi iusto libero magni deleniti quod quam \nconsequuntur! Commodi minima excepturi repudiandae velit hic maxime\ndoloremque.",
"status": "published",
"published": "2021-10-01T14:46:34.872576Z"
}

ValueError: Expected object or value while reading JSON using Pandas with Chunksize and lines=True

I've looked into mutliple stackoverflow threads and pandas document but I am still getting following error when trying to read json as below
source: https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-json-reader and section Line delimited json
import pandas as pd
chunks = pd.read_json('/Users/abc/PycharmProjects/test-json/records1.json', lines=True, chunksize=10)
for chunk in chunks:
print(chunk)
break
error
Traceback (most recent call last):
File "/Users/abc/PycharmProjects/abc_code/test.py", line 4, in <module>
for chunk in chunks:
File "/Users/abc/.local/share/virtualenvs/abc_code-bU7zXpdS/lib/python3.8/site-packages/pandas/io/json/_json.py", line 744, in __next__
obj = self._get_object_parser(lines_json)
File "/Users/abc/.local/share/virtualenvs/abc_code-bU7zXpdS/lib/python3.8/site-packages/pandas/io/json/_json.py", line 716, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/Users/abc/.local/share/virtualenvs/abc_code-bU7zXpdS/lib/python3.8/site-packages/pandas/io/json/_json.py", line 831, in parse
self._parse_no_numpy()
File "/Users/abc/.local/share/virtualenvs/abc_code-bU7zXpdS/lib/python3.8/site-packages/pandas/io/json/_json.py", line 1079, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None
**ValueError: Expected object or value**
PS:
It works fine if I read without lines=True, chunksize=10 params but isn't working with these params. reason I am using these params is because we get big jsons and I don't want to load all of that in memory in single go.
For testing purpose, let's say I want to filter people from this json for whom "isActive": true.
I am using python v3.7.9 and pandas v1.3.1
thanks for the help!!
For reference, please find below the sample snippet of json which I am trying to read
[{
"_id": "5c21a63a1e4bb8d6a7d1ffa0",
"index": 0,
"guid": "ae4d6f3d-ad3f-4923-8b87-8456d427f96e",
"isActive": true,
"balance": "$3,624.10",
"picture": "http://placehold.it/32x32",
"age": 30,
"eyeColor": "blue",
"name": {
"first": "Alvarado",
"last": "Castro"
},
"company": "ARTWORLDS",
"email": "alvarado.castro#artworlds.com",
"phone": "+1 (937) 418-3892",
"address": "190 Joralemon Street, Talpa, Hawaii, 4597",
"about": "Veniam ut officia exercitation eiusmod officia nulla id est consectetur. Laboris excepteur id dolore consequat dolore ad deserunt anim anim. Eu ea ad cupidatat consequat duis id ut elit adipisicing ea minim irure ullamco voluptate. Irure reprehenderit duis esse esse dolore voluptate ad. Reprehenderit veniam qui velit cupidatat. Veniam consectetur aliquip quis sit laboris sint adipisicing occaecat occaecat consequat sint.",
"registered": "Friday, October 5, 2018 6:26 PM",
"latitude": "-59.473097",
"longitude": "118.295711",
"tags": [
"adipisicing",
"incididunt",
"deserunt",
"commodo",
"nisi"
],
"range": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9
],
"friends": [{
"id": 0,
"name": "Mcgee Delacruz"
},
{
"id": 1,
"name": "Marquita Robinson"
},
{
"id": 2,
"name": "Mullen Foreman"
}
],
"greeting": "Hello, Alvarado! You have 7 unread messages.",
"favoriteFruit": "banana"
}]

RethinkDB Python Recursive Document Filter

Trying to do a recursive filter query on RethinkDB with they Python wrapper. Having a lot of trouble getting it work.
Tried a lot of variations of the query, to no avail. Essentially, I'm trying to find the rows which do not have a document nested under label with a particular user_id.
In plain english: if the current user already labeled this example, don’t return it to them again.
My non-working query:
open_tasks = rdbt \
.order_by(index=r.desc('labels_completed')) \
.filter(r.row['locked'] == False) \
.filter(lambda task:
task['labels']['user_id'] != current_user.id) \
.limit(qty) \
.run(conn)
My dataset
[
{
"id": "e54893b4-b1d0-49c5-b6aa-9aa9e7d2b73b",
"image": "https://dl.dropboxusercontent.com/u/5822/crowdlabeler/ABLXOTODWJKTXECYZTST.jpg",
"labels": [
{
"account_qty_labeled": 54,
"account_signup_date": "Tue Aug 04 2015 10:12:25 GMT-04:00",
"compensation": 0.01,
"dataset_id": 144,
"label": {
"$$hashKey": "object:45",
"answer": "Yes",
"selected": true
},
"label_duration_sec": 3,
"labeled_at": "Wed Aug 05 2015 16:26:04 GMT-05:00",
"sess_duration_sec": 3,
"sess_qty_labeled": 0,
"user_id": 1
}
],
"labels_completed": 0,
"locked": false,
"text": "Lorem ipsum dolor sit amet, consectetur adipisicing elit. Cupiditate adipisci vero minus laudantium reprehenderit exercitationem eius, suscipit facilis laboriosam consequuntur, eligendi quis mollitia excepturi deserunt dicta, dolorem quaerat pariatur provident sint explicabo. Magnam possimus dolorum beatae quidem excepturi quibusdam dolore reprehenderit accusantium quae ad libero, voluptatum laborum, incidunt, voluptate reiciendis."
},
{
"id": "9f08869e-79fd-49c0-a184-c43d2a1c95cf",
"image": "https://dl.dropboxusercontent.com/u/5822/crowdlabeler/ACSGHDYECQWQXDHIOBYC.jpg",
"labels": [],
"labels_completed": 0,
"locked": false,
"text": "Lorem ipsum dolor sit amet, consectetur adipisicing elit. Cupiditate adipisci vero minus laudantium reprehenderit exercitationem eius, suscipit facilis laboriosam consequuntur, eligendi quis mollitia excepturi deserunt dicta, dolorem quaerat pariatur provident sint explicabo. Magnam possimus dolorum beatae quidem excepturi quibusdam dolore reprehenderit accusantium quae ad libero, voluptatum laborum, incidunt, voluptate reiciendis."
},
{
"id": "9fba0a39-4cfd-4a97-b48f-e8bf2b0d46c5",
"image": "https://dl.dropboxusercontent.com/u/5822/crowdlabeler/ADMNIUYKUHAIOHMAFXBK.jpg",
"labels": [],
"labels_completed": 0,
"locked": false,
"text": "Lorem ipsum dolor sit amet, consectetur adipisicing elit. Cupiditate adipisci vero minus laudantium reprehenderit exercitationem eius, suscipit facilis laboriosam consequuntur, eligendi quis mollitia excepturi deserunt dicta, dolorem quaerat pariatur provident sint explicabo. Magnam possimus dolorum beatae quidem excepturi quibusdam dolore reprehenderit accusantium quae ad libero, voluptatum laborum, incidunt, voluptate reiciendis."
}
]
Thank you for your help!

This was a tough one but the solution ended up being two-fold.
A slight modification to the schema when saving, then using RethinkDB's contains method.
Here is the modified query that works well.
open_tasks = rdbt \
.order_by(index=r.desc('labels_completed')) \
.filter(r.row['locked'] == False) \
.filter(lambda task:
r.not_(task['labeler_ids'].contains(current_user.id))) \
.limit(qty) \
.run(conn)