Develop Reference

Access nested property in pymongo

I'm trying to access "msgId", which is nested. However, it returns the property name instead of the value. What am I doing wrong?
Code:
def get_captain_message_id(game_id, user):
data = {
"gameId":game_id,
"messageIds":str(user.id)
}
return mongo.db['GameData'].find_one(data)['msgId']

That's because you're trying to access a property of the inner document as if it belonged to the outer one.
First you have to change your query data to match the document whose messageIds object contains the user.id property.
data = {
"gameId":game_id,
f"messageIds.{user.id}": {"$exists": True}
}
Then you can access msgId value with the appropriate indexing pipeline:
return mongo.db['GameData'].find_one(data)['messageIds'][str(user.id)]['msgId']

SQLAlchemy insert from a JSON list to database

I have a list with a JSON like so:
print(type(listed)) # <class 'list'>
print (listed)
[
{
"email": "x#gmail.com",
"fullname": "xg gf",
"points": 5,
"image_url", "https://imgur.com/random.pmg"
},
{
... similar json for the next user and so on
}
]
I'm trying to insert them into my postgres database that has a model like this:
class Users(db.Model):
__tablename__ = 'users'
email = db.Column(db.String(), primary_key=True)
displayName = db.Column(db.String())
image = db.Column(db.String())
points = db.Column(db.Integer())
But I'm quite stuck, I've tried several approaches but none worked, anyone can guide me with an example on how to do it properly?

Here's a solution without pandas, using SQLAlchemy Core
create engine
engine = sqlalchemy.create_engine('...')
load the metadata using the engine as the bind parameter
metadata = sqalchemy.Metadata(bind=engine)
make a reference to the table
users_table = sqlalchemy.Table('users', metadata, autoload = True)
you can then start your inserts
for user in json:
query = users_table.insert()
query.values(**user)
my_session = Session(engine)
my_session.execute(query)
my_session.close()
This creates a session for every user in json, but I thought you might like it anyway. Its very flexible and works for any table, you don't even need a model. Just make sure the json doesnt contain any columns that dont exist in the db (this means you will need to use "img_url" (column name) in both the json key and in the db column name)

Here is an example json list, like you provided.
json = [
{
"email": "x#gmail.com",
"fullname": "xg gf",
"points": 5,
"image_url": "https://imgur.com/random.pmg"
},
{
"email": "onur#gmail.com",
"fullname": "o g",
"points": 7,
"image_url": "https://imgur.com/random_x.pmg"
}
]
Now create an empty dataframe all_df and run iterations inside your json list.
Each iteration creates a dataframe with the data from dictionary inside the list, transpose it and append to all_df.
import pandas as pd
all_df = pd.DataFrame()
for i in json:
df = pd.DataFrame.from_dict(data=i, orient='index').T
all_df = all_df.append(df)
Output:
Now you can go ahead create a session to your database and push all_df
all_df.to_sql(con=your_session.bind, name='your_table_name', if_exists='your_preferred_method', index=False)

Using marshmallow-sqlalchemy
validate the incoming JSON
create general utilities for loading and dumping data
Define schemas
schema.py
from marshmallow import EXCLUDE
from marshmallow_sqlalchemy import ModelSchema
from app import db
class UserSchema(ModelSchema):
class Meta(ModelSchema.Meta):
model = Users
sqla_session = db.session
user_schema_full = UserSchema(only=(
'email',
'displayName',
'image',
'points'
))
utils.py
Exact details below don't matter but create general utility for going from JSON to ORM objects and ORM objects to JSON. schema_partial used for auto generated primary keys.
def loadData(data, schema_partial, many=False,
schema_full=None, instance=None):
try:
if instance is not None:
answer = schema_full.load(data, instance=instance, many=many)
else:
answer = schema_partial.load(data, many=many)
except ValidationError as errors:
raise InvalidData(errors, status_code=400)
return answer
def loadUser(data, instance=None, many=False):
return loadData(data=data,
schema_partial=user_schema_full,
many=many,
schema_full=user_schema_full,
instance=instance)
def dumpData(load_object, schema, many=False):
try:
answer = schema.dump(load_object, many=many)
except ValidationError as errors:
raise InvalidDump(errors, status_code=400)
return answer
def dumpUser(load_object, many=False):
return dumpData(load_object, schema=user_schema_full, many=many)
Use loadUser and dumpUser within api to produce clean flat code.
api.py
#app.route('/users/', methods=['POST'])
def post_users():
"""Post many users"""
users_data = request.get_json()
users = loadUser(users_data, many=True)
for user in users:
db.session.add(user)
object_dump = dumpUser(users, many=True)
db.session.commit()
return jsonify(object_dump), 201

Django & Mongoengine get data from embedded document?

I am having trouble retrieving data from the embedded document in mongoengine.
models.py
from mongoengine import Document, EmbeddedDocument, fields
class ProductFields(EmbeddedDocument):
key_name = fields.StringField(required=True)
value = fields.DynamicField(required=True)
class Product(Document):
name = fields.StringField(required=True)
description = fields.StringField(required=True, null=True)
fields = fields.ListField(fields.EmbeddedDocumentField(ProductFields))
views.py
class ProductListView(APIView):
def get(self, request):
# list_products = Product.objects.all()
result=[]
productfields = ProductFields
for product in Product.objects:
data={
"name":product.name,
"description":product.description,
# "key":product.fields.key_name,
# "value":ProductFields.value,
}
print (data)
# print(productfields.key_name)
result.append(data)
return Response({"products":result,"message":"list of products.","requestStatus":1},status=status.HTTP_200_OK)
Output:
{
"description": "test description",
"name": "product1"
"fields":[
{ "key_name" : value},
{ "key_name" : value},
]
}
How do I get the above-desired output? Print function doesn't work because mongoengine returns object and not the value.

Here I see you're using APIView from Django Rest FrameWork. Have a look at django-rest-framework-mongoengine. If you're already familiar with DRF, you can use this extension to create your API endpoints with MongoDB easily.
You must have found some workaround by now even though you can mark this answer as correct so that if anyone else runs into the same problem in future, they can get the solution.

Cosmos DB - Delete Document with Python

In this SO question I had learnt that I cannot delete a Cosmos DB document using SQL.
Using Python, I believe I need the DeleteDocument() method. This is how I'm getting the document ID's that are required (I believe) to then call the DeleteDocument() method.
# set up the client
client = document_client.DocumentClient()
# use a SQL based query to get a bunch of documents
query = { 'query': 'SELECT * FROM server s' }
result_iterable = client.QueryDocuments('dbs/DB/colls/coll', query, options)
results = list(result_iterable);
for x in range(0, len (results)):
docID = results[x]['id']
Now, at this stage I want to call DeleteDocument().
The inputs into which are document_link and options.
I can define document_link as something like
document_link = 'dbs/DB/colls/coll/docs/'+docID
And successfully call ReadAttachments() for example, which has the same inputs as DeleteDocument().
When I do however, I get an error...
The partition key supplied in x-ms-partitionkey header has fewer
components than defined in the the collection
...and now I'm totally lost
UPDATE
Following on from Jay's help, I believe I'm missing the partitonKey element in the options.
In this example, I've created a testing database, it looks like this
So I think my partition key is /testPART
When I include the partitionKey in the options however, no results are returned, (and so print len(results) outputs 0).
Removing partitionKey means that results are returned, but the delete attempt fails as before.
# Query them in SQL
query = { 'query': 'SELECT * FROM c' }
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
options['partitionKey'] = '/testPART'
result_iterable = client.QueryDocuments('dbs/testDB/colls/testCOLL', query, options)
results = list(result_iterable)
# should be > 0
print len(results)
for x in range(0, len (results)):
docID = results[x]['id']
print docID
client.DeleteDocument('dbs/testDB/colls/testCOLL/docs/'+docID, options=options)
print 'deleted', docID

According to your description, I tried to use pydocument module to delete document in my azure document db and it works for me.
Here is my code:
import pydocumentdb;
import pydocumentdb.document_client as document_client
config = {
'ENDPOINT': 'Your url',
'MASTERKEY': 'Your master key',
'DOCUMENTDB_DATABASE': 'familydb',
'DOCUMENTDB_COLLECTION': 'familycoll'
};
# Initialize the Python DocumentDB client
client = document_client.DocumentClient(config['ENDPOINT'], {'masterKey': config['MASTERKEY']})
# use a SQL based query to get a bunch of documents
query = { 'query': 'SELECT * FROM server s' }
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
result_iterable = client.QueryDocuments('dbs/familydb/colls/familycoll', query, options)
results = list(result_iterable);
print(results)
client.DeleteDocument('dbs/familydb/colls/familycoll/docs/id1',options)
print 'delete success'
Console Result:
[{u'_self': u'dbs/hitPAA==/colls/hitPAL3OLgA=/docs/hitPAL3OLgABAAAAAAAAAA==/', u'myJsonArray': [{u'subId': u'sub1', u'val': u'value1'}, {u'subId': u'sub2', u'val': u'value2'}], u'_ts': 1507687788, u'_rid': u'hitPAL3OLgABAAAAAAAAAA==', u'_attachments': u'attachments/', u'_etag': u'"00002100-0000-0000-0000-59dd7d6c0000"', u'id': u'id1'}, {u'_self': u'dbs/hitPAA==/colls/hitPAL3OLgA=/docs/hitPAL3OLgACAAAAAAAAAA==/', u'myJsonArray': [{u'subId': u'sub3', u'val': u'value3'}, {u'subId': u'sub4', u'val': u'value4'}], u'_ts': 1507687809, u'_rid': u'hitPAL3OLgACAAAAAAAAAA==', u'_attachments': u'attachments/', u'_etag': u'"00002200-0000-0000-0000-59dd7d810000"', u'id': u'id2'}]
delete success
Please notice that you need to set the enableCrossPartitionQuery property to True in options if your documents are cross-partitioned.
Must be set to true for any query that requires to be executed across
more than one partition. This is an explicit flag to enable you to
make conscious performance tradeoffs during development time.
You could find above description from here.
Update Answer:
I think you misunderstand the meaning of partitionkey property in the options[].
For example , my container is created like this:
My documents as below :
{
"id": "1",
"name": "jay"
}
{
"id": "2",
"name": "jay2"
}
My partitionkey is 'name', so here I have two paritions : 'jay' and 'jay1'.
So, here you should set the partitionkey property to 'jay' or 'jay2',not 'name'.
Please modify your code as below:
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
options['partitionKey'] = 'jay' (please change here in your code)
result_iterable = client.QueryDocuments('dbs/db/colls/testcoll', query, options)
results = list(result_iterable);
print(results)
Hope it helps you.

Using the azure.cosmos library:
install and import azure cosmos package:
from azure.cosmos import exceptions, CosmosClient, PartitionKey
define delete items function - in this case using the partition key in query:
def deleteItems(deviceid):
client = CosmosClient(config.cosmos.endpoint, config.cosmos.primarykey)
# Create a database if not exists
database = client.create_database_if_not_exists(id=azure-cosmos-db-name)
# Create a container
# Using a good partition key improves the performance of database operations.
container = database.create_container_if_not_exists(id=container-name, partition_key=PartitionKey(path='/your-pattition-path'), offer_throughput=400)
#fetch items
query = f"SELECT * FROM c WHERE c.device.deviceid IN ('{deviceid}')"
items = list(container.query_items(query=query, enable_cross_partition_query=False))
for item in items:
container.delete_item(item, 'partition-key')
usage:
deviceid=10
deleteItems(items)
github full example here: https://github.com/eladtpro/python-iothub-cosmos

Graphene/Django (GraphQL): How to use a query argument in order to exclude nodes matching a specific filter?

I have some video items in a Django/Graphene backend setup. Each video item is linked to one owner.
In a React app, I would like to query via GraphQL all the videos owned by the current user on the one hand and all the videos NOT owned by the current user on the other hand.
I could run the following GraphQl query and filter on the client side:
query AllScenes {
allScenes {
edges {
node {
id,
name,
owner {
name
}
}
}
}
}
I would rather have two queries with filters parameters directly asking relevant data to my backend. Something like:
query AllScenes($ownerName : String!, $exclude: Boolean!) {
allScenes(owner__name: $ownerName, exclude: $exclude) {
edges {
node {
id,
name,
owner {
name
}
}
}
}
}
I would query with ownerName = currentUserName and exclude = True/False yet I just cannot retrieve my exclude argument on my backend side. Here is the code I have tried in my schema.py file:
from project.scene_manager.models import Scene
from graphene import ObjectType, relay, Int, String, Field, Boolean, Float
from graphene.contrib.django.filter import DjangoFilterConnectionField
from graphene.contrib.django.types import DjangoNode
from django_filters import FilterSet, CharFilter



class SceneNode(DjangoNode):

 class Meta:
model = Scene



class SceneFilter(FilterSet):
owner__name = CharFilter(lookup_type='exact', exclude=exclude)

class Meta:
model = Scene
fields = ['owner__name']

class Query(ObjectType):

scene = relay.NodeField(SceneNode)
all_scenes = DjangoFilterConnectionField(SceneNode, filterset_class=SceneFilter, exclude=Boolean())

def resolve_exclude(self, args, info):
exclude = args.get('exclude')
return exclude


class Meta:
abstract = True
My custom SceneFilter is used but I do not know how to pass the exclude arg to it. (I do not think that I am making a proper use of the resolver). Any help on that matter would be much appreciated!

Switching to graphene-django 1.0, I have been able to do what I wanted with the following query definition:
class Query(AbstractType):
selected_scenes = DjangoFilterConnectionField(SceneNode, exclude=Boolean())
def resolve_selected_scenes(self, args, context, info):
owner__name = args.get('owner__name')
exclude = args.get('exclude')
if exclude:
selected_scenes = Scene.objects.exclude(owner__name=owner__name)
else:
selected_scenes = Scene.objects.filter(owner__name=owner__name)
return selected_scenes
BossGrand proposed an other solution on GitHub