Develop Reference

Problem with continuation token when querying from Cosmos DB

I'm facing a problem with continuation when querying items from CosmosDB.
I've already tried the following solution but with no success. I'm only able to query the first 10 results of a page even though I get a token that is not NULL.
The token has a size of 10733 bytes and looks like this.
{"token":"+RID:gtQwAJ9KbavOAAAAAAAAAA==#RT:1#TRC:10#FPP:AggAAAAAAAAAAGoAAAAAKAAAAAAAAAAAAADCBc6AEoAGgAqADoASgAaACoAOgBKABoAKgA6AE4AHgAuAD4ASgAeACoAPgBOAB4ALgA+AE4AHgAqAD4ASgAeAC4APgBOAB4ALgA+AE4AIgA2AEYAFgAmADYARgAaACYAPgBKABYAKgA6AE4AHgAuAD4ATgAeAC4APgBOAB4ALgA+AE4AIgAuAD4ATgAeAC4APgBOACIAMgA+AFIAIgAyAD4AUgAmADIAQgAWACIALgBCABIAIgAyAEIAEgAiADIAQgAOACYANgBKAB4AJgA6AEYAGgAqADoATgAeAC4APgB....etc...etc","range":{"min":"","max":"05C1BF3FB3CFC0"}}
Code looks like this. Function QueryDocuments did not work. Instead I had to use QueryItems.
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 10
q = client.QueryItems(collection_link, query, options)
results_1 = q._fetch_function(options)
#this is a string representing a JSON object
token = results_1[1]['x-ms-continuation']
data = list(q._fetch_function({'maxItemCount':10,'enableCrossPartitionQuery':True, 'continuation':token}))
Is there a solution to this? Thanks for your help.

Please use pydocumentdb package and refer to below sample code.
from pydocumentdb import document_client
endpoint = "https://***.documents.azure.com:443/";
primaryKey = "***";
client = document_client.DocumentClient(endpoint, {'masterKey': primaryKey})
collection_link = "dbs/db/colls/coll"
query = "select c.id from c"
query_with_optional_parameters = [];
q = client.QueryDocuments(collection_link, query, {'maxItemCount': 2})
results_1 = q._fetch_function({'maxItemCount': 2})
print(results_1)
token = results_1[1]['x-ms-continuation']
results_2 = q._fetch_function({'maxItemCount': 2, 'continuation': token})
print(results_2)
Output:

Mongoengine query with date range

I'm trying to retrieve data from mongodb via mongoengine within a specified time span. Below is the db model used.
class DeviationReport(db.Document):
meta = {'collection': 'DeviationReport'}
created_at = db.DateTimeField()
date = db.DateTimeField()
author = db.StringField()
read_by = db.ListField(default=[])
prod_line = db.ReferenceField(ProductionLine)
product = db.ReferenceField(Product)
description = db.StringField()
What I've tried is the code below. It does however not return any results. I've used a similar approach when I've needed to build dynamic queries depending on user input.
kwargs = {}
start = datetime.datetime(2018, 12, 11)
end = datetime.datetime(2019, 03, 13)
kwargs['created_at'] = { '$lt': end, '$gt': start }
DeviationReport.objects(**kwargs)
I've obviously made sure that there are objects within the date range, and I've read other similar posts where the query below has been successfully used. How do I get my query to return everything between 'start' and 'end', or how do I rewrite it to do as I wish?
Thanks you.

I worked around/solved the problem by first getting my results sans date filtering using **kwargs and then filtered that using Q. It may not be optimal, but it works for what I need it to do.
reports = DeviationReport.objects(**kwargs)
reports = reports.filter((Q(date__gte=start) & Q(date__lte=end)))

There's a number of ways to achieve your query, adjust the collection and params accordingly using the example below:
date_to = datetime.datetime.utcnow() # The end date
date_from = date_to - datetime.timedelta(days=120) # The start date
query_a = Application.objects(category="rest_api").filter(
date_created__gte=date_from,
date_created__lte=date_to
)
query_b = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to
).filter(category="rest_api")
query = {"category": "rest_api"}
query_c = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to,
**query
)

Querying with Q as suggested above did not work for me, but a raw query did:
raw_query = {'date': {'$gte': start, '$lt': end}}
reports = DeviationReport.objects(__raw__=raw_query)

AWS DynamoDB ExclusiveStartKey default value

I'm trying to make a query to DynamoDB, and if a LastEvaluatedKey is returned (meaning the query exceeds 1 MB) I want to make other queries in order to fetch all the required data from the table, using LastEvaluatedKey as ExclusiveStartKey for the next query.
This is the code I have for now:
query_response = table.query(
KeyConditionExpression=Key('brand').eq(brand)
)
pagination_key = None
if 'LastEvaluatedKey' in query_response:
pagination_key = query_response['LastEvaluatedKey']
while pagination_key:
next_query_response = table.query(
KeyConditionExpression=Key('brand').eq(brand),
ExclusiveStartKey=pagination_key
)
However, I'd like to refacto this code by extracting the query into a method, passing it pagination_key as an argument. To do this, I'd have to be able to either set ExclusiveStartKey to False, None or some other default value for the first call but I didn't find anything on this, or I'd have to be able to exclude the ExclusiveStartKey alltogether, but I don't know how to do this either.

Using keyword Arguments **kwargs it might look like this. Also,
I am setting up the query before and only updating the ExclusiveStartKeyevery time.
query = { "KeyConditionExpression": Key('brand').eq(brand) }
ExclusiveStartKey = None
while True:
if ExclusiveStartKey is not None:
query['ExclusiveStartKey'] = ExclusiveStartKey
query_response = table.query(**query)
if 'LastEvaluatedKey' in query_response:
ExclusiveStartKey = query_response['LastEvaluatedKey']
else:
break

I found an easy way of building the parameters:
query_params = { 'KeyConditionExpression': Key('brand').eq(brand) }
if pagination_key:
query_params['ExclusiveStartKey'] = pagination_key
query_response = table.query(query_params)

MongoDB Dynamic Field Creation

I have a dictionary to be inserted dynamically in MongoDB.
Current MongoDB document-
"Customers":{
"Payment":{
"CustomerDetails":{
"Source":"Visa Card",
"Name" :"John",
}
}
}
The document that I am trying to insert into this through python dictionary object -
final= {"CustomerPayable":["Month":"Feb-1FN-2018","Details":
["Code":"ghg23","AmtPayable": "123.00"]]}
The query I am trying -
db.collection.update({"UserID":UserID},{ '$push':{
'Customers.Payment.Output':final}})
I wanted the dynamic field of "Output" to be created through the above query. Expected output-
"Customers":{
"Payment":{
"CustomerDetails":{
"Source":"Visa Card",
"Name" :"John",
},
"Output":{"CustomerPayable":["Month":"Feb-1FN-2018",Details:
["Code":"ghg23","AmtPayable": "123.00"]]}
}
}
Any help is great.Thanks in advance

The following code should achieve your desired results.
from pymongo import MongoClient
client = MongoClient()
db = client.stackoverflow
collection = db.stackoverflow
a = {"Customers":{ "Payment":{ "CustomerDetails":{ "Source":"Visa Card", "Name" :"John"}}}}
collection.insert(a)
# Prints object before update.
cur = collection.find_one({"Customers.Payment.CustomerDetails.Name":"John"})
print(cur)
final = {"CustomerPayable":{"Month":"Feb-1FN-2018","Details":
{"Code":"ghg23","AmtPayable": "123.00"}}}
collection.update({"Customers.Payment.CustomerDetails.Name":"John"},
{'$push':{'Customers.Payment.Output':final}})
# Prints object after update.
cur = collection.find_one({"Customers.Payment.CustomerDetails.Name":"John"})
print(cur)
A couple things wrong with your code are:
In you final declaration you tried to use dictionary syntax inside of a list.
In your update query you don't have a field called UserID for I changed it to query on Name
Anyways, I hope this helps.

Cosmos DB - Delete Document with Python

In this SO question I had learnt that I cannot delete a Cosmos DB document using SQL.
Using Python, I believe I need the DeleteDocument() method. This is how I'm getting the document ID's that are required (I believe) to then call the DeleteDocument() method.
# set up the client
client = document_client.DocumentClient()
# use a SQL based query to get a bunch of documents
query = { 'query': 'SELECT * FROM server s' }
result_iterable = client.QueryDocuments('dbs/DB/colls/coll', query, options)
results = list(result_iterable);
for x in range(0, len (results)):
docID = results[x]['id']
Now, at this stage I want to call DeleteDocument().
The inputs into which are document_link and options.
I can define document_link as something like
document_link = 'dbs/DB/colls/coll/docs/'+docID
And successfully call ReadAttachments() for example, which has the same inputs as DeleteDocument().
When I do however, I get an error...
The partition key supplied in x-ms-partitionkey header has fewer
components than defined in the the collection
...and now I'm totally lost
UPDATE
Following on from Jay's help, I believe I'm missing the partitonKey element in the options.
In this example, I've created a testing database, it looks like this
So I think my partition key is /testPART
When I include the partitionKey in the options however, no results are returned, (and so print len(results) outputs 0).
Removing partitionKey means that results are returned, but the delete attempt fails as before.
# Query them in SQL
query = { 'query': 'SELECT * FROM c' }
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
options['partitionKey'] = '/testPART'
result_iterable = client.QueryDocuments('dbs/testDB/colls/testCOLL', query, options)
results = list(result_iterable)
# should be > 0
print len(results)
for x in range(0, len (results)):
docID = results[x]['id']
print docID
client.DeleteDocument('dbs/testDB/colls/testCOLL/docs/'+docID, options=options)
print 'deleted', docID

According to your description, I tried to use pydocument module to delete document in my azure document db and it works for me.
Here is my code:
import pydocumentdb;
import pydocumentdb.document_client as document_client
config = {
'ENDPOINT': 'Your url',
'MASTERKEY': 'Your master key',
'DOCUMENTDB_DATABASE': 'familydb',
'DOCUMENTDB_COLLECTION': 'familycoll'
};
# Initialize the Python DocumentDB client
client = document_client.DocumentClient(config['ENDPOINT'], {'masterKey': config['MASTERKEY']})
# use a SQL based query to get a bunch of documents
query = { 'query': 'SELECT * FROM server s' }
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
result_iterable = client.QueryDocuments('dbs/familydb/colls/familycoll', query, options)
results = list(result_iterable);
print(results)
client.DeleteDocument('dbs/familydb/colls/familycoll/docs/id1',options)
print 'delete success'
Console Result:
[{u'_self': u'dbs/hitPAA==/colls/hitPAL3OLgA=/docs/hitPAL3OLgABAAAAAAAAAA==/', u'myJsonArray': [{u'subId': u'sub1', u'val': u'value1'}, {u'subId': u'sub2', u'val': u'value2'}], u'_ts': 1507687788, u'_rid': u'hitPAL3OLgABAAAAAAAAAA==', u'_attachments': u'attachments/', u'_etag': u'"00002100-0000-0000-0000-59dd7d6c0000"', u'id': u'id1'}, {u'_self': u'dbs/hitPAA==/colls/hitPAL3OLgA=/docs/hitPAL3OLgACAAAAAAAAAA==/', u'myJsonArray': [{u'subId': u'sub3', u'val': u'value3'}, {u'subId': u'sub4', u'val': u'value4'}], u'_ts': 1507687809, u'_rid': u'hitPAL3OLgACAAAAAAAAAA==', u'_attachments': u'attachments/', u'_etag': u'"00002200-0000-0000-0000-59dd7d810000"', u'id': u'id2'}]
delete success
Please notice that you need to set the enableCrossPartitionQuery property to True in options if your documents are cross-partitioned.
Must be set to true for any query that requires to be executed across
more than one partition. This is an explicit flag to enable you to
make conscious performance tradeoffs during development time.
You could find above description from here.
Update Answer:
I think you misunderstand the meaning of partitionkey property in the options[].
For example , my container is created like this:
My documents as below :
{
"id": "1",
"name": "jay"
}
{
"id": "2",
"name": "jay2"
}
My partitionkey is 'name', so here I have two paritions : 'jay' and 'jay1'.
So, here you should set the partitionkey property to 'jay' or 'jay2',not 'name'.
Please modify your code as below:
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 2
options['partitionKey'] = 'jay' (please change here in your code)
result_iterable = client.QueryDocuments('dbs/db/colls/testcoll', query, options)
results = list(result_iterable);
print(results)
Hope it helps you.

Using the azure.cosmos library:
install and import azure cosmos package:
from azure.cosmos import exceptions, CosmosClient, PartitionKey
define delete items function - in this case using the partition key in query:
def deleteItems(deviceid):
client = CosmosClient(config.cosmos.endpoint, config.cosmos.primarykey)
# Create a database if not exists
database = client.create_database_if_not_exists(id=azure-cosmos-db-name)
# Create a container
# Using a good partition key improves the performance of database operations.
container = database.create_container_if_not_exists(id=container-name, partition_key=PartitionKey(path='/your-pattition-path'), offer_throughput=400)
#fetch items
query = f"SELECT * FROM c WHERE c.device.deviceid IN ('{deviceid}')"
items = list(container.query_items(query=query, enable_cross_partition_query=False))
for item in items:
container.delete_item(item, 'partition-key')
usage:
deviceid=10
deleteItems(items)
github full example here: https://github.com/eladtpro/python-iothub-cosmos