Develop Reference

Pymongo Insert Doc with String ID's rather than ObjectIDs

I am trying to get pymongo to insert new documents which have id's in string format rather than ObjectId's. The app I am building integrates meteor and python and meteor inserts string id's so having to work with both string and Objectids adds complexity.
Example:
Meteor-inserted doc:
{
"_id" : "22FHWpvqrAeyfvh7B"
}
Pymongo-inserted doc:
{
"_id" : ObjectId("5880387d1fd21c2dc66e9b7d")
}

You could just switch your Meteor app to insert ObjectIds instead of strings. Just use the idGeneration option property and set the value to MONGO.
Here is an example.
var todos = new Mongo.Collection('todos', {
idGeneration: 'MONGO'
});
It is described in the Meteor docs here.
Or, if you want Meteir to keep strings and can't figure out how to configure Pymongo to store as strings, then you can do the approach described here to convert between ObjectIds and strings in Pymongo..

What is the $project (aggregation) API for Pymongo?

I'm rather new to MongoDB, I can find some commands in shell to execute my query, however, I can not find a proper function in PyMongo API manual.
For example, I would like to project some of the fields of the document to a new document. I suppose the $project could do it, but there is no such support in Pymongo. How could I execute the same query both in shell and Python? For example:
db.books.aggregate( [ { $project : { title : 1 , author : 1 } } ] )

For projecting you may use the query as
db.books.aggregate([{'$project':{ 'title':'$title', 'author':'$author'}}])

Mongoengine filter query on list embedded field based on last index

I'm using Mongoengine with Django.
I have an embedded field in my model. that is a list field of embedded documents.
import mongoengine
class OrderStatusLog(mongoengine.EmbeddedDocument):
status_code = mongoengine.StringField()
class Order(mongoengine.DynamicDocument):
incr_id = mongoengine.SequenceField()
status = mongoengine.ListField(mongoengine.EmbeddedDocumentField(OrderStatusLog))
Now I want to filter the result on Order collection based on the last value in status field.
e.g. Order.objects.filter(status__last__status_code="scode")
I guess there is no such thing __last. I tried the approach mentioned in the docs http://docs.mongoengine.org/guide/querying.html#querying-lists
but didn't work.
I can solve this by looping over all the documents in the collection but thats not efficient, how can we write this query efficiently.

I'm not sure MongoEngine can do that (yet). AFAIK, you'd need to use the aggregation pipeline.
In the Mongo shell, using the '$slice' and the $arrayElemAt operators:
db.order.aggregate([{ $project: {last_status: { $arrayElemAt: [{ $slice: [ "$status", -1 ] }, 0 ]} }}, {$match: {'last_status.status_code':"scode"}} ])
And in Python:
pipeline = [
{'$project': {'last_status': { '$arrayElemAt': [{ '$slice': [ "$status", -1 ] }, 0 ]} }},
{'$match': {'last_status.status_code':'scode'}}
]
agg_cursor = Order.objects.aggregate(*pipeline)
result = [ Order.objects.get(id=order['_id']) for order in agg_cursor ]
The trick here is that objects.aggregate provides a PyMongo cursor, not a MongoEngine cursor, so if you need MongoEngine objects, you can proceed in two steps: first filter using the aggregation framework to get the ids of matched items, then get them through a MongoEngine query.
This is what I do. From my tests, it had proven to be much more efficient than fetching everything and filtering in the python code.
If there is a simpler way, I'm interested to ear about it. Otherwise, this could be a feature request for MongoEngine. You may want to open an issue there.

Check for existence of multiple fields in MongoDB document

I am trying to query a database collection that holds documents of processes for those documents that have specific fields. For simplicity imagine the following general document schema:
{
"timestamp": ISODate("..."),
"result1": "pass",
"result2": "fail"
}
Now, when a process is started a new document is inserted with only the timestamp. When that process reaches certain stages the fields result1 and result2 are added over time. Some processes however do not reach the stages 1 or 2 and therefore have no result fields.
I would like to query the database to retrieve only those documents, which have BOTH result1 and result2.
I am aware of the $exists operator, but as far as I can tell this only works for one field at a time, i.e. db.coll.find({"result1": {$exists: true}}). The $exists operator cannot be used as a top level operator. E.g. this does not work:
db.coll.find({"$exists": {"result1": true, "result2": true}})
To check for both results I would need:
db.coll.find({"result1": {"$exists": true}, "result2": {"$exists": true}})
Now that already becomes tedious for more than one variable.
Is there a better way to do this?
(Also, I am doing this in Python, so if there is a solution for just the pymongo driver that would make me happy already.)

I don't know about better, but you can always process with JavaScript via $where:
jsStr = """var doc = this;
return ['result1','result2','result3']
.every(function(key) {
return doc.hasOwnProperty(key)
});"""
coll.find({ "$where": jsStr })
But you are going to have to specify an array of "keys" to check for somewhere.
If you think you have a lot of keys to type out, then why not just "build" your query expression:
whitelist = [ "result1", "result2", "result3" ]
query = {}
for key in whitelist:
query[key] = { "$exists": True }
coll.find(query)
That saves a bit of typing and since all MongoDB queries are just data structures anyway then using basic data manipulation to build queries makes sense.

How about using $and:
db.coll.find({"$and": [
{ "fld1": { "$exists": true }}
, { "fld2": { "$exists": true }}
, { "fld3": { "$exists": true }}
]})

How to use $push/$addToSet mongodb update modifiers on python dicts

Mongodb updates provide the $push modifier to append to an array. My problem is that i want this to happen on a dict e.g
If my record looks like this initially:
{"collaborations":{'id1':{'role':'dev','scope':'dev'}}}
I want to add another item("id2" below) to the "collaborations" field dict to look something like this:
{"collaborations":{'id1':{'role':'dev','scope':'dev'},'id2':{'role':'qa','scope':'qa'}}}
I am trying with $push:
my_record.update({match_criteria},{$push,{"collaborations":{'id2':{'role':'qa','scope':'qa'}}}})
and also with $addToSet:
my_record.update({match_criteria},{$,{"collaborations":{'id2':{'role':'qa','scope':'qa'}}}})
With both of these, mongodb throws as error "Cannot apply $addToSet($push) modifier to non-array".
How can this be done for dict types? Any ideas?

The problem is that $addToSet and $push modifiers work with arrays.
To update sub-document (that is what you need here) just use $set modifier with dot notation to access sub-document (field.subfield):
my_record.update({
match_criteria
}, {
'$set': {
'collaborators.id2': {
// new document fields here
}
}
})