I have an API from which I receive a query. This API is in Python.
I call it from a django app (views.py). Then, I want to query my MongoDB collection, using mongoengine:
api_response = requests.get("http://*******", {'query':query}) #We call the API
json_resp = api_response.json()
person = Person.objects(__raw__=json_resp).to_json() #We search for the json_query in the DB (raw) and the output is JSON
It works fine but I have a problem with dates... Indeed, my Person model is as follow:
class Person(DynamicDocument):
# Meta Variables
meta = {
'collection':'personsExample'
}
#Document variables
PersonID = models.CharField(max_length = 6)
FirstName = models.CharField(max_length = 50)
LastName = models.CharField(max_length = 50)
Gender = models.CharField(max_length = 6) #male/female
BirthDate = models.DateField()
CitizenCountryCode = models.CharField(max_length = 2)
My personsExample collection was imported via mongoimport from a CSV file:
mongoimport --host localhost --db persons --collection personsExample --type csv --file reducedPersonsExtract.csv --headerline
As the birth dates were set as string, I have converted them using:
db.personsExample.find().forEach(function(el){el.BirthDate = new ISODate(el.BirthDate); db.personsExample.save(el)})
The problem I have now is that it gives BirthDate field as follow:
"BirthDate" : ISODate("1970-12-21T00:00:00Z")
But in my json query, date is stored as
datetime.datetime(1970,12,21,0,0,0).isoformat()
Which gives:
{
"BirthDate": "1970-12-21T00:00:00"
}
Thus, the query doesn't work, I would need to make my query with
{
"BirthDate": ISODate"1970-12-21T00:00:00Z"
}
(But I can't create such objects (ISODate) with Python... )
Or to find another way to store the date in MongoDB.
Would you happen to know how I could solve my problem please?
I have succeeded to to what I wanted to. In fact, I don't convert dates to ISODate in the DB, I store them as string "YYYY-MM-DD". Then, I format the date in my API (which sends the JSON query to the App which uses MongoDB) :
my_dict['BirthDate'] = datetime.datetime(YYYY,MM,DD).isoformat()
Then, in my App :
json_resp = api_response.json() #Deserialization of the response
json_resp['BirthDate'] = datetime.datetime.strptime(json_resp['BirthDate'], "%Y-%m-%dT%H:%M:%S")
It may not be the best solution but it works.
Related
I'm trying to retrieve data from mongodb via mongoengine within a specified time span. Below is the db model used.
class DeviationReport(db.Document):
meta = {'collection': 'DeviationReport'}
created_at = db.DateTimeField()
date = db.DateTimeField()
author = db.StringField()
read_by = db.ListField(default=[])
prod_line = db.ReferenceField(ProductionLine)
product = db.ReferenceField(Product)
description = db.StringField()
What I've tried is the code below. It does however not return any results. I've used a similar approach when I've needed to build dynamic queries depending on user input.
kwargs = {}
start = datetime.datetime(2018, 12, 11)
end = datetime.datetime(2019, 03, 13)
kwargs['created_at'] = { '$lt': end, '$gt': start }
DeviationReport.objects(**kwargs)
I've obviously made sure that there are objects within the date range, and I've read other similar posts where the query below has been successfully used. How do I get my query to return everything between 'start' and 'end', or how do I rewrite it to do as I wish?
Thanks you.
I worked around/solved the problem by first getting my results sans date filtering using **kwargs and then filtered that using Q. It may not be optimal, but it works for what I need it to do.
reports = DeviationReport.objects(**kwargs)
reports = reports.filter((Q(date__gte=start) & Q(date__lte=end)))
There's a number of ways to achieve your query, adjust the collection and params accordingly using the example below:
date_to = datetime.datetime.utcnow() # The end date
date_from = date_to - datetime.timedelta(days=120) # The start date
query_a = Application.objects(category="rest_api").filter(
date_created__gte=date_from,
date_created__lte=date_to
)
query_b = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to
).filter(category="rest_api")
query = {"category": "rest_api"}
query_c = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to,
**query
)
Querying with Q as suggested above did not work for me, but a raw query did:
raw_query = {'date': {'$gte': start, '$lt': end}}
reports = DeviationReport.objects(__raw__=raw_query)
Currently I am using Django 1.10.
For example we have the models below:
class ModelOne(models.Model):
description = models.CharField(max_length = 50)
class ModelTwo(models.Model):
description = models.CharField(max_length = 50)
m1 = models.ForeignKey( ModelOne)
class ModelThree(models.Model):
description = models.CharField(max_length = 50)
m2 = models.ForeignKey(ModelTwo)
Everytime a request is made then a JSON response is return using the code below:
from app.models import *
from django.http import HttpRequest,JsonResponse
def ViewThatReceiveRequest(request):
qs = list(ModelThree.objects.filter(description__icontains='beauty').select_related('m2__m1')
data = []
for key in qs:
temp_obj = {}
# if necessary I will convert the values before equating
temp_obj['desc3'] = key.description
temp_obj['desc2'] = key.m2.description
temp_obj['desc1'] = key.m2.m1.description
data.append(temp_obj)
return JsonResponse( {'message' : 'success', 'qs' : data }, safe = False)
Note:
- I am using Django as my back-end and ReactJs for my front-end so I only need JSON Response
My question,
- Do I need to do these for every views that receive request?
- Do we have other way to solve these simpler?
Since you only need description field from all 3 models, you can also use values method or maybe values_list of the queryset and get data.
So this new query would be
ModelThree.objects.filter(description__icontains='beauty').select_related('m2__m1).values_list('description','m2__description','m2__m1__description')
Next if you want to run some changes to your data, you can run a map method in case you want to make similar changes to all the data.
Next you can make the appropriate json and send to to client side.
I have simple REST/json api. I am trying to update a mongoengine model Listing using a PUT call. I get one from the mongodb and update it using my own deserilize method with the incoming json. The update does not work as the json has a few DBRef and EmbeddedDocuments. The object does get updated with the correct values before the save is executed. There is no error but the object does not get saved. Any thoughts?
obj = Listing.objects.get(pk=id)
obj.deserialize(**request.json)
obj.save()
obj.reload()
return obj
class Listing():
name = db.StringField()
l_type = db.StringField( choices=listing_const.L_TYPE.choices(), )
expiry = db.ComplexDateTimeField( auto_now=False,auto_now_add=False, )
a_data = db.EmbeddedDocumentField(Media)
lcrr = db.ReferenceField( 'LCRR', reverse_delete_rule=3, dbref=False, )
meta = {
'db_alias': config.get_config()['MONGODB_SETTINGS']['alias'],
'cascade':True
}
I have two collections ScenarioDrivers and ModelDrivers which has One to Many relationship with each other.
class ScenarioDrivers(Document):
meta = {
'collection': 'ScenarioDrivers'
}
ScenarioId = ReferenceField('ModelScenarios')
DriverId = ReferenceField('ModelDrivers')
DriverCalibrationMethod = StringField()
SegmentName = StringField()
DriverValue = ListField()
CalibrationStatus = StringField()
AdjustedValues = ListField(default=[])
CreateDate = DateTimeField(default=ObjectId().generation_time)
LastUpdateDate = DateTimeField(default=datetime.utcnow())
class ModelDrivers(Document):
meta = {
'collection': 'ModelDrivers'
}
PortfolioModelId = ReferenceField('PortfolioModels')
DriverName = StringField()
CreateDate = DateTimeField(default=ObjectId().generation_time)
LastUpdateDate = DateTimeField(default=datetime.utcnow())
FieldFormat = StringField()
DriverData = ListField()
My query is like this.
class GetCalibratedDrivers(Resource):
def get(self, scenario_id):
scenario_drivers_list = []
scenario_drivers = ScenarioDrivers.objects(ScenarioId=scenario_id).exclude('ScenarioId').select_related(1)
for scenario_driver in scenario_drivers:
scenario_driver_dict = {
'id': str(scenario_driver.id),
'DriverId': str(scenario_driver.DriverId.id),
'SegmentName': scenario_driver.SegmentName,
'CalibrationMethod': scenario_driver.DriverCalibrationMethod,
'CalibratedValues': exchange(scenario_driver.DriverValue),
'AdjustedValues': scenario_driver.AdjustedValues,
'LastUpdateDate': formatted_date(scenario_driver.LastUpdateDate),
'FieldFormat': scenario_driver.DriverId.FieldFormat
}
scenario_drivers_list.append(scenario_driver_dict)
return {
'DriverCalibrations': scenario_drivers_list
}
The Query matches 1140 records and then I construct a dictionary and make it a list.
But this API call takes 30s to process just 1140 records. Where I am missing? Please help. I am using latest version of Pymongo and MongoEngine.
I think the problem is not with your query, it is with you looping over 1140 records. I do not see any use of referenced objects so you should consider removing select_related(1). Once you do that, if you want to convert reference object ids to string, you can use as_pymongo() which will do that by default for you. And finally if you must read some data in specific format like formatted_date or exchange, it is better to save them as part of your document. i.e. save FormattedLastUpdateDate with LastUpdateDate. In MongoDB, you have to think about your read specific logic when you save the document.
I'm writing a python script to populate a mongodb database, my models look like the following:
from mongoengine import *
from mongoengine.django.auth import User
class TrackGroup (Document):
name = StringField(required=True)
users = ListField(ReferenceField('User'))
devices = ListField(ReferenceField('Device'))
class Device(Document):
name = StringField(max_length=50, required=True)
grp = ListField(ReferenceField(TrackGroup))
class User(User):
first_name = StringField(max_length=50)
last_name = StringField(max_length=50)
grp = ListField(ReferenceField(TrackGroup))
And my script goes like this:
#Create a group
gName = 'group name'
g = TrackGroup(name=gName)
g.users = []
g.devices = []
g.save()
#create a user
u = User.create_user(username='name', password='admin', email='mail#ex.com')
gRef = g
u.grp = [gRef, ]
u.first_name = 'first'
u.last_name = 'last'
u.save()
gRef.users.append(u)
gRef.save()
#create a device
dev = Device(name='name').save()
gRef = g
dev.grp = [gRef, ]
dev.save()
gRef.devices.append(dev)
gRef.save() #Problem happens here
The problem happens when I call gRef.save() I get the following error:
raise OperationError(message % unicode(err))
mongoengine.errors.OperationError: Could not save document (LEFT_SUBFIELD only supports Object: users.0 not: 7)
I looked around for a while, and here it says that this means I'm trying to set a filed with an empty key, like this: (The example is from the link above, not mine)
{
"_id" : ObjectId("4e52b5e08ead0e3853320000"),
"title" : "Something",
"url" : "http://www.website.org",
"" : "",
"tags" : [ "international"]
}
I don't know where such a field can come from, but I opened a mongo shell and looked at my documents from the three collections, and I couldn't find such a field.
Note: If I add the device first, the same error occurs while saving the group after adding the user.
I had the same error, and this trick work for me:
the_obj_causing_error.reload()
/* make some change */
the_obj_causing_error.price = 5
the_obj_causing_error.save()
just try reload() the object before changing it.