I need test the accuracy of a server mongodb.
I am trying to insert a sequence of data, take the moment and it was sent to the database to know when it was inserted. I'm trying this:
#!/usr/bin/python
from pymongo import Connection
from datetime import date, timedelta, datetime
class FilterData:
#classmethod
def setData(self, serialData):
try:
con = Connection('IP_REMOTE', 27017, safe=True)
db = con['resposta']
inoshare = db.resposta
inoshare.insert(serialData)
con.close()
except Exception as e:
print "Erro no filter data: ", e.message, e.args
obj = FilterData()
inicio = datetime.now()
termino = inicio + timedelta(seconds=10)
contador = 1
while inicio <= termino:
print contador, inicio.strftime('%d-%m-%Y %H:%M:%S')
pacote = {'contador':contador, 'datahora':$currentDate()}
obj.setData(pacote)
contador += 1
But the variables of mongodb (using $) are not recognized in python. How to proceed to accomplish this integration?
Obs: IP_REMOTE = my valid IP on REMOTE server
then tried the following, but only inserts a single record.
#!/usr/bin/python
from pymongo import Connection
from datetime import date, timedelta, datetime
import time
class FilterData:
def __init__(self):
self.con = Connection('54.68.148.224', 27017, safe=True)
self.db = self.con['resposta']
self.inoshare = self.db.resposta
def setData(self, serialData):
try:
self.inoshare.update({}, serialData, upsert=True)
except Exception as e:
print "Erro no filter data: ", e.message, e.args
def desconect(self):
self.con.close()
obj = FilterData()
inicio = datetime.now()
termino = inicio + timedelta(seconds=30)
while inicio <= termino:
print inicio.strftime('%d-%m-%Y %H:%M:%S')
pacote = {'$currentDate': {'datahora': { '$type': 'date' }}}
obj.setData(pacote)
inicio = datetime.now()
time.sleep(1)
obj.desconect()
Operator expressions in MongoDB are represented in the data structure as a string. These are also "update operators", so $currentDate is meant to be used in the "update object" portion of an .update() method.
So something like this to insert a new record with the "$currentDate" from the server:
db = con['resposta']
inoshare = db.resposta
inoshare.update({}, {
'$currentDate': {
'datahora': { '$type': 'date' }
}
},upsert=True)
Presuming of course there is nothing in your collection. Otherwise make sure the "query" portion of the .update() statement does not match a document when you want to "insert"/"upsert" as it were.
All the documentation options in the MongoDB manual pages are as JSON notation relevant to the MongoDB shell, but however this is not that different from the notation of many dyamically typed languages such as python, ruby and Perl.
BTW. Unless you are really testing in distinct scripts, then do not make a connection and disconnect before and after every operation. Database collections should stay open for the life-cycle of your application.
You should pass the python code to mongo like this,
>>> from datetime import datetime
>>> datetime.now()
Your code:
pacote = {'contador':contador, 'datahora':datetime.now()}
Thanks to everyone who helped me. I understand now, first do an insert and then an update. Like this:
class FilterData:
def __init__(self):
self.con = Connection('IP_REMOTE', 27017, safe=True)
self.db = self.con['resposta']
self.inoshare = self.db.resposta
self.contador = 1
def setData(self, serialData):
try:
self.inoshare.insert({'contador': self.contador}, serialData, upsert=True)
print self.contador, datetime.now().strftime('%d-%m-%Y %H:%M:%S.%f')
self.inoshare.update({'contador': self.contador}, serialData, upsert=True)
self.contador += 1
except Exception as e:
print "Erro no filter data: ", e.message, e.args
def desconect(self):
self.con.close()
that way I can check the time that the query was sent and the moment she was executed on the remote server.
On-site host I have the following output, for example:
1 08-11-2014 15:37:45.079000
1 08-11-2014 15:38:04.039000
2 08-11-2014 15:38:05.410000
3 08-11-2014 15:38:06.785000
4 08-11-2014 15:38:08.153000
5 08-11-2014 15:38:09.522000
6 08-11-2014 15:38:10.886000
7 08-11-2014 15:38:12.243000
8 08-11-2014 15:38:13.609000
And on the remote server I get the following output:
{"contador" : 1, "datahora" : ISODate("2014-11-08T18:38:05.323Z") }
{"contador" : 2, "datahora" : ISODate("2014-11-08T18:38:06.687Z") }
{"contador" : 3, "datahora" : ISODate("2014-11-08T18:38:08.060Z") }
{"contador" : 4, "datahora" : ISODate("2014-11-08T18:38:09.429Z") }
{"contador" : 5, "datahora" : ISODate("2014-11-08T18:38:10.796Z") }
{"contador" : 6, "datahora" : ISODate("2014-11-08T18:38:12.162Z") }
{"contador" : 7, "datahora" : ISODate("2014-11-08T18:38:13.527Z") }
{"contador" : 8, "datahora" : ISODate("2014-11-08T18:38:14.893Z") }
That way I can identify the time difference between the time of the update and the moment he really was iserido in the database. Note: The clocks are synchronized.
Related
(Pymongo) several functions cover with Commit and rollback, and one of them stll proceed while whole function should stop
I read the manual, that all the example commit and rollback only cover two operation, is that the limit?, usually should contain 3 or more operations and either operate in the same time or not operate if error https://pymongo.readthedocs.io/en/stable/api/pymongo/client_session.html
I tried to contain 3 operation inside commit and rollback but
mycol_two.insert_one() didn't stop proceed like other function when error occur
brief description:
I have three collections in same DB
collection "10_20_cash_all"
collection "10_20_cash_log"
collection "10_20_cash_info"
commit and rollback on line 39 to 44
line 42 print( 3/0 ) , I intent to make an error, expect all function would stop proceed
import pymongo
import datetime
import json
from bson.objectid import ObjectId
from bson import json_util
import re
myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["(practice_10_14)-0004444"]
mycol_one = mydb["10_20_cash_all"]
mycol_two = mydb["10_20_cash_log"]
mycol_3rd = mydb["10_20_cash_info"]
# already store 100$ in bank
# doc_two = {"ID" : 100998 , "Cash_log$" : 5 } # withdraw 70$ from bank
doc_two = input("Enter ID and log amount$: ")
doc_3rd = input("Enter extra info: ")
doc_two_dic = json.loads(doc_two)
doc_3rd_dic = json.loads(doc_3rd)
# doc_3rd = {"note" : "today is good" }
ID_input = doc_two_dic['ID']
print("ur id is :" + str(ID_input))
doc_one = {"ID" : ID_input}
with myclient.start_session() as s:
cash_all_result = mycol_one.find_one(doc_one, session=s)
def cb(s):
try:
while True:
cash_all_result = mycol_one.find_one(doc_one, session=s)
mycol_two.insert_one(doc_two_dic, session=s)
print( 3/0 )
mycol_3rd.insert_one(doc_3rd_dic, session=s)
print( "now total is :" + str(cash_all_result['Cash_$']) )
Cash_total_int = int(cash_all_result['Cash_$'])
log_int = int(doc_two_dic['Cash_log$'])
if Cash_total_int < log_int:
print("error: withdraw is over ur balance")
break
new_Cash_total = Cash_total_int - log_int
print("now total is :" + str(new_Cash_total))
newvalues_json = { "$set" : {"Cash_$" : new_Cash_total } }
mycol_one.update_one(doc_one , newvalues_json, session=s)
fail_condition_json = {"ok" : 1 , "fail reason" : "no error "}
print(fail_condition_json)
return fail_condition_json
except Exception as e:
fail_condition_json = {"ok" : 0 , "fail reason" : "error raise on start_session()"}
print(fail_condition_json)
return fail_condition_json
s.with_transaction(cb)
command prompt:
Enter ID and log amount$: {"ID" : 100998 , "Cash_log$" : 5 }
Enter extra info: {"note" : "today is good" }
ur id is :100998
{'ok': 0, 'fail reason': 'error raise on start_session()'}
the "10_20_cash_log" still store new value which shoud empty/not run like '"10_20_cash_info"' is empty
{
"_id" : ObjectId("635262e502725626c39cbe9e"),
"ID" : 100998,
"Cash_log$" : 5
}
Here is the script which i have working inside MongoDBs Compass and working inside Studio 3T
db.getCollection("collection").find({
geometry: {
$geoIntersects: {
$geometry: {
type: "Polygon" ,
coordinates: [ [
[-2.4478329206542915, 52.679303638992494], [-2.4423397565917915, 52.677534343091544], [-2.445601322753901, 52.67430779548455], [-2.4509228254394477, 52.676129262942176], [-2.4478329206542915, 52.679303638992494]
] ]
}
}
}
})
and it returns the following results:
As you can see the code seems to work fine, but when I try to run this inside a Python script it keeps failing and I need a little help pointing out the seemingly not so obvious error.
I have changed the fin statement to a simple field find and it works, just not when its a geoIntersects find statement.
However, if I try to do this inside python I keep getting errors.
Please see below for a copy of my python script, sorry for the redacted comments but need to keep data secure.
import pymongo
#mongodb service vars
dbStartc = 'mongodb+srv://'
dbDomain = '#cluster0.o7cud.azure.mongodb.net'
dbUser = 'redacted'
dbPass = 'redacted'
dbName = 'redacted'
dbColl = 'redacted'
dbSettings = '/?retryWrites=true&w=majority'
#test vars
dbName_connected = False
dbColl_connected = False
try:
#conect to the mongodb instance
mongoURL = dbStartc + dbUser + ':' + dbPass + dbDomain + dbSettings
mongocon = pymongo.MongoClient(mongoURL)
#connect to the database
dblist = mongocon.list_database_names()
if dbName in dblist:
mongocdb = mongocon[dbName]
#print("MongoDB Service - Database connected")
dbName_connected = True
#connect to the collection we need
collist = mongocdb.list_collection_names()
if dbColl in collist:
mongocol = mongocdb[dbColl]
#print("MongoDB Service - Collection connected")
dbColl_connected = True
#pre checks test
if dbName_connected and dbColl_connected :
#print("MongoDB Service - Connection Checks Complete")
find_result = []
found_count = 0
found_count = mongocol.count_documents( )
if found_count > 0 :
print("Collection Document Count: " + found_count)
mydoc = mongocol.find({ geometry: { $geoIntersects: { $geometry: { type: "Polygon" , coordinates: [ [ [-2.44783, 52.67930], [-2.44233, 52.67753], [-2.44560, 52.67430], [-2.45092, 52.67612], [-2.44783, 52.67930] ] ] } } } })
#for x in
for x in mydoc:
find_result += [x]
print(find_result)
else :
print("MongoDB Service - Connection Checks Failed")
except Exception as ex:
print ("Something you were not expecting went wrong! (" + ex + ")")
Here is the error:
Any help in getting python to work would be greatly appreciated
I am relatively new to mongoDb in python, so kindly help
I have created a collection called waste:
class Waste(Document):
meta = {'collection': 'Waste'}
item_id = IntField(required=True)
date_time_record = DateTimeField(default=datetime.utcnow)
waste_id = IntField(unique=True, required=True)
weight = FloatField(required= True)
I want to do a range query for a given start and end date:
I have tried the following query:
start = datetime(start_year, start_month, start_day)
end = datetime(end_year, end_month, end_day)
kwargs['date_time_record'] = {'$lte': end, '$gte': start}
reports = Waste.objects(**kwargs).get()
But I keep getting the error: DoesNotExist: Waste matching query does not exist.
the date value being sent as:
{
"start_year": 2020,
"start_month" : 5,
"start_day" : 10,
"end_year": 2020,
"end_month" : 5,
"end_day" : 20
}
when I try to get the first object from the collection, the output in json is:
{"_id": {"$oid": "5ebbcf126fdbb9db9f74d24a"}, "item_id": 96387295, "date_time_record": {"$date": 1589366546870}, "waste_id": 24764942, "weight": 32546.0}
a $date is added and I am unable to decipher the numbers in the date field. But when I look at the data using the mongo compass it looks just fine:
There exist a record in the given date range so I am unable to understand where am I going wrong.
I got this working by using Q:
the query I used is
reports = Waste.objects((Q(date_time_record__gte=start) & Q(date_time_record__lte=end)))
The response is:
[{"_id": {"$oid": "5ebbcf126fdbb9db9f74d24a"}, "item_id": 96387295, "date_time_record": {"$date": 1589366546870}, "waste_id": 24764942, "weight": 32546.0}]
import pymongo
uri = 'mongodb://127.0.0.1:27017'
client = pymongo.MongoClient(uri)
db = client.TeamCity
students = db.students.find({})
for student in students:
print (student)
Python Result:
Blank
MongoDB: Results
db.students.find({})
{ "_id" : ObjectId("5788483d0e5b9ea516d4b66c"), "name" : "Jose", "mark" : 99 }
{ "_id" : ObjectId("57884cb3f7edc1fd01c3511e"), "name" : "Jordan", "mark" : 100
}
import pymongo
uri = 'mongodb://127.0.0.1:27017'
client = pymongo.MongoClient(uri)
db = client.TeamCity
students = db.students.find({})
print (students.count())
Python Result:
0
mongoDB Results
db.students.find({}).count()
2
What am I missing?
For
import pymongo
uri = 'mongodb://127.0.0.1:27017'
client = pymongo.MongoClient(uri)
db = client.TeamCity
students = db.students.find({})
print (students)
Python Result :
So I think it is able to connect to the db successfully but not returning results
Try your pymongo code like so, i.e. changing TeamCity to Teamcity
Print all students:
import pymongo
uri = 'mongodb://127.0.0.1:27017'
client = pymongo.MongoClient(uri)
db = client.Teamcity
students = db.students.find({})
for student in students:
print (student)
Count all students:
import pymongo
uri = 'mongodb://127.0.0.1:27017'
client = pymongo.MongoClient(uri)
db = client.Teamcity
students = db.students.find({})
print (students.count())
I know this question has been answered long ago, but I ran into the same kind of problem today and it happened to have a different reason, so I'm adding an answer here.
Code working on shell:
> db.customers.find({"cust_id": 2345}, {"pending_questions": 1, _id: 0})
{ "pending_questions" : [ 1, 5, 47, 89 ] }
Code not working in PyMongo (cust_id set through a web form):
db.customers.find({"cust_id": cust_id}, {"pending_questions": 1, "_id": 0})
It turned out that the numbers in the shell are being interpreted as ints, whereas the numbers used in Python code are being interpreted by PyMongo as floats, and hence return no matches. This proves the point:
cust_id = int(request.args.get('cust_id'))
db.customers.find({"cust_id": cust_id}, {"pending_questions": 1, "_id": 0})
which produces the result:
[1.0, 5.0, 47.0, 89.0]
The simple solution was to typecast everything to int in the python code. In conclusion, the data type inferred by the shell may be different from the data type inferred by PyMongo and this may be one reason that a find query that returns results on the shell doesn't return anything when run in PyMongo.
I have a pymongo collection in the form of:
{
"_id" : "R_123456789",
"supplier_ids" : [
{
"id" : "S_987654321",
"file_version" : ISODate("2016-03-15T00:00:00Z"),
"latest" : false
},
{
"id" : "S_101010101",
"file_version" : ISODate("2016-03-29T00:00:00Z"),
"latest" : true
}
]
}
when I get new supplier data, if the supplier ID has changed, I want to capture that by setting latest on the previous 'latest' to False and the $push the new record.
$set is not working as I am trying to employ it (commented code after 'else'):
import pymongo
from dateutil.parser import parse
new_id = 'S_323232323'
new_date = parse('20160331')
with pymongo.MongoClient() as client:
db = client.transactions
collection_ids = db.ids
try:
collection_ids.insert_one({"_id": "R_123456789",
"supplier_ids": ({"id": "S_987654321",
"file_version": parse('20160315'),
"latest": False},
{"id": "S_101010101",
"file_version": parse('20160329'),
"latest": True})})
except pymongo.errors.DuplicateKeyError:
print('record already exists')
record = collection_ids.find_one({'_id':'R_123456789'})
for supplier_id in record['supplier_ids']:
print(supplier_id)
if supplier_id['latest']:
print(supplier_id['id'], 'is the latest')
if supplier_id['id'] == new_id:
print(new_id, ' is already the latest version')
else:
# print('setting', supplier_id['id'], 'latest flag to False')
# <<< THIS FAILS >>>
# collection_ids.update_one({'_id':record['_id']},
# {'$set':{'supplier_ids.latest':False}})
print('appending', new_id)
data_to_append = {"id" : new_id,
"file_version": new_date,
"latest": True}
collection_ids.update_one({'_id':record['_id']},
{'$push':{'supplier_ids':data_to_append}})
any and all help is much appreciated.
This whole process seems unnaturally verbose - should I be using a more streamlined approach?
Thanks!
You can try with positional operators.
collection_ids.update_one(
{'_id':record['_id'], "supplier_ids.latest": true},
{'$set':{'supplier_ids.$.latest': false}}
)
This query will update supplier_ids.latest = false, if it's true in document and matches other conditions.
The catch is you have to include field array as part of condition too.
For more information see Update