I am trying to write a simple python script to collect certain command outputs from mongodb, as part of validating the database before and after backup.
Below is the script.
import pymongo
import json
client = pymongo.MongoClient('localhost',username='admin',password='admin')
db = client['mydb']
collections = list(db.list_collection_names())
for i in collections:
print(db.$i.estimated_document_count())
All the collections are stored in the list called collections and I want to run it in for loop so that I can get document count in each collection. I know the last print statement here is wrong. How to get it right? I want $i to substitute the collection name during each iteration so that I can get the document count of that collection.
When I run "print(db.audit.estimated_document_count())" it gives me the document count in audit collection. But how to iterate through the list in for loop and substitute the value of i in the command?
Also, for validating backup/restore is there any other commands that I should run against database to verify backup/restore?
You cannot use a computed value as an identifier in at "dot" expression, at least not without resorting to dirty tricks.
What you can do, is to find some other mechanism for getting a collection given its name. According to the tutorial and API reference, you can use db['foo'] instead of db.foo.
Related
I would like to search my database for a value. I know you can do this for a key with db.search() but there does not seem to be any kind of similar function for searching for a value
I've tried using the contains() function but I have the same issue. It checks if the key is contained in the database. I would like to know if a certain value is contained in the database.
I would like to do something like this that would search for values in tinydb
db.search('value')
If I was able to execute the above command and get the value(if it does exist) or Nothing if it doesn't that would be ideal. Alternatively, if the able returned True or False accordingly, that would be fine as well
I don't know if this is what you are looking for but which the following command you can check for a specific field value:
from tinydb import Query
User = Query()
db.search(User.field_name == 'value')
I'm new here (doing some reading to see if TinyDB would even be applicable for my use case) so perhaps wrong, and also aware that this question is a little old. But I wonder if you can't address this by iterating over each field and searching within for your value. Then, you couldget the key or field wherein a value match was located.
I am having trouble with the parameter of an SNMP query in a python script. An SNMP query takes an OID as a parameter. The OID I use here is written in the code below and, if used alone in a query, should return a list of states for the interfaces of the IP addresses I am querying onto.
What I want is to use that OID with a variable appended to it in order to get a very precise information (if I use the OID alone I will only get a list of thing that would only complexify my problem).
The query goes like this:
oid = "1.3.6.1.4.1.2011.5.25.119.1.1.3.1.2."
variable = "84.79.84.79"
query = session.get(oid + variable)
Here, this query will return a corrupted SNMPObject, as in the process of configuration of the device I am querying on, another number is added, for some reason we do not really care about here, between these two elements of the parameter.
Below is a screenshot showing some examples of an SNMP request that only takes as a parameter the OID above, without the variable appended, on which you may see that my variable varies, and so does the highlighted additional number:
Basically what I am looking for here is the response, but unfortunately I cannot predict for each IP address I am querying what will that "random" number be.
I could use a loop that tries 20 or 50 queries and only saves the response of the only one that would have worked, but it's ugly. What would be better is some built-in function or library that would just say to the query:
"SNMP query on that OID, with any integer appended to it, and with my variable appended to that".
I definitely don't want to generate a random int, as it is already generated in the configuration of the device I am querying, I just want to avoid looping just to get a proper response to a precise query.
I hope that was clear enough.
Something like this should work:
from random import randint
variable = "84.79.84.79"
numbers = "1.3.6.1.4.1.2011.5.25.119.1.1.3.1.2"
query = session.get('.'.join([numbers, str(randint(1,100)), variable])
What is the proper way of moving a number of records from one collection to another. I have come across several other SO posts such as this which deal with achieve the same goal but none have a python implementation.
#Taking a number of records from one database and returning the cursor
cursor_excess_new = db.test_collection_new.find().sort([("_id", 1)]).limit(excess_num)
# db.test.insert_many(doc for doc in cursor_excess_new).inserted_ids
# Iterating cursor and Trying to write to another database
# for doc in cursor_excess_new:
# db.test_collection_old.insert_one(doc)
result = db.test_collection_old.bulk_write([
for doc in cursor_excess_new:
InsertMany(doc for each doc in cur)
pprint(doc)
])
If I use insert_many, I get the following error: pymongo.errors.OperationFailure: Writes to config servers must have batch size of 1, found 10
bulk_write is giving me a syntax error at the start of for loop.
What is the best practice and correct way of transferring records from one collection to another in pymongo so that it is atomic?
Collection.bulk_write accepts as argument an iterable of query operations.
pymongo has pymongo.operations.InsertOne operation not InsertMany.
For your situation, you can build a list of InsertOne operations for each document in the source collection. Then do a bulk_write on the destination using the built-up list of operations.
from pymongo import InsertOne
...
cursor_excess_new = (
db.test_collection_new
.find()
.sort([("_id", 1)])
.limit(excess_num)
)
queries = [InsertOne(doc) for doc in cursor_excess_new]
db.test_collection_old.bulk_write(queries)
You don't need "for loop".
myList=list(collection1.find({}))
collection2.insert_many(myList)
collection1.delete_many({})
If you need to filter it you can use the following code:
myList=list(collection1.find({'status':10}))
collection2.insert_many(myList)
collection1.delete_many({'status':10})
But be careful, because it has no warranty to move successfully so you need to control transactions. If you're going to use the following code you should consider that your MongoDb shouldn't be Standalone and you need to active Replication and have another instance.
with myClient.start_session() as mySession:
with mySession.start_transaction():
...yourcode...
Finally, the above code has the warranty to move (insert and delete) successfully but the transaction isn't in your hand and you can't get the result of this transaction so you can use the following code to control both moving and transaction:
with myClient.start_session() as mySession:
mySession.start_transaction()
try:
...yourcode...
mySession.commit_transaction()
print("Done")
except Exception as e:
mySession.abort_transaction()
print("Failed",e)
In a Django project, I'm using redis as a fast backend.
I can LPUSH multiple values in a redis list like so:
lpush(list_name,"1","2","3")
However, I can't do it if I try
values_list = ["1","2","3"]
lpush(list_name,values_list)
For the record, this doesn't return an error. Instead it creates a list list_name with a single value. E.g. ['["1","2","3"]']. This is not usable if later one does AnObject.objects.filter(id__in=values_list). Nor does it work if one does AnObject.objects.filter(id__in=values_list[0]) (error: invalid literal for int() with base 10: '[').
What's the best way to LPUSH numerous values in bulk into a redis list (python example is preferred).
lpush (list_name, *values_list)
This will unpack contents in value_list as parameters.
If you have enormous numbers of values to insert into db, you can pipeline them. Or you can use the command line tool redis-cli --pipe to populate a database
Using Python with SPARQLWrapper, JSON, urlib2 & cgi. Had trouble passing a working SPARQL query with some NULL values to python so I populated the blanks with a literal and will try to filter at the output. I have this results section example:
for result in results["results"]["bindings"]:
project = result["project"]["value"].encode('utf-8')
filename = result["filename"]["value"].encode('utf-8')
url = result["url"]["value"].encode('utf-8')
...and I print the %s. Is there a way to filter a value, i.e., IF VALUE NE "string" then PRINT? Or is there another workaround? I'm at the tail-end of a small project, I know I need a better wrapper, I just need to get these results filtered before I can move on. T very much IA...
I'm one of the developers of the SPARQLWrapper library, and the question had been already answered at the mailing list.
Regarding optionals values on the original query, the result set will come with no values for those variables. The problems is that we'd need to parse the query to populate such missing entries, and we want to avoid such parsing; therefore you'd need to check it for avoiding runtime problems with KeyError.
Usually I use a code like:
for result in results["results"]["bindings"]:
party = result["party"]["value"] if ("party" in result) else None