I am using the python package pymongo to retrieve data from a mongodb database.
>>> r = collection.find() # returns an object of class 'Cursor'
Then I convert to a list
>>> l = list(r) # returns a 'list' of 'dict'
here is what print(l) returns:
>>> [{u'date': datetime.datetime(2009, 11, 10, 10, 45), u'_id': 1, u'name': u'name1', u'value': 11},{u'date': datetime.datetime(2013, 11, 10, 10, 45), u'_id': 2, u'name': u'name2', u'value': 22}]
Now I need to convert to JSON so that I can manipulate it.
>>> json.dumps(l)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 178, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2009, 11, 12, 11, 14) is not JSON serializable
I have also tried to follow http://api.mongodb.org/python/1.7/api/pymongo/json_util.html without success:
Edit: the recent version of the link is http://api.mongodb.org/python/current/api/bson/json_util.html
>>> json.dumps(l, default=json_util.default)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'json_util' is not defined
Note: precisely I need to push this result to R using the R package rPython and its function rPython::python.get("l")
Side Question: What is the u (u'Date', u'name', etc..) before each field in the list of dict?
The pymongo documentation you pointed is obsolete. If you're using version 1.7 I recommend updating. With a more recent version you can do this:
from bson.json_util import dumps
dumps(l)
https://pymongo.readthedocs.io/en/stable/api/bson/json_util.html
Side answer: u'name', u'date', u'_id' etc are the names of the fields of the document on the database.
from bson import json_util
json.dumps(result,default=json_util.default)
In my situation, this error is due to mongo DB id object in flask
all you have to do is convert id (NOTE: if you need id convert it else you can pop it too)
I'm sharing my solution which I figured out hope this helps someone
from flask import jsonify
def get_data(self,data):
data['_id'] = str(data['_id'])
return data
app = Flask(__name__)
#app.route('/')
def apimethod():
temp = [self.get_data(i) for i in self.collection.find()]
return jsonify(temp)
also dumps from pymongo don't help alot
from bson.json_util import dumps,loads
because it is returning a string instead of dict which was expected in my situation to create API and I have to load again if I did dumps.
This thread helped me - thank you.
Wanted to share my final solution to get the JSON back into a JSON/Dictionary Object: (Based on your example)
from bson.json_util import dumps, loads
r = collection.find()
l = list(r) # Converts object to list
d = dumps(l) # Converts to String
dict_needed = loads(d[0]) # Serializes String and creates dictionary
Now you have the JSON in a dictionary object and can edit as needed.
I was facing the same issue, I wrote a code that converts document to dictionary. You can use that for reference. Pass the object obtained by find_one() into documentToJson() method and the results of find() into convertDocumentsToJson. There is type in the name Json, instead the code converts to Dict rather than json.
from bson.json_util import dumps
class UtilService:
def __init__(self):
pass
#staticmethod
def pinCodeParser(path):
location = {}
f = open(path)
for line in f:
words = line.split()
location[words[1]] = (words[-3],words[-2])
return location
#staticmethod
def listHelper(str):
s = []
str = str.split(',')
for e in str:
s.append(e.replace("[","").replace("]",""))
return s
#staticmethod
def parseList(str):
if ',' in str:
return UtilService.listHelper(str)
return str
#staticmethod
def trimStr(str):
return str.replace('"','')
#staticmethod
def documentToJson(document):
document = eval(dumps(document))
mp = {}
for key, value in document.iteritems():
if "_id" in key:
mp["id"] = str(value["$oid"])
else:
mp[ UtilService.trimStr(key) ] = UtilService.parseList( value )
return mp
#staticmethod
def convertDocumentsToJson(documents):
result = []
for document in documents:
result.append(UtilService.documentToJson(document))
return result
Related
I am trying to implement ScroogeCoin using the fastecdsa library. I currently run into an error that happens when my create_coins function is called. The error points to the signing function (tx["signature"]) and says that it cannot convert an integer type to a byte.
import hashlib
import json
from fastecdsa import keys, curve, ecdsa
class ScroogeCoin(object):
def __init__(self):
self.private_key, self.public_key = keys.gen_keypair(curve.secp256k1)
self.address = hashlib.sha256(json.dumps(self.public_key.x).encode()).hexdigest()
self.chain = []
self.current_transactions = []
def create_coins(self, receivers: dict):
"""
Scrooge adds value to some coins
:param receivers: {account:amount, account:amount, ...}
"""
tx = {
"sender" : self.address, # address,
# coins that are created do not come from anywhere
"location": {"block": -1, "tx": -1},
"receivers" : receivers,
}
tx["hash"] = hashlib.sha256(json.dumps(tx).encode()).hexdigest()# hash of tx
tx["signature"] = ecdsa.sign(self.private_key, tx["hash"])# signed hash of tx
self.current_transactions.append(tx)
...
When this function is ran in the main function:
...
Scrooge = ScroogeCoin()
users = [User(Scrooge) for i in range(10)]
Scrooge.create_coins({users[0].address:10, users[1].address:20, users[3].address:50})
...
It produces this error:
Traceback (most recent call last):
File "D:\Scrooge_coin_assignmnet.py", line 216, in <module>
main()
File "D:\Scrooge_coin_assignmnet.py", line 197, in main
Scrooge.create_coins({users[0].address:10, users[1].address:20, users[3].address:50})
File "D:\Scrooge_coin_assignmnet.py", line 27, in create_coins
tx["signature"] = ecdsa.sign(self.private_key, tx["hash"])# signed hash of tx
File "C:\Users\d\AppData\Local\Programs\Python\Python311\Lib\site-packages\fastecdsa\ecdsa.py", line 36, in sign
rfc6979 = RFC6979(msg, d, curve.q, hashfunc, prehashed=prehashed)
File "C:\Users\d\AppData\Local\Programs\Python\Python311\Lib\site-packages\fastecdsa\util.py", line 25, in __init__
self.msg = msg_bytes(msg)
File "C:\Users\d\AppData\Local\Programs\Python\Python311\Lib\site-packages\fastecdsa\util.py", line 153, in msg_bytes
raise ValueError('Msg "{}" of type {} cannot be converted to bytes'.format(
ValueError: Msg "21783419755125685845542189331366569080312572314742637241373298325693730090205" of type <class 'int'> cannot be converted to bytes
I've tried to play around and change it to a byte by using encode on the tx["hash"] as well things like bytes.fromhex() but it still gives the same error. I wanted to ask others who are more skilled and see if they can see how I am messing up.
I am parsing dictionaries into nested Json (Python 2.7). The problem I have that sometimes there is no value for the variable so it takes None, it depends if the variable came or not. It would be ok, but I want to avoid escape characters before quotations. Before I get nested Json I do json.loads() to remove string escapes.
The problem is to return None if there is no dict.
Example code:
import json
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': json.loads(data1),
'data2': json.loads(data2)}
output = json.dumps(sensor)
print(output)
Expected outcome:
{"data2": null, "data1": {"foo": "bar"}, "header": "Info"}
Received error:
Traceback (most recent call last):
File "\jsonDecoding.py", line 7, in <module>
'data2': json.loads(data2)}
File "\Python\Python35\lib\json\__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'NoneType'
Tried solution:
class Decoder(json.JSONDecoder):
def default(self, obj):
if obj is None::
return None
else: return super(Decoder, self).default(obj)
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': json.loads(data1),
'data2': json.loads(data2, cls=Decoder)}
output = json.dumps(sensor)
I thought implementing Decoder.default() should solve the problem but it calls the class but does not call the default method.
There are plenty of talks about None in place of key or value but I did not find in place of whole object
I think you make the problem harder than it is. We can construct a None-safe function:
def nonesafe_loads(obj):
if obj is not None:
return json.loads(obj)
This will return the json.loads(obj) in case obj is not None, and otherwise it will return None. Or we can construct a "best effort" JSON loads, that aims to json.loads the string, or in case that fails, returns the obj itself:
def besteffort_loads(obj):
try:
return json.loads(obj)
except (TypeError, ValueError):
return obj
Now we can just use our nonesafe_loads in the program:
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': nonesafe_loads(data1),
'data2': nonesafe_loads(data2)}
output = json.dumps(sensor)
I think the query is correct but still an error.
findQ = {"fromid": wordid}, {"toid":1}
res= self.db.wordhidden.find(findQ)
However, find_one(findQ) works. So I can't find the wrong thing.
I use python 3.6 and pymongo.
Here is my code:
def getallhiddenids(self,wordids,urlids):
l1={}
for wordid in wordids:
findQ = {"fromid": wordid}, {"toid":1}
res= self.db.wordhidden.find(findQ)
for row in res: l1[row[0]]=1
for urlid in urlids:
findQ = {"toid": urlid}, {"fromid":1}
res= self.db.hiddenurl.find(findQ)
This is an error:
Traceback (most recent call last):
File "C:\Users\green\Desktop\example.py", line 9, in <module>
neuralnet.trainquery([online], possible, notspam)
File "C:\Users\green\Desktop\nn.py", line 177, in trainquery
self.setupnetwork(wordids,urlids)
File "C:\Users\green\Desktop\nn.py", line 105, in setupnetwork
self.hiddenids=self.getallhiddenids(wordids,urlids)
File "C:\Users\green\Desktop\nn.py", line 93, in getallhiddenids
res= self.db.wordhidden.find(findQ)
File "C:\Users\green\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pymongo\collection.py", line 1279, in find
return Cursor(self, *args, **kwargs)
File "C:\Users\green\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pymongo\cursor.py", line 128, in __init__
validate_is_mapping("filter", spec)
File "C:\Users\green\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\pymongo\common.py", line 400, in validate_is_mapping
"collections.Mapping" % (option,))
TypeError: filter must be an instance of dict, bson.son.SON, or other type
that inherits from collections.Mapping
find_one(findQ) works
The error is because PyMongo find() requires a dictionary or a bson.son object. What you have passed in is a Python tuple object is the form of ({"fromid": wordid}, {"toid":1}). You could correct this by invoking the find() method as below:
db.wordhidden.find({"fromid": wordid}, {"toid": 1})
Technically your invocation of find_one() does not work either. It just that the parameter filter has been altered by find_one(). see find_one() L1006-1008. Which basically format your tuple filter into :
{'_id': ({"fromid": wordid}, {"toid":1}) }
The above would (should) not returned any matches in your collection.
Alternative to what you're doing, you could store the filter parameter into two variables, for example:
filterQ = {"fromid": wordid}
projectionQ = {"toid": 1}
cursor = db.wordhidden.find(filterQ, projectionQ)
I was trying to migrate data from SQL Server to MongoDB but was getting below type error in the last phase while importing data to MongoDB.
mongoImp = dbo.insert_many(jArray)
File "/home/lrsa/.local/lib/python2.7/site-packages/pymongo/collection.py", line 710, in insert_many
blk.ops = [doc for doc in gen()]
File "/home/lrsa/.local/lib/python2.7/site-packages/pymongo/collection.py", line 702, in gen
common.validate_is_document_type("document", document)
File "/home/lrsa/.local/lib/python2.7/site-packages/pymongo/common.py", line 407, in validate_is_document_type
"collections.MutableMapping" % (option,))
TypeError: document must be an instance of dict, bson.son.SON, bson.raw_bson.RawBSONDocument, or a type that inherits from collections.MutableMapping
I have also checked the type(jArray) which is a str. Tried with converting the data type to list as well but could not succeed.
My Code:
import pyodbc
import json
import collections
import pymongo
from bson import json_util
odbcArray = []
mongoConStr = '192.168.10.107:36006'
sqlConStr = 'DRIVER={MSSQL-NC1311};SERVER=tcp:192.168.10.103,57967;DATABASE=AdventureWorks;UID=testuser;PWD=testuser'
mongoConnect = pymongo.MongoClient(mongoConStr)
sqlConnect = pyodbc.connect(sqlConStr)
dbo = mongoConnect.eaedw.sqlData
dbDocs = dbo.find()
sqlCur = sqlConnect.cursor()
sqlCur.execute("""
SELECT TOP 2 BusinessEntityID,Title, Demographics, rowguid, ModifiedDate
FROM Person.Person
""")
tuples = sqlCur.fetchall()
for tuple in tuples:
doc = collections.OrderedDict()
doc['id'] = tuple.BusinessEntityID
doc['title'] = tuple.Title
doc['dgrap'] = tuple.Demographics
doc['rowi'] = tuple.rowguid
doc['mtime'] = tuple.ModifiedDate
odbcArray.append(doc)
jArray = json.dumps(odbcArray, default=json_util.default)
mongoImp = dbo.insert_many(jArray)
mongoConnect.close()
sqlConnect.close()
Check out this bulk insert example from MongoDB:s webpage. Skip the json.dumps call (which turns your array of documents into a json formatted string) and insert odbcArray directly:
mongoImp = dbo.insert_many(odbcArray)
I have the following code:
import json
src_vol1 = {'provider_id':'src1'}
src_vol2 = {'provider_id':'src2'}
get_snapshot_params = lambda src_volume, trg_volume: {
'volumeId': src_volume['provider_id'],
'snapshotName': trg_volume['id']}
trg_vol1 = {'id':'trg1'}
trg_vol2 = {'id':'trg2'}
src_vols = [src_vol1, src_vol2]
trg_vols = [trg_vol1, trg_vol2]
snapshotDefs = map(get_snapshot_params , src_vols, trg_vols)
params = {'snapshotDefs': snapshotDefs}
json.dumps(params)
I need it work on both Python3 and Python2.7, but on Python3 I get
Traceback (most recent call last):
File "./prog.py", line 16, in <module>
File "/usr/lib/python3.4/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.4/json/encoder.py", line 192, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.4/json/encoder.py", line 250, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.4/json/encoder.py", line 173, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <map object at 0xb72a1a0c> is not JSON serializable
I tried to put dict() around the params but it didn't work.
What is the difference? I didn't find anything in the documentation.
map behaves differently between python2 and 3.
To reproduce the python2 behavior, replace map(...) with list(map(...)).
This still works in python2, but in python2 it makes a pointless extra copy of the list returned by map, which can consume more memory and run slower.
To avoid it, you can try something like:
try:
from itertools import imap as map # py2
except ImportError:
pass # py3, map is already defined apropriately
Or you can also check for system version and then re-define map into map_ based on system version:
import sys
ver = sys.version[:3]
if ver < '3': #Python 2
map_ = map #use same map method
elif ver >= '3': #Python 3
map_ = lambda f,x : list(map(f,x))
snapshotDefs = map_(get_snapshot_params , src_vols, trg_vols)