First, a simple question on terms,
Encoding(json.dumps) means, converting something to json string,
decoding(json.loads) means, converting json string to json type(?)
I have a list of objects which I got from
>>> album_image_list = AlbumImage.objects.all().values(*fields)[offset:count]
>>> json.dumps(album_image_list[0], cls=DjangoJSONEncoder)
'{"album": 4, "album__title": "g jfd", "created_at": "2012-08-18T02:23:49Z", "height": 1024.0, "width": 512.0, "url_image": "http://--:8000/media/101ac908-df50-42cc-af6f-b172c8829a31.jpg"}'
but when I try the same on whole list (album_image_list),it fails...
>>> json.dumps(album_image_list, cls=DjangoJSONEncoder)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.6/json/__init__.py", line 237, in dumps
**kw).encode(obj)
File "/usr/lib/python2.6/json/encoder.py", line 367, in encode
chunks = list(self.iterencode(o))
File "/usr/lib/python2.6/json/encoder.py", line 317, in _iterencode
for chunk in self._iterencode_default(o, markers):
File "/usr/lib/python2.6/json/encoder.py", line 323, in _iterencode_default
newobj = self.default(o)
File "/home/--/virtualenvs/aLittleArtist/lib/python2.6/site-packages/django/core/serializers/json.py", line 75, in default
return super(DjangoJSONEncoder, self).default(o)
File "/usr/lib/python2.6/json/encoder.py", line 344, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: [{'album': 4L, 'album__title': u'g jfd', 'created_at': datetime.datetime(2012, 8, 18, 2, 23, 49, tzinfo=<UTC>), 'height': 1024.0, 'width': 512.0, 'url_image': u'http://--:8000/media/101ac908-df50-42cc-af6f-b172c8829a31.jpg'}, {'album': 4L, 'album__title': u'g jfd', 'created_at': datetime.datetime(2012, 8, 18, 1, 54, 51, tzinfo=<UTC>), 'height': 512.0, 'width': 512.0, 'url_image': u'http://--:8000/media/e85d1cf7-bfd8-4e77-b90f-d1ee01c67392.jpg'}] is not JSON serializable
>>>
Why does it succeed on one element and fails on the list?
If you want to just dump a dictionary to JSON, just use json.dumps. It can easily be made to serialize objects by passing in a custom serialization class - there's one included with Django that deals with datetimes already:
from django.core.serializers.json import DjangoJSONEncoder
json.dumps(mydictionary, cls=DjangoJSONEncoder)
.values() doesn't actually return a list. It returns a ValuesQuerySet which is not serializable by the json module. Try converting album_image_list to a list:
json.dumps(list(album_image_list), cls=DjangoJSONEncoder)
Which DjangoJSONEncoder you are using?
it looks like DjangoJSONEncoder may not support encoding list of results.
Try this:
JSON Serializing Django Models with simplejson
class DateTimeJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
return obj.isoformat()
else:
return super(DateTimeJSONEncoder, self).default(obj)
updated_at=DateTimeJSONEncoder().encode(p.updated_at)
This will help you to serialize datetime object
Related
I have a query from python to mongodb through pymongo.
Some records work for me, but on one record it stops working, it looks like there is a different date format, but how then it got into the find() :
from bson import ObjectId
import config_auth
from operator import itemgetter
from datetime import datetime
import pyodbc
import pymongo
mydb = config_auth.mydb
def load():
temp_arr = []
for item in mydb.questionaries.find({ 'created_at' : {"$gt": datetime(2019,10,30), "$lt": datetime(2019,10,31)}}):
temp=[]
temp.append(str(item['_id']))
print(item['created_at'])
temp_arr.append(tuple(temp))
After this i have this error:
2019-10-30 15:36:09.920000
2019-10-30 15:36:02.344000
2019-10-30 15:36:02.344000
2019-10-30 15:33:47.360000
2019-10-30 15:33:47.360000
Traceback (most recent call last):
File "c:/Users/d.konoplya/Desktop/python/etl_finservice/questionaries.py", line 115, in <module>
print(load())
File "c:/Users/d.konoplya/Desktop/python/etl_finservice/questionaries.py", line 16, in load
for item in mydb.questionaries.find({ 'created_at' : {"$gt": datetime(2019,10,30), "$lt": datetime(2019,10,31)}}):
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\cursor.py", line 1156, in next
if len(self.__data) or self._refresh():
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\cursor.py", line 1093, in _refresh
self.__send_message(g)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\cursor.py", line 955, in __send_message
address=self.__address)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\mongo_client.py", line 1346, in _run_operation_with_response
exhaust=exhaust)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\mongo_client.py", line 1464, in _retryable_read
return func(session, server, sock_info, slave_ok)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\mongo_client.py", line 1340, in _cmd
unpack_res)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\server.py", line 131, in run_operation_with_response
user_fields=user_fields)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\cursor.py", line 1030, in _unpack_response
legacy_response)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\pymongo\message.py", line 1538, in unpack_response
self.documents, codec_options, user_fields)
File "C:\Users\d.konoplya\AppData\Local\Programs\Python\Python37\lib\site-packages\bson\__init__.py", line 1098, in _decode_all_selective
bson.errors.InvalidBSON: year 0 is out of range
{
"_id" : ObjectId("5db9849eb491a900016f913b"),
"updated_at" : ISODate("2019-10-30T13:10:29.320Z"),
"created_at" : ISODate("2019-10-30T12:39:58.277Z"),
"state" : "credit_issued",
"registred_in_1c_at" : ISODate("2019-10-30T13:33:19.504Z"),
"signer_id" : ObjectId("5d584ab05aeafd000191518a")
}
But each record has a date> 0. What could be the problem?
Thanks
If you did:
import datetime
This could cause problems; try
from datetime import datetime
It's a shame they couldn't have named them something different to prevent the bear traps that are so easy to fall in.
I actually found it, this happens because MongoDB / BSON can represent dates outside the datetime.datetime range of Python.
PyMongo decodes BSON datetime values to instances of Python’s
datetime.datetime. Instances of datetime.datetime are limited to years
between datetime.MINYEAR (usually 1) and datetime.MAXYEAR (usually
9999). Some MongoDB drivers (e.g. the PHP driver) can store BSON
datetimes with year values far outside those supported by
datetime.datetime.
Read more about this in the pymongo FAQ: https://pymongo.readthedocs.io/en/stable/faq.html#why-do-i-get-overflowerror-decoding-dates-stored-by-another-language-s-driver
They suggest the following workarounds.
One is, to query for valid date ranges:
>>> from datetime import datetime
>>> coll = client.test.dates
>>> cur = coll.find({'dt': {'$gte': datetime.min, '$lte': datetime.max}})
The other is, to exclude the offending field from the results:
>>> cur = coll.find({}, projection={'dt': False})
I'm trying to parse data that I receive from a curl request through python. The data is in the following format:
{'meta': {'from': '1520812800',
'granularity': 'daily',
'to': '1523232000',
'total': 6380},
'data': [{'count': 660, 'date': '2018-03-12'},
{'count': 894, 'date': '2018-03-13'}]}
Originally, the data was returned as a string probably because I used response.text to retrieve the data. I converted the string into a dictionary using ast.literal_eval(response.text). I managed to parse the "data" key and ignore "meta". So currently,
data = [{"date":"2018-03-12","count":660},{"date":"2018-03-13","count":894}]}`.
I am trying to export the values for "date" and "count" to a csv file. In my code I have this:
keys = data[0].keys()
print("----------KEYS:---------")
print keys #['date','count']
print("------------------------")
with open('mycsv.csv','wb') as output_file:
thewriter = csv.DictWriter(output_file, fieldnames =
['date','count'])
thewriter.writeheader()
thewriter.writerow(data)
However, python does not like this and gives me an error:
Traceback (most recent call last):
File "curlparser.py", line 45, in <module>
thewriter.writerow(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 152, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 148, in _dict_to_list
+ ", ".join([repr(x) for x in wrong_fields]))
ValueError: dict contains fields not in fieldnames: {"date":"2018-03-12","count":660},{"date":"2018-03-13","count":894}
Django 1.7 introduced the JsonResponse objects, which I try to use to return a list of values to my ajax request.
I want to pass
>>> Genre.objects.values('name', 'color')
[{'color': '8a3700', 'name': 'rock'}, {'color': 'ffff00', 'name': 'pop'}, {'color': '8f8f00', 'name': 'electronic'}, {'color': '9e009e', 'name': 'chillout'}, {'color': 'ff8838', 'name': 'indie'}, {'color': '0aff0a', 'name': 'techno'}, {'color': 'c20000', 'name': "drum'n'bass"}, {'color': '0000d6', 'name': 'worldmusic'}, {'color': 'a800a8', 'name': 'classic'}, {'color': 'dbdb00', 'name': 'hiphop'}]
to a JsonResponse object.
However, my attempts fail.
>>> JsonResponse({'foo': 'bar', 'blib': 'blab'}) # works
<django.http.response.JsonResponse object at 0x7f53d28bbb00>
>>> JsonResponse(Genre.objects.values('name', 'color')) # doesn't work
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/marcel/Dokumente/django/FlushFM/env/lib/python3.4/site-packages/django/http/response.py", line 476, in __init__
raise TypeError('In order to allow non-dict objects to be '
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False
This is probably due to the different data structure of Genre.objects.values().
How would this be done right?
[edit]
With safe=False I get
>>> JsonResponse(Genre.objects.values('name', 'color'), safe=False)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/marcel/Dokumente/django/FlushFM/env/lib/python3.4/site-packages/django/http/response.py", line 479, in __init__
data = json.dumps(data, cls=encoder)
File "/usr/lib/python3.4/json/__init__.py", line 237, in dumps
**kw).encode(obj)
File "/usr/lib/python3.4/json/encoder.py", line 192, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.4/json/encoder.py", line 250, in iterencode
return _iterencode(o, 0)
File "/home/marcel/Dokumente/django/FlushFM/env/lib/python3.4/site-packages/django/core/serializers/json.py", line 109, in default
return super(DjangoJSONEncoder, self).default(o)
File "/usr/lib/python3.4/json/encoder.py", line 173, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: [{'color': '8a3700', 'name': 'rock'}, {'color': 'ffff00', 'name': 'pop'}, {'color': '8f8f00', 'name': 'electronic'}, {'color': '9e009e', 'name': 'chillout'}, {'color': 'ff8838', 'name': 'indie'}, {'color': '0aff0a', 'name': 'techno'}, {'color': 'c20000', 'name': "drum'n'bass"}, {'color': '0000d6', 'name': 'worldmusic'}, {'color': 'a800a8', 'name': 'classic'}, {'color': 'dbdb00', 'name': 'hiphop'}] is not JSON serializable
What works is
>>> JsonResponse(list(Genre.objects.values('name', 'color')), safe=False)
<django.http.response.JsonResponse object at 0x7f53d28bb9e8>
But isn't there a better way to generate a dict out of a Model object?
For future reference, .values() returns a ValuesQuerySet that behaves like a iterable full of dictionaries, so using the list() will make a new instance of a list with all the dictionaries in it. With that, you can create a new dict and serialize that.
response = JsonResponse(dict(genres=list(Genre.objects.values('name', 'color'))))
IIRC, it's not safe to have a JSON object that has a list as root and that's probably why Django is complaining. I couldn't find any reference about that now to provide a source, sorry.
To pass nondictionary values to the JsonResponse as you retrieved with Genres.object.values('name','color') you can simple set the safe argument to false and it will return JSON.
from django.http import JsonResponse
def django_json(request):
data = Genres.object.values('name','color')
return JsonResponse(data, safe=False)
That should return a list of JSON of the values you specified. Check out my article How to Return a Json Response with Django for more detailed info on how this works.
Alternatively, if you would like to return a queryset back as JSON you can use Djangos core serializer like this:
from django.core.serializers import serialize
from django.http import JsonResponse
from .models import Genre
def django_models_json(request):
qs = Genre.objects.all()
data = serialize("json", qs, fields=('name', 'color'))
return JsonResponse(data)
This will return the same as above.
I am performing the following operation:
Prepare some documents: docs = [ doc1, doc2, ... ]. The documents have maybe attachments
I POST to _bulk_docs the list of documents
I get an Exception > Problems updating list of documents (length = 1): (500, ('badarg', '58'))
My bulk_docs is (in this case just one):
[ { '_attachments': { 'image.png': { 'content_type': 'image/png',
'data': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3NCSVQICAjb4U/gAAAgAElEQVR4nO...'}},
'_id': '08b8fc66-cd90-47a1-9053-4f6fefabdfe3',
'_rev': '15-ff3d0e8baa56e5ad2fac4937264fb3f6',
'docmeta': { 'created': '2013-10-01 14:48:24.311257',
'updated': [ '2013-10-01 14:48:24.394157',
'2013-12-11 08:19:47.271812',
'2013-12-11 08:25:05.662546',
'2013-12-11 10:38:56.116145']},
'org_id': 45345,
'outputs_id': None,
'properties': { 'auto-t2s': False,
'content_type': 'image/png',
'lang': 'es',
'name': 'dfasdfasdf',
'text': 'erwerwerwrwerwr'},
'subtype': 'voicemail-st',
'tags': ['RRR-ccc-dtjkqx'],
'type': 'recording'}]
This is the detailed exception:
Traceback (most recent call last):
File "portal_support_ut.py", line 470, in test_UpdateDoc
self.ps.UpdateDoc(self.org_id, what, doc_id, new_data)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/ps/complex_ops.py", line 349, in UpdateDoc
success, doc = database.UpdateDoc(doc_id, new_data)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/updater.py", line 38, in UpdateDoc
res = self.SaveDoc(doc_id, doc)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/saver.py", line 88, in SaveDoc
else : self.bulk_append(doc, flush, update_revision)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 257, in bulk_append
if force_send or flush or not self.timer.use_timer : self.BulkSend(show_progress=True)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 144, in BulkSend
results = self.UpdateDocuments(self.bulk)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 67, in UpdateDocuments
results = self.db.update(bulkdocs)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/client.py", line 764, in update
_, _, data = self.resource.post_json('_bulk_docs', body=content)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 527, in post_json
**params)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 546, in _request_json
headers=headers, **params)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 542, in _request
credentials=self.credentials)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 398, in request
raise ServerError((status, error))
ServerError: (500, ('badarg', '58'))
What does that badarg mean? Is it possible to send attachments when doing _bulk_docs?
The solution is to remove the data:image/png;base64, prefix before sending the attachment to coudhdb.
For a python alternative, see here.
This was answered in our mailing list, repeating the answer here for completeness.
The data field was malformed in two ways;
'data': 'data:image/png;base64,iVBORw0KGgoAA....'
The 'data:image/png;base64,' prefix is wrong, and the base64 part was malformed (CouchDB obviously needs to decode it to store it).
I want to create a mongodb to store the homework results, I create a homework which is a dictionary storing the results' array of each subject.
import pymongo
DBCONN = pymongo.Connection("127.0.0.1", 27017)
TASKSINFO = DBCONN.tasksinfo
_name = "john"
taskid = TASKSINFO.tasksinfo.insert(
{"name": _name,
"homework": {"bio": [], "math": []}
})
TASKSINFO.tasksinfo.update({"_id": taskid},
{"$push": {"homework.bio", 92}})
When I tried to push some information to db, there's error:
Traceback (most recent call last):
File "mongo_push_demo.py", line 13, in <module>
{"$push": {"homework.bio", 92}})
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.5-py2.7-linux-i686.egg/pymongo/collection.py", line 479, in update
check_keys, self.__uuid_subtype), safe)
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.5-py2.7-linux-i686.egg/pymongo/message.py", line 110, in update
encoded = bson.BSON.encode(doc, check_keys, uuid_subtype)
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.5-py2.7-linux-i686.egg/bson/__init__.py", line 567, in encode
return cls(_dict_to_bson(document, check_keys, uuid_subtype))
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.5-py2.7-linux-i686.egg/bson/__init__.py", line 476, in _dict_to_bson
elements.append(_element_to_bson(key, value, check_keys, uuid_subtype))
File "/usr/local/lib/python2.7/dist-packages/pymongo-2.5-py2.7-linux-i686.egg/bson/__init__.py", line 466, in _element_to_bson
type(value))
bson.errors.InvalidDocument: cannot convert value of type <type 'set'> to bson
{"$push": {"homework.bio", 92}})
It should be :, not ,.
{'a', 1} is a set of two elements in Python, that's why you get the error.