Develop Reference

redmine - updating a list type custom field failed

I'm trying to update a list type custom field, with json like
{'id': xx, "name": "xxxx", "multiple": True,
'value': ['string1', 'string2', 'string3']}]
I sent request using python-redmine. And our redmine version is 4.1.1.stable.
Python code:
redmine.project.update(xx,
custom_fields=[{'id': xx, "name": "xxxx", "multiple": True,
'value': ['new_string1', 'new_string2', 'new_string3']}])
Python gave me below error, and the value failed to update. Tried various ways to input the value, but no luck. Thanks in advance for any help!
Traceback (most recent call last):
File "/Users/solidfire/.pyenv/versions/3.8.2/envs/psirt-tools/lib/python3.8/site-packages/redminelib/managers/base.py", line 248, in update
response = self.redmine.engine.request(self.resource_class.http_method_update, url, data=request)
File "/Users/solidfire/.pyenv/versions/3.8.2/envs/psirt-tools/lib/python3.8/site-packages/redminelib/engines/base.py", line 87, in request
return self.process_response(self.session.request(method, url, **kwargs))
File "/Users/solidfire/.pyenv/versions/3.8.2/envs/psirt-tools/lib/python3.8/site-packages/redminelib/engines/base.py", line 182, in process_response
raise exceptions.ValidationError(', '.join(': '.join(e) if isinstance(e, list) else e for e in errors))
redminelib.exceptions.ValidationError: <field name> is not included in the list

/model/train http API giving 500 error when providing “nlu” data in json

I am trying to train model using httpapi and json data blow is the code.
import requests
import json
data = {
"config": "language: en\npipeline:\n- name: WhitespaceTokenizer\n- name: RegexFeaturizer\n- name: LexicalSyntacticFeaturizer\n- name: CountVectorsFeaturizer\n- name: CountVectorsFeaturizer\nanalyzer: \"char_wb\"\nmin_ngram: 1\nmax_ngram: 4\n- name: DIETClassifier\nepochs: 100\n- name: EntitySynonymMapper\n- name: ResponseSelector\nepochs: 100",
"nlu": json.dumps({
"rasa_nlu_data": {
"regex_features": [],
"entity_synonyms": [],
"common_examples": [
{
"text": "i m looking for a place to eat",
"intent": "restaurant_search",
"entities": []
},
{
"text": "I want to grab lunch",
"intent": "restaurant_search",
"entities": []
},
{
"text": "I am searching for a dinner spot",
"intent": "restaurant_search",
"entities": []
},
]
}
}),
"force": False,
"save_to_default_model_directory": True
}
r = requests.post('http://localhost:5005/model/train', json=data)
It gives me 500 error. Below is the log for error:
2020-09-30 07:40:37,511 [DEBUG] Traceback (most recent call last):
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/server.py", line 810, in train
None, functools.partial(train_model, **info)
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/train.py", line 50, in train
additional_arguments=additional_arguments,
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/train.py", line 83, in train_async
config, domain, training_files
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/importers/importer.py", line 79, in load_from_config
config = io_utils.read_config_file(config_path)
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/utils/io.py", line 188, in read_config_file
content = read_yaml(read_file(filename))
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/rasa/utils/io.py", line 124, in read_yaml
return yaml_parser.load(content) or {}
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/ruamel/yaml/main.py", line 343, in load
return constructor.get_single_data()
File "/home/Documents/practice/rasa/test1/venv/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
node = self.composer.get_single_node()
File "_ruamel_yaml.pyx", line 706, in _ruamel_yaml.CParser.get_single_node
File "_ruamel_yaml.pyx", line 724, in _ruamel_yaml.CParser._compose_document
File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
File "_ruamel_yaml.pyx", line 891, in _ruamel_yaml.CParser._compose_mapping_node
File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.parser.ParserError: while parsing a block mapping
in "<unicode string>", line 1, column 1
did not find expected key
in "<unicode string>", line 11, column 1
When I train model using terminal commands and json file, it is trained successfully. I think I am missing some formatting required for /model/train api. Can someone tell me where am I going wrong?
I am using rasa version 1.10.14.
Thankyou in advance.

Turns out that the string in config was not proper. It was giving error when training model due to double quotes used with escape characters. I made some tweaks in the config and it trained the model successfully

Exporting response.txt to csv file

I'm trying to parse data that I receive from a curl request through python. The data is in the following format:
{'meta': {'from': '1520812800',
'granularity': 'daily',
'to': '1523232000',
'total': 6380},
'data': [{'count': 660, 'date': '2018-03-12'},
{'count': 894, 'date': '2018-03-13'}]}
Originally, the data was returned as a string probably because I used response.text to retrieve the data. I converted the string into a dictionary using ast.literal_eval(response.text). I managed to parse the "data" key and ignore "meta". So currently,
data = [{"date":"2018-03-12","count":660},{"date":"2018-03-13","count":894}]}`.
I am trying to export the values for "date" and "count" to a csv file. In my code I have this:
keys = data[0].keys()
print("----------KEYS:---------")
print keys #['date','count']
print("------------------------")
with open('mycsv.csv','wb') as output_file:
thewriter = csv.DictWriter(output_file, fieldnames =
['date','count'])
thewriter.writeheader()
thewriter.writerow(data)
However, python does not like this and gives me an error:
Traceback (most recent call last):
File "curlparser.py", line 45, in <module>
thewriter.writerow(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 152, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 148, in _dict_to_list
+ ", ".join([repr(x) for x in wrong_fields]))
ValueError: dict contains fields not in fieldnames: {"date":"2018-03-12","count":660},{"date":"2018-03-13","count":894}

bulk update failing when document has attachments?

I am performing the following operation:
Prepare some documents: docs = [ doc1, doc2, ... ]. The documents have maybe attachments
I POST to _bulk_docs the list of documents
I get an Exception > Problems updating list of documents (length = 1): (500, ('badarg', '58'))
My bulk_docs is (in this case just one):
[ { '_attachments': { 'image.png': { 'content_type': 'image/png',
'data': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3NCSVQICAjb4U/gAAAgAElEQVR4nO...'}},
'_id': '08b8fc66-cd90-47a1-9053-4f6fefabdfe3',
'_rev': '15-ff3d0e8baa56e5ad2fac4937264fb3f6',
'docmeta': { 'created': '2013-10-01 14:48:24.311257',
'updated': [ '2013-10-01 14:48:24.394157',
'2013-12-11 08:19:47.271812',
'2013-12-11 08:25:05.662546',
'2013-12-11 10:38:56.116145']},
'org_id': 45345,
'outputs_id': None,
'properties': { 'auto-t2s': False,
'content_type': 'image/png',
'lang': 'es',
'name': 'dfasdfasdf',
'text': 'erwerwerwrwerwr'},
'subtype': 'voicemail-st',
'tags': ['RRR-ccc-dtjkqx'],
'type': 'recording'}]
This is the detailed exception:
Traceback (most recent call last):
File "portal_support_ut.py", line 470, in test_UpdateDoc
self.ps.UpdateDoc(self.org_id, what, doc_id, new_data)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/ps/complex_ops.py", line 349, in UpdateDoc
success, doc = database.UpdateDoc(doc_id, new_data)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/updater.py", line 38, in UpdateDoc
res = self.SaveDoc(doc_id, doc)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/saver.py", line 88, in SaveDoc
else : self.bulk_append(doc, flush, update_revision)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 257, in bulk_append
if force_send or flush or not self.timer.use_timer : self.BulkSend(show_progress=True)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 144, in BulkSend
results = self.UpdateDocuments(self.bulk)
File "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py", line 67, in UpdateDocuments
results = self.db.update(bulkdocs)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/client.py", line 764, in update
_, _, data = self.resource.post_json('_bulk_docs', body=content)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 527, in post_json
**params)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 546, in _request_json
headers=headers, **params)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 542, in _request
credentials=self.credentials)
File "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py", line 398, in request
raise ServerError((status, error))
ServerError: (500, ('badarg', '58'))
What does that badarg mean? Is it possible to send attachments when doing _bulk_docs?

The solution is to remove the data:image/png;base64, prefix before sending the attachment to coudhdb.
For a python alternative, see here.

This was answered in our mailing list, repeating the answer here for completeness.
The data field was malformed in two ways;
'data': 'data:image/png;base64,iVBORw0KGgoAA....'
The 'data:image/png;base64,' prefix is wrong, and the base64 part was malformed (CouchDB obviously needs to decode it to store it).

python json serialize datetime

First, a simple question on terms,
Encoding(json.dumps) means, converting something to json string,
decoding(json.loads) means, converting json string to json type(?)
I have a list of objects which I got from
>>> album_image_list = AlbumImage.objects.all().values(*fields)[offset:count]
>>> json.dumps(album_image_list[0], cls=DjangoJSONEncoder)
'{"album": 4, "album__title": "g jfd", "created_at": "2012-08-18T02:23:49Z", "height": 1024.0, "width": 512.0, "url_image": "http://--:8000/media/101ac908-df50-42cc-af6f-b172c8829a31.jpg"}'
but when I try the same on whole list (album_image_list),it fails...
>>> json.dumps(album_image_list, cls=DjangoJSONEncoder)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.6/json/__init__.py", line 237, in dumps
**kw).encode(obj)
File "/usr/lib/python2.6/json/encoder.py", line 367, in encode
chunks = list(self.iterencode(o))
File "/usr/lib/python2.6/json/encoder.py", line 317, in _iterencode
for chunk in self._iterencode_default(o, markers):
File "/usr/lib/python2.6/json/encoder.py", line 323, in _iterencode_default
newobj = self.default(o)
File "/home/--/virtualenvs/aLittleArtist/lib/python2.6/site-packages/django/core/serializers/json.py", line 75, in default
return super(DjangoJSONEncoder, self).default(o)
File "/usr/lib/python2.6/json/encoder.py", line 344, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: [{'album': 4L, 'album__title': u'g jfd', 'created_at': datetime.datetime(2012, 8, 18, 2, 23, 49, tzinfo=<UTC>), 'height': 1024.0, 'width': 512.0, 'url_image': u'http://--:8000/media/101ac908-df50-42cc-af6f-b172c8829a31.jpg'}, {'album': 4L, 'album__title': u'g jfd', 'created_at': datetime.datetime(2012, 8, 18, 1, 54, 51, tzinfo=<UTC>), 'height': 512.0, 'width': 512.0, 'url_image': u'http://--:8000/media/e85d1cf7-bfd8-4e77-b90f-d1ee01c67392.jpg'}] is not JSON serializable
>>>
Why does it succeed on one element and fails on the list?

If you want to just dump a dictionary to JSON, just use json.dumps. It can easily be made to serialize objects by passing in a custom serialization class - there's one included with Django that deals with datetimes already:
from django.core.serializers.json import DjangoJSONEncoder
json.dumps(mydictionary, cls=DjangoJSONEncoder)

.values() doesn't actually return a list. It returns a ValuesQuerySet which is not serializable by the json module. Try converting album_image_list to a list:
json.dumps(list(album_image_list), cls=DjangoJSONEncoder)

Which DjangoJSONEncoder you are using?
it looks like DjangoJSONEncoder may not support encoding list of results.
Try this:
JSON Serializing Django Models with simplejson

class DateTimeJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
return obj.isoformat()
else:
return super(DateTimeJSONEncoder, self).default(obj)
updated_at=DateTimeJSONEncoder().encode(p.updated_at)
This will help you to serialize datetime object