I am trying to get additional fields when making a call through Asana's Python API Tasks.find_by_project(). My code for the call is:
project_tasks = Tasks(self.client).find_by_project(project_gid, opt_fields= ["name", "memberships", "gid"])
And I am getting:
{'id': 408541814417314, 'gid': '408541814417314', 'memberships': [{}], 'name': 'Reports - Develop quality control report to run for MES'}
It seems like I can only access the fields that are populated by the compact task record, but I need additional fields and would like to get them without re-looping through all the tasks and getting the complete task. Oddly, it returns an empty list, but when I look at the full tasks record there are memberships for this task.
I saw this question, which seems to be similar but the given (attempted) solution doesn't work for me (I get no additional fields):
How can I access custom fields from Asana API using Python?
In case anyone else runs into this issue, I had tyo work with Asana to get this figured out. memberships isn't callable, you have to call Tasks(self.client).find_by_project(project_gid, opt_fields= ["name", "memberships.section", "gid"]) or Tasks(self.client).find_by_project(project_gid, opt_fields= ["name", "memberships.project", "gid"]) you can also apparently call opt_expand=['memberships'] to get all of the data.
from asana:
Thanks for your patience!
We heard back from our Platform Team regarding the issue. What you are
experiencing is currently expected behavior, but it is not intuitive,
because the membership object doesn't have any data of its own.
If you wanted to get the nested data, you can specify which data you
want opt_fields=['memberships.project', 'memberships.section'] in
their opt_fields request. Another option is to use
opt_expand=['memberships'] to get all of the data.
Hope this helps! Let me know if there's anything else I can assist you
with.
Related
I wrote a program that sends the following
to App: 8=FIX.4.4|9=156|35=V|34=2|49=id|52=sometime|56=id1|146=1|55=EURUSD|460=4|167=FOR|262=1|263=1|264=1|265=0|267=2|269=0|269=1|10=114|
I receive this. I get the bid and the offer as expected:
from App 8=FIX.4.4|9=217|35=W|34=4|49=id1|52=sometime|56=id|42=sometime1|55=EURUSD|262=1|268=2|269=0|270=1.12438|271=50000|269=1|270=1.12442|271=50000|10094=sometime2|10=002|
But as I request snapshot + update on full refresh, it sends back the following;
to App: 8=FIX.4.4|9=118|35=j|34=3|49=id|52=sometime|56=id1|45=2|58=Conditionally Required Field Missing (299)|372=W|380=5|10=210|
Data Dictionary of my broker is the following: DataDictionary
UseDataDictionary=Y
ValidateUserDefinedFields=N # tried with Y, same
DataDictionary=C:\Users\Documents\FIX44.xml
Any idea of what I did wrong please?
Thank you folks!
Check your counterparty's documentation for what fields they expect you to send in the MarketDataRequest (35=V) message.
In the default DataDictionary, QuoteEntryID (tag 299) doesn't belong to MarketDataRequest or in any of the repeating groups it contains. This means that your counterparty has made a DD customization and added it somewhere.
So your main mistake is that you are not looking at your counterparty's docs, and your local DD is not in sync with theirs. That latter part is not burning you here in this question, but it will burn you later. Get your DD in sync!
Back to this issue: Sure, you're adding QuoteEntryID to the message, but you're adding it to the top-level of the message body, and your counterparty probably isn't looking for it there. If you look again at the default DataDictionary, QuoteEntryID always belong to a group, so your counterparty probably wants it in in a group also. You just need to read their docs to find out which group it is.
TLDR: Counterparties always customize the DataDictionary -- always read your counterparty's docs!
My question may be silly, and I am no expert. What am trying to do is to manipulate the JSON generated by django.core.serializers.serialize .
I'v searched the web to try and find how to serialize some picked fields in my Model.
There is a an argument fields that takes an array of what I want to serialize, but what do I do if I want to add some calculated field that is not stored in the DB?
I've searched a lot with no results, maybe there is something like a method or a class I don't know of yet that may solve it?
This is just a work around ,I'v managed to use model_to_dict to get a dict object then manipulate it as i want then json.dumps to convert it .
Am not worried about deserialization as i can manage to do so as in the serialization but however am only doing this so i can get forward in my app , and this is a silly work around , so am not accepting it as an answer ,still waiting for the experts for some more robust solution.
This morning my GAE application generated several error log: "too much contention on these datastore entities. please try again.". In my mind, this type of error only happens when multiple requests try modify the same entity or entities in the same entity group.
When I got this error, my code is inserting new entities. I'm confused. Does this mean there is a limitation of how fast we can create new entity?
My code of model definition and calling sequence is show below:
# model defnition
class ExternalAPIStats(ndb.Model):
uid = ndb.StringProperty()
api = ndb.StringProperty()
start_at = ndb.DateTimeProperty(auto_now_add=True)
end_at = ndb.DateTimeProperty()
# calling sequence
stats = ExternalAPIStats(userid=current_uid, api="eapi:hr:get_by_id", start_at=start_at, end_at=end_at)
stats.put() # **too much contention** happen here
That's pretty mysterious to me. I was wondering how I shall deal with this problem. Please let me know if any suggestion.
Without seeing how the calls are made(you show the calling code but how often is it called, via loop or many pages calling the same put at the same time) but I believe the issue is better explained here. In particular
You will also see this problem if you create new entities at a high rate with a monotonically increasing indexed property like a timestamp, because these properties are the keys for rows in the index tables in Bigtable.
with the 'start_at' being the culprit. This article explains in more detail.
Possibly (though untested) try doing your puts in batches. Do you run queries on the 'start_at' field? If not removing its indexes will also fix the issue.
How is the puts called (ie what I was asking above in a loop, multiple pages calling)? With that it might be easier to narrow down the issue.
Here is everything you need to know about Datastore Contention and how to avoid it:
https://developers.google.com/appengine/articles/scaling/contention?hl=en
(Deleted)
UPDATE:
You are reaching writes per second limit on the same entity group. Default it is 1 write per second.
https://cloud.google.com/datastore/docs/concepts/limits
Source: https://stackoverflow.com/a/47800087/1034622
I want to store 'status updates' in mongodb. Therefore this collection/array can get very big.I think one option would be to save the documents in an array nested in the user/group/... document.(Different collections need their own 'status updates')The other way would be to create another collection save the messages their and relate the user/group/... to the status updates via another objectId
I want to know
what is faster
what is easier to administrate and query
I think I'm not going to use an orm/drm just "plain" pymongo.
I haven't found any clear answer in the docs, maybe someone already tested this?
This is an older presentation, but still relevant for these kinds of questions, and discusses some of the tradeoffs.
http://www.10gen.com/presentations/mongosf2011/schemascale
TLDR(W) - it depends how many updates is "very big", and how you're accessing them. If you always need to access the full set at once and they're < 16MB, you can embed, if you generally need only a few at a time you can link. There's also a hybrid approach which is to embed recent and link the rest.
I have a Django app that uses django-piston to send out XML feeds to internal clients. Generally, these work pretty well but we have some XML feeds that currently run over 15 minutes long. This causes timeouts, and the feeds become unreliable.
I'm trying to ponder ways that I can improve this setup. If it requires some re-structuring of the data, that could be possible too.
Here is how the data collection currently looks:
class Data(models.Model)
# fields
class MetadataItem(models.Model)
data = models.ForeignKey(Data)
# handlers.py
data = Data.objects.filter(**kwargs)
for d in data:
for metaitem in d.metadataitem_set.all():
# There is usually anywhere between 55 - 95 entries in this loop
label = metaitem.get_label() # does some formatting here
data_metadata[label] = metaitem.body
Obviously, the core of the program is doing much more, but I'm just pointing out where the problem lies. When we have a data list of 300 it just becomes unreliable and times out.
What I've tried:
Getting a collection of all the data id's, then doing a single large query to get all the MetadataItem's. Finally, filtering those in my loop. This was to preserve some queries which it did reduce.
Using .values() to reduce model instance overhead, which did speed it up but not by much.
One idea I'm thinking one simpler solution to this is to write to a cache in steps. So to reduce time out; I would write the first 50 data sets, save to cache, adjust some counter, write the next 50, etc. Still need to ponder this.
Hoping someone can help lead me into the right direction with this.
The problem in the piece of code you posted is that Django doesn't include objects that are connected through a reverse relationship automatically, so you have to make a query for each object. There's a nice way around this, as Daniel Roseman points out in his blog!
If this doesn't solve your problem well, you could also have a look at trying to get everything in one raw sql query...
You could maybe further reduce the query count by first getting all Data id's and then using select_related to get the data and it's metadata in a single big query. This would greatly reduce the number of queries, but the size of the queries might be impractical/too big. Something like:
data_ids = Data.objects.filter(**kwargs).values_list('id', flat = True)
for i in data_ids:
data = Data.objects.get(pk = i).select_related()
# data.metadataitem_set.all() can now be called without quering the database
for metaitem in data.metadataitem_set.all():
# ...
However, I would suggest, if possible, to precompute the feeds from somewhere outside the webserver. Maybe you could store the result in memcache if it's smaller than 1 MB. Or you could be one of the cool new kids on the block and store the result in a "NoSQL" database like redis. Or you could just write it to a file on disk.
If you can change the structure of the data, maybe you can also change the datastore?
The "NoSQL" databases which allow some structure, like CouchDB or MongoDB could actually be useful here.
Let's say for every Data item you have a document. The document would have your normal fields. You would also add a 'metadata' field which is a list of metadata. What about the following datastructure:
{
'id': 'someid',
'field': 'value',
'metadata': [
{ 'key': 'value' },
{ 'key': 'value' }
]
}
You would then be able to easily get to a data record and get all it's metadata. For searching, add indexes to the fields in the 'data' document.
I've worked on a system in Erlang/OTP that used Mnesia which is basically a key-value database with some indexing and helpers. We used nested records heavily to great success.
I added this as a separate answer as it's totally different than the other.
Another idea is to use Celery (www.celeryproject.com) which is a task management system for python and django. You can use it to perform any long running tasks asynchronously without holding up your main app server.