I'm using Facebook's Python SDK to extract Ads Insights, but I can't find the right way to retrieve ALL fields without having to declare them, which is pretty cumbersome.
My current code looks like this:
ads = tempaccount.get_insights(
params={'date_preset': 'yesterday',
'level': 'ad'},
fields=[AdsInsights.Field.account_id,
AdsInsights.Field.account_name,
AdsInsights.Field.ad_id,
AdsInsights.Field.ad_name,
AdsInsights.Field.adset_id,
AdsInsights.Field.adset_name,
AdsInsights.Field.campaign_id,
AdsInsights.Field.campaign_name,
AdsInsights.Field.cost_per_outbound_click,
AdsInsights.Field.outbound_clicks,
AdsInsights.Field.spend])
Is there a way to force the "fields" attribute to bring all possible fields without declaring them?
Unfortunately, you will need to specify all the required fields. It's not possible to get all fields without explicitly specifying them.
My code also looks similar to yours when querying the insights endpoint.
This worked for me:
from facebookads.adobjects.adsinsights import AdsInsights
for property, value in vars(AdsInsights.Field).items():
print(property, ":", value)
Got from this post: How to enumerate an object's properties in Python?
I am try to understand Avro Serialization on Confluent Kafka along with Schema Registry usage. It was all going well till the end but the final expectations from AVRO made lots of Confusions to me. As per my reading and understanding, Avro Serialization gives us the flexibility that when we have a change in schema, we can simply manage that without impacting the older producer/consumer.
Following the same, I have developed a python producer which will Check for a Schema existence in Schema-Registry, if absent, create it and start Producing the json messages show below. When I need to change schema, I simply update it in my producer and this produces messages with new schema.
My Old Schema :
data = '{"schema":"{\\"type\\":\\"record\\",\\"name\\":\\"value\\",\\"namespace\\":\\"my.test\\",\\"fields\\":[{\\"name\\":\\"fname\\",\\"type\\":\\"string\\"},{\\"name\\":\\"lname\\",\\"type\\":\\"string\\"},{\\"name\\":\\"email\\",\\"type\\":\\"string\\"},{\\"name\\":\\"principal\\",\\"type\\":\\"string\\"},{\\"name\\":\\"ipaddress\\",\\"type\\":\\"string\\"},{\\"name\\":\\"mobile\\",\\"type\\":\\"long\\"},{\\"name\\":\\"passport_make_date\\",\\"type\\":[\\"string\\",\\"null\\"],\\"logicalType\\":\\"timestamp\\",\\"default\\":\\"None\\"},{\\"name\\":\\"passport_expiry_date\\",\\"type\\":\\"string\\",\\"logicalType\\":\\"date\\"}]}"}'
Sample Data from Producer-1 :
{u'mobile': 9819841242, u'lname': u'Rogers', u'passport_expiry_date': u'2026-05-21', u'passport_make_date': u'2016-05-21', u'fname': u'tom', u'ipaddress': u'208.103.236.60', u'email': u'tom_Rogers#TEST.co.nz', u'principal': u'tom#EXAMPLE.COM'}
My New Schema:
data = '{"schema":"{\\"type\\":\\"record\\",\\"name\\":\\"value\\",\\"namespace\\":\\"my.test\\",\\"fields\\":[{\\"name\\":\\"fname\\",\\"type\\":\\"string\\"},{\\"name\\":\\"lname\\",\\"type\\":\\"string\\"},{\\"name\\":\\"email\\",\\"type\\":\\"string\\"},{\\"name\\":\\"principal\\",\\"type\\":\\"string\\"},{\\"name\\":\\"ipaddress\\",\\"type\\":\\"string\\"},{\\"name\\":\\"mobile\\",\\"type\\":\\"long\\"},{\\"name\\":\\"new_passport_make_date\\",\\"type\\":[\\"string\\",\\"null\\"],\\"logicalType\\":\\"timestamp\\",\\"default\\":\\"None\\"},{\\"name\\":\\"new_passport_expiry_date\\",\\"type\\":\\"string\\",\\"logicalType\\":\\"date\\"}]}"}'
Sample Data from Producer-2 :
{u'mobile': 9800647004, u'new_passport_make_date': u'2011-05-22', u'lname': u'Reed', u'fname': u'Paul', u'new_passport_expiry_date': u'2021-05-22', u'ipaddress': u'134.124.7.28', u'email': u'Paul_Reed#nbc.com', u'principal': u'Paul#EXAMPLE.COM'}
Case 1: when I have 2 producers with above 2 schemas running together, I can successfully consume message with below code. All is well till here.
while True:
try:
msg = c.poll(10)
except SerializerError as e:
xxxxx
break
print msg.value()
Case 2: When I go little deeper in JSON fields, things mixes up and breaks.
At first, say I have one producer running with ‘My Old Schema’ above and one consumer consuming these messages successfully.
print msg.value()["fname"] , msg.value()["lname"] , msg.value()["passport_make_date"], msg.value()["passport_expiry_date"]
When I run 2nd producer with ‘My New Schema’ mentioned above, my Old Consumers breaks as there is No Field passport_expiry_date and passport_make_date which is True.
Question:
Sometime I think, this is expected as it’s me(Developer) who is using the field names which are Not in the Message. But how Avro can help here? Shouldn't the missing field be handled by Avro? I saw examples in JAVA where this situation was handled properly but did not find any example in Python. For example, below github has perfect example of handling this scenario. When the field is not present, Consumer simply prints 'None'.
https://github.com/LearningJournal/ApacheKafkaTutorials
Case 3: When I run the combinations like Old Producer with Old Consumer and then in another terminals New Producer with New Consumer, Producers/Consumers mixes up and things break saying no json field.
Old Consumer ==>
print msg.value()["fname"] , msg.value()["lname"] , msg.value()["passport_make_date"], msg.value()["passport_expiry_date"]
New Consumer ==>
print msg.value()["fname"] , msg.value()["lname"] , msg.value()["new_passport_make_date"], msg.value()["new_passport_expiry_date"]
Question:
Again I think, this is expected. But, then Avro makes me think the right Consumer should get the right message with right schema. If I use msg.value() and always parse the fields at consumer side using programming without any role of Avro, then Where is the benefit of using avro? What is the benefit of sending schema with the messages/storing in SR?
Lastly, is there any way to check the schema attached to a message? I understand, in Avro, schema ID is attached with the message which is used further with Schema Registry while Reading and Writing messages. But I never see it with the messages.
Thanks much in Advance.
It's not clear what compatibility setting you're using on the registry, but I will assume backwards, which means you would have needed to add a field with a default.
Sounds like you're getting a Python KeyError because those keys don't exist.
Instead of msg.value()["non-existing-key"], you can try
option 1: treat it like a dict()
msg.value().get("non-existing-key", "Default value")
option 2: check individually for all the keys that might not be there
some_var = None # What you want to parse
val = msg.value()
if "non-existing-key" not in val:
some_var = "Default Value"
Otherwise, you must "project" the newer schema over the older data, which is what the Java code is doing by using a SpecificRecord subclass. That way, the older data would be parsed with the newer schema, which has the newer fields with their defaults.
If you used GenericRecord in Java instead, you would have similar problems. I'm not sure in Python there is an equivalent to Java's SpecificRecord.
By the way, I don't think the string "None" can be applied for a logicalType=timestamp
I am using simple_salesforce and get all records of a custom object called "SER__Condition__c". I know for a fact that that is the name because I got a list of table names from our administrator.
"api" is an instance of "simple_salesforce.Salesforce".
This is the command I'm executing:
pprint(api.query('SELECT Id FROM SER__Condition__c'))
Which returns this error:
File "path\to\lib\simple_salesforce\api.py", line 698, in _exception_handler
raise exc_cls(result.url, result.status_code, name, response_content)
simple_salesforce.api.SalesforceMalformedRequest: Malformed request https://xxx.salesforce.com/services/data/v29.0/query/?q=SELECT+Id+FROM+SER__Condition__c. Response content: [{'message': "\nSELECT Id FROM SER__Condition__c\n
^\nERROR at Row:1:Column:16\nsObject type 'SER__Condition__c' is not supported. If you are attempting to use a custom object, be sure to append the '__c' after the entity name. Please reference your WSDL or the describe call for the appropriate names.", 'errorCode': 'INVALID_TYPE'}]
Using the exact same command for a default object returns all the records as expected:
pprint(api.query('SELECT Id FROM Account'))
The same also holds true for these two:
api.Account.get('xxxxxxxxxxxxxxxxxx')
api.SER__Condition__c.get('xxxxxxxxxxxxxxxx')
It probably is a permissions issue. Make sure the SER__Condition__c object is visible to the user you are running the query as.
I am 90% sure the issue is with the name of the object. Per Salesforce, the naming convention for a custom object cannot include two consecutive underscores. From the Salesforce error message on object creation: "Error: The Object Name field can only contain underscores and alphanumeric characters. It must be unique, begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores."
If you change "SER__Condition__c" to "SER_Condition__c" (a single underscore between "SER" and "Contition") it should fix the issue.
Yes, I resolved this issue by editing the user profile's custom object permissions. It looks like it defaults to none.
I'm trying to validate json using python validictory, but when I get a validation error the validation ends and missing field name is reported, but if there is more than one field missing I only get a message about the first one. I need to know about all the fields that are missing. Is there an function which will return all missing fields?
This script only needs to validate a json document, so answers using other libraries, languages are welcome.
When you call .validate() set fail_fast property to False. That will force validictory to return all errors instead of only first one.
how do i validate a field conditionally based on the presence of another field. for example, only make "state" required only if "country" is "US".
thanks,
steve
EDIT:
so i figured to do this:
chained_validators = [validators.RequireIfPresent('state', present="country")]
but the error message is associated with "_the_form" instead of "state". is there a way to link it to the field instead?
Had the same problem during a project in my company. We wrote our own Formencode validator for this. We currently try to merge it with the main project. In the meantime you can download it here: https://github.com/GevatterGaul/formencode
There is also a Howto in, well, german: http://techblog.auf-nach-mallorca.info/2014/08/19/dynamische_formulare_validieren_mit_formencode/
But let me give you a quick rundown in the context of your example:
from formencode import validators
v = validators.RequireIfMatching('country', expected_value='US', required_fields=['state'])
v.to_python(dict(country='US', state='Virginia'))
The main benefit is, that in comparison to validators.RequireIfPresent, validators.RequireIfMatching only requires a field when a given field matches a given value. In your example, only if 'country' is 'US', then it requires the 'state' field.
Hope I could help.