I'm using an API that returns different dicts depending on the query. So because I can't be certain I'll have the desired keys, I'm using dict.get() to avoid raising a KeyError. But I am currently inserting these results into a database, and would like to avoid filling rows with None.
What would be the preferred way to deal with this?
Use NULL as default value with dict.get(). In case key is not present in your dict object, it will return NULL instead of None. 'NULL' in databases (most) is equivalent to None in Python. For example:
>>> my_dict = {}
# v Returns `NULL` if key not found in `my_dict`
>>> my_dict.get('key', 'NULL')
'NULL'
In case, you have column as NOT NULL, set them as empty string. For example:
>>> my_dict.get('key', '')
''
Related
I am producing a dataset starting from a series of JSON files associated with a certain ID (authors_df contains a bunch of ids) and I am using for to do this.
I tried with a subset of authors and it works fine.
The problem is that some of the id have have an incomplete Json file. Thus I tried to include some 'else' conditions to make the code work also with incomplete data (json files of length 0).
the problem is that I don't know how to do.
I tried if len(json_value['resonanceCategorizations']['1']['fullData']) > 0 else null
but it does not work (KeyError: '1'). I guess I have to set a different condition encompassing JSON structure of the complete files rather than using null
here is my code, it all works but the problem is with the line with else null.
json_values_per_author = {}
datalist = []
datadict = {}
for index, row in authors_df.iterrows():
#get the author
author = row['author']
print(author)
#build the url
url = f'http://keystone-db.default.svc.cluster.local:5000/keystonedb/profiles/resonance/categorization?profileId={author}®ionId=1'
#get the json value
json_value = requests.get(url).json()
full_data = json_value['resonanceCategorizations']['1']['fullData'] if len(json_value['resonanceCategorizations']['1']['fullData']) > 0 else null
datalist.append({
"author": author,
"seed1": full_data[0]['seed'],
"seed2": full_data[1]['seed'] if len(full_data) > 2 else 'NA',
"seed3": full_data[2]['seed'] if len(full_data) > 3 else 'NA'
})
another thing I tried was
z = {"000": [{"seed": 0, "globalSegmentId": 0, "globalSegmentName": "Nope", "regionId": 0, "resonance": 0, "isGlobal": true, "globalRegion": 1}]}
full_data = json_value['resonanceCategorizations']['1']['fullData'] if len(json_value['resonanceCategorizations']['1']['fullData']) > 0 else z
basically creating a "null" JSON value to input as a default if there is no data
alternatively, it would be fine if I could just avoid appending the authors with no data.
If you are having problems with missing keys in dictionary, have a look at returning default value from dictionary
get(key[, default])
Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.
So in your case it might look like
full_data = json_value.get('resonanceCategorizations', {}).get('1', {}).get('fullData')
The problem is, it is unclear which key was not found, if 'resonanceCategorizations' or '1' is not found, you can not apply len to it.
There are two approaches you can take. The first one is to use the dict.get method. Consider the following example:
my_dict = {"a": 1, "b":2}
print(my_dict["a"]) # prints 1
print(my_dict.get("a")) # prints 1
print(my_dict.get("a", None)) # prints 1
print(my_dict["c"]) # raises KeyError
print(my_dict.get("c")) # raises KeyError
print(my_dict.get("c", None)) # prints None
This way, you can check whether the given field exists in a dictionary, of course, you need to do this everytime you access a field, and handle if the output is None.
Another approach is to use a try-catch block.
try:
value = some_dictionary["a"]["b"]["c"]
except KeyError:
value = None
The disadvantage of this method is that you do not know whether a, a.b or a.b.c was missing.
I want to return None if a dictionary x doesn't have the key field or if it does have the key but the value is an empty dict.
I had results = x.get('field') which returns None if the key isn't present. If it is present x.get('field') will return an dict. I want results to also be None if x.get('field') returns {}. Is there a nice pythonic way of doing this?
Currently I have results = x.get('field') if x.get('field') else None. This seems very convoluted.
You can use or operator to explicitly return None if the dict contains a empty dict
results = x.get('field') or None
But be aware that you would get None, for other types of values whole bool status is false, for example 0, '' etc.
Python is duck typed, and a dictionary can contain any kind of value. You would have to explicitly check if the return value is a dict and whether it's empty.
value = x.get('field')
if value == {}:
value = None
I have a PostgreSQL database table that has a JSON datatype column. When I store JSON strings in the database, None is not preserved. This can be reproduced with this code snippet:
import json,psycopg2
dictionary = {}
dictionary[None] = 0
conn = psycopg2.connect("host='localhost' dbname='testdb' user='postgres'
password='postgres'")
cursor = conn.cursor()
cursor.execute("""INSERT INTO testtable (json_column) VALUES (%s)""",
(json.dumps(dictionary), ))
cursor.execute("""SELECT json_column FROM testtable""")
result = cursor.fetchall()[0][0].keys()[0]
print result
print type(result)
if result is None:
print 'result is None'
else:
print 'result is not None'
The output of this Python code is:
drew#debian:~/python$ python test.py
null
<type 'unicode'>
result is not None
drew#debian:~/python$
How can I store None as a key in the JSON column? The JSON object also has keys for 'None' and 'null', so the value stored must be None or null.
From RFC 7159:
An object structure is represented as a pair of curly brackets
surrounding zero or more name/value pairs (or members). A name is a
string.
And from the Python docs:
Keys in key/value pairs of JSON are always of the type str. When a
dictionary is converted into JSON, all the keys of the dictionary are
coerced to strings.
JSON does not have any concept of non-string dict keys. If you want those, you don't want JSON. The closest you'll get with JSON is detecting the string None gets converted to and hoping that you never need to actually use the string 'null' as a dict key.
I would like to use the in_ operator in sqlalchemy using two values, one of them being NULL (mysql NULL), I don't know how to pass it via Python?
So I have a Python cgi that contains a bunch of parameters that I format then finally store inside a dict queryValues (the key being the column name and the value being a value sent by the user stored inside a fieldStorage)
for attr,value in queryValues.items() : #queryValues is a dict of parameters
valueWithNone = value.append(None) #I want to includ NULL
and_args_of_cr = [(and_(getattr(TableCR.c,attr).in_(valueWithNone)))]
I tried None and sqlalchemy.sql.null(), also tried putting directly in_(value,None) but value has the form ['Yes'] so I don't know how to do this.
But it's not working, how can I do this please?
The line value.append(None) is an in-place modification and does not return anything, so valueWithNone will be None. This is probably what you're after:
for attr,value in queryValues.items():
queryvalues = value[:] # Create a copy of list
queryvalues.append(None)
and_args_of_cr = [(and_(getattr(TableCR.c,attr).in_(queryvalues)))]
None seems to work as a dictionary key, but I am wondering if that will just lead to trouble later. For example, this works:
>>> x={'a':1, 'b':2, None:3}
>>> x
{'a': 1, None: 3, 'b': 2}
>>> x[None]
3
The actual data I am working with is educational standards. Every standard is associated with a content area. Some standards are also associated with content subareas. I would like to make a nested dictionary of the form {contentArea:{contentSubArea:[standards]}}. Some of those contentSubArea keys would be None.
In particular, I am wondering if this will lead to confusion if I look for a key that does not exist at some point, or something unanticipated like that.
Any hashable value is a valid Python Dictionary Key. For this reason, None is a perfectly valid candidate. There's no confusion when looking for non-existent keys - the presence of None as a key would not affect the ability to check for whether another key was present. Ex:
>>> d = {1: 'a', 2: 'b', None: 'c'}
>>> 1 in d
True
>>> 5 in d
False
>>> None in d
True
There's no conflict, and you can test for it just like normal. It shouldn't cause you a problem. The standard 1-to-1 Key-Value association still exists, so you can't have multiple things in the None key, but using None as a key shouldn't pose a problem by itself.
You want trouble? here we go:
>>> json.loads(json.dumps({None:None}))
{u'null': None}
So yea, better stay away from json if you do use None as a key. You can patch this by custom (de/)serializer, but I would advise against use of None as a key in the first place.
None is not special in any particular way, it's just another python value. Its only distinction is that it happens to be the return value of a function that doesn't specify any other return value, and it also happens to be a common default value (the default arg of dict.get(), for instance).
You won't cause any run-time conflicts using such a key, but you should ask yourself if that's really a meaningful value to use for a key. It's often more helpful, from the point of view of reading code and understanding what it does, to use a designated instance for special values. Something like:
NoSubContent = SubContentArea(name=None)
{"contentArea":
{NoSubContent:[standards],
SubContentArea(name="Fruits"): ['apples', 'bananas']}}
jsonify does not support a dictionary with None key.
From Flask import jsonify
def json_():
d = {None: 'None'}
return jsonify(d)
This will throw an error:
TypeError: '<' not supported between instances of 'NoneType' and 'str'
It seems to me, the larger, later problem is this. If your process is creating pairs and some pairs have a "None" key, then it will overwrite all the previous None pairs. Your dictionary will silently throw out values because you had duplicate None keys. No?
Funny though, even this works :
d = {None: 'None'}
In [10]: None in d
Out[10]: True