Develop Reference

Use a nested dict (or json) with django-environ?

I've got a nested dict format that I wanted to setup in an environment.
It looks like this:
DEPARTMENTS_INFORMATION={
"Pants": {
"name": "Pants Department",
"email": "pants#department.com",
"ext": "x2121"
},
"Shirts": {
"name": "Shirt Department",
"email": "shirts#department.com",
"ext": "x5151"
},
"Socks": {
"name": "Sock Department",
"email": "socks#department.com",
"ext": " "
}
}
I am using django-environ for this and tried using it like this:
DEPARTMENTS = env.dict("DEPARTMENTS_INFORMATION", default={})
But it's giving me this error:
ValueError: dictionary update sequence element #0 has length 1; 2 is required
I'm not sure how to make the nested dictionary an environment variable - any help appreciated!

You can create 2 files:
file.env (you have to write the dict in one line)
DEPARTMENTS_INFORMATION={"Pants": {"name": "Pants Department","email": "pants#department.com","ext": "x2121"},"Shirts": {"name": "Shirt Department","email": "shirts#department.com","ext": "x5151"},"Socks": {"name": "Sock Department","email": "socks#department.com","ext": " "}}
main.py
import environ
#start the environ
env = environ.Env()
#load the env file
environ.Env.read_env("file.env")
#read the data
data = env.json("DEPARTMENTS_INFORMATION")
print(data)
Hope this helps.

Pull key from json file when values is known (groovy or python)

Is there any way to pull the key from JSON if the only thing I know is the value? (In groovy or python)
An example:
I know the "_number" value and I need a key.
So let's say, known _number is 2 and as an output, I should get dsf34f43f34f34f
{
"id": "8e37ecadf4908f79d58080e6ddbc",
"project": "some_project",
"branch": "master",
"current_revision": "3rtgfgdfg2fdsf",
"revisions": {
"43g5g534534rf34f43f": {
"_number": 3,
"created": "2019-04-16 09:03:07.459000000",
"uploader": {
"_account_id": 4
},
"description": "Rebase"
},
"dsf34f43f34f34f": {
"_number": 2,
"created": "2019-04-02 10:54:14.682000000",
"uploader": {
"_account_id": 2
},
"description": "Rebase"
}
}
}

With Groovy:
def json = new groovy.json.JsonSlurper().parse("x.json" as File)
println(json.revisions.findResult{ it.value._number==2 ? it.key : null })
// => dsf34f43f34f34f

Python 3: (assuming that data is saved in data.json):
import json
with open('data.json') as f:
json_data = json.load(f)
for rev, revdata in json_data['revisions'].items():
if revdata['_number'] == 2:
print(rev)
Prints all revs where _number equals 2.

using dict-comprehension:
print({k for k,v in d['revisions'].items() if v.get('_number') == 2})
OUTPUT:
{'dsf34f43f34f34f'}

remove duplicates data from complex json file in python

I have a complex json file it included nested dics in it.
it looks like this
{
"objectivelist": [{
"measureid": "1122",
"gradeID": "4222332",
"graduationdate": "May",
"system": {
"platform": "MAC",
"TeacherName": "Mike",
"manager": "Jim",
"studentinfomation": {
"ZIP": "94122",
"city": "SF"
}
},
"measureid": "1122",
"gradeID": "4222332",
"graduationdate": "May",
"system": {
"platform": "MAC",
"TeacherName": "joshe",
"manager": "steven"
},
"studentinfomation": {
"ZIP": "94122",
"city": "SF"
}
}]
}
Here the grade ID and Measured ID are the same, so the result should only need to show one times, and my result should be like this:
{"measureid":"1122","gradeID"4222332","graduationdate":"May"}
I do not need the managername, teachername etc.
not sure how to do this. I try to use comprehensation but do not know who to use it in nest dictionary.
Thank you guys.

Depending on how huge the json file is you may need better solution. We will hash fields which are of interest to us and build the unique json iteratively.
check_set = set()
output = []
interesting_fields = ['measureid', 'gradeID', 'graduationdate']
for dat in X['objectivelist']:
m = hashlib.md5()
m.update(dat['measureid'].encode('utf-8'))
m.update(dat['gradeID'].encode('utf-8'))
m.update(dat['graduationdate'].encode('utf-8'))
digest = m.hexdigest()
if digest not in check_set:
output.append({key: dat[key] for key in ['measureid', 'gradeID', 'graduationdate']})
check_set.add(digest)
And you can find your output in output.

python json remove() element issue for a while condition

I have a json file in the following format. What I need to do is to get rid of the whole dictionary of elements where the "URI" does not contain "http"
[
{
"Exchange Name": "Standard & Poor's Indices",
"Number": 33.0,
"URI": "http://us.spindices.com/documents/additional-material/spdji-fixed-income-calendar-2016.xlsx",
"ref_id": "33_Standard_&_Poor's_Indices"
},
{
"Exchange Name": "ISE Mercury",
"Number": 36.0,
"URI": "follow OPRA",
"ref_id": "36_ISE_Mercury"
},
{
"Exchange Name": "Aequitas Neo",
"Number": 37.0,
"URI": "email for holidays",
"ref_id": "37_Aequitas_Neo"
},
{
"Exchange Name": "FINRA SPDS 144A",
"Number": 38.0,
"URI": "https://www.finra.org/industry/trace/trace-holiday-calendar",
"ref_id": "38_FINRA_SPDS_144A"
}
]
So far I have stuck with the following function. Here the problem is that remove() does not actually remove a 'URI' element from the string. But after I run the code for the second time, it works. I think I need to use a while loop for this, but how do I implement it in this setting.
def sys_validate_data():
with open('./data_out/uniq_set.json') as jf:
json_decoded = json.load(jf)
for ix in json_decoded:
if "http" not in ix["URI"]:
json_decoded.remove(ix)
with open('./data_out/uniq_set.json', 'w') as fpw:
json.dump(list(json_decoded), fpw, sort_keys=True, indent=4)

Don't modify a list while iterating over it. Doing so can produce unexpected behavior. Instead, you can use a list comprehension to filter elements from your JSON list:
def sys_validate_data():
with open('./data_out/uniq_set.json') as jf:
json_decoded = [ix for ix in json.load(jf) if "http" in ix["URI"]]
....

Use a list comprehension and don't alter a list while iterating:
validated_json = [entry for entry in json_decoded if entry['URI'].startswith('http')]
Extended example:
def sys_validate_data():
with open('./data_out/uniq_set.json') as jf:
json_decoded = json.load(jf)
validated_json = [entry for entry in json_decoded if entry['URI'].startswith('http')]
with open('./data_out/uniq_set.json', 'w') as fpw:
json.dump(validated_json, fpw, sort_keys=True, indent=4)

How do I read a json file into python?

I'm new to JSON and Python, any help on this would be greatly appreciated.
I read about json.loads but am confused
How do I read a file into Python using json.loads?
Below is my JSON file format:
{
"header": {
"platform":"atm"
"version":"2.0"
}
"details":[
{
"abc":"3"
"def":"4"
},
{
"abc":"5"
"def":"6"
},
{
"abc":"7"
"def":"8"
}
]
}
My requirement is to read the values of all "abc" "def" in details and add this is to a new list like this [(1,2),(3,4),(5,6),(7,8)]. The new list will be used to create a spark data frame.

Open the file, and get a filehandle:
fh = open('thefile.json')
https://docs.python.org/2/library/functions.html#open
Then, pass the file handle into json.load(): (don't use loads - that's for strings)
import json
data = json.load(fh)
https://docs.python.org/2/library/json.html#json.load
From there, you can easily deal with a python dictionary that represents your json-encoded data.
new_list = [(detail['abc'], detail['def']) for detail in data['details']]
Note that your JSON format is also wrong. You will need comma delimiters in many places, but that's not the question.

I'm trying to understand your question as best as I can, but it looks like it was formatted poorly.
First off your json blob is not valid json, it is missing quite a few commas. This is probably what you are looking for:
{
"header": {
"platform": "atm",
"version": "2.0"
},
"details": [
{
"abc": "3",
"def": "4"
},
{
"abc": "5",
"def": "6"
},
{
"abc": "7",
"def": "8"
}
]
}
Now assuming you are trying to parse this in python you will have to do the following.
import json
json_blob = '{"header": {"platform": "atm","version": "2.0"},"details": [{"abc": "3","def": "4"},{"abc": "5","def": "6"},{"abc": "7","def": "8"}]}'
json_obj = json.loads(json_blob)
final_list = []
for single in json_obj['details']:
final_list.append((int(single['abc']), int(single['def'])))
print(final_list)
This will print the following: [(3, 4), (5, 6), (7, 8)]