I am a beginner to Python.
I wanted to know if Argparse and JSON could be used together.
Say, I have variables p,q,r
I could add them to argparse as -
parser.add_argument('-p','--param1',help='x variable', required=True)
parser.add_argument('-q','--param2',help='y variable', required=True)
parser.add_argument('-r','--param3',help='z variable', required=True)
Now suppose I wanted to read the same variables from JSON file, is it possible to do it?
So I could input the values either from command line or a JSON file.
JSON input file -
{
"testOwner": "my name",
"tests": [
"test1",
"test2",
"test3"
],
"testParameters": {
"test1": {
"param1": "0",
"param2": "20",
"param3" : "True"
},
"test2": {
"param1": "cc"
}
}
}
The args Namespace from parse_args can be transformed into a dictionary with:
argparse_dict = vars(args)
The JSON values are also in a dictionary, say json_dict. You can copy selected values from one dictionary to the other, or do a whole scale update:
argparse_dict.update(json_dict)
This way the json_dict values over write the argparse ones.
If you want to preserve both, you either need to have different argument (key) names, or the values have to be lists, which you can append or extend. That takes a bit more work, starting with using the correct nargs value in argparse.
The revised parser produces, with a test input:
In [292]: args=parser.parse_args('-p one -q two -r three'.split())
In [293]: args
Out[293]: Namespace(param1='one', param2='two', param3='three')
In [295]: args_dict = vars(args)
In [296]: args_dict
Out[296]: {'param1': 'one', 'param2': 'two', 'param3': 'three'}
The JSON string, when parsed (json.loads?) produces a dictionary like:
In [317]: json_dict
Out[317]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc'}},
'tests': ['test1', 'test2', 'test3']}
I produced this by pasting your display into my Ipython session, but I think the JSON loader produces the same thing
The argparse values could be added with:
In [318]: json_dict['testParameters']['test3']=args_dict
In [319]: json_dict
Out[319]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc'},
'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
'tests': ['test1', 'test2', 'test3']}
Here I added it as a 3rd test set, taking (by conincidence) a name from the tests list. json_dict['testParameters']['test2']=args_dict would replace the values of test2.
One way to add the args values to the undefined values of 'test2' is:
In [320]: args_dict1=args_dict.copy()
In [322]: args_dict1.update(json_dict['testParameters']['test2'])
In [324]: json_dict['testParameters']['test2']=args_dict1
In [325]: json_dict
Out[325]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc', 'param2': 'two', 'param3': 'three'},
'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
'tests': ['test1', 'test2', 'test3']}
I used this version of update to give priority to the 'cc' value in the JSON dictionary.
Turns out to be pretty easy with the following caveats
The setup overrides values in config files with values on the command line
It only uses default values if options have not been set on the command line nor the settings file
It does not check that the settings in the config file are valid
import argparse
import json
parser = argparse.ArgumentParser()
parser.add_argument('--save_json',
help='Save settings to file in json format. Ignored in json file')
parser.add_argument('--load_json',
help='Load settings from file in json format. Command line options override values in file.')
args = parser.parse_args()
if args.load_json:
with open(args.load_json, 'rt') as f:
t_args = argparse.Namespace()
t_args.__dict__.update(json.load(f))
args = parser.parse_args(namespace=t_args)
# Optional: support for saving settings into a json file
if args.save_json:
with open(args.save_json, 'wt') as f:
json.dump(vars(args), f, indent=4)
Given that your JSON file contains a dict of the form:
d = {"name": ["-x", "--xvar"], "help": "Help message", "required": True}
After creating the parser you could "unpack" the dict like so:
parser = argparse.ArgumentParser()
parser.add_argument(*(d.pop("name")), **d)
# Put the 'name' as name/flag and then unpack the rest of
# the dict as the rest of the arguments
parser.parse_args("--xvar 12".split())
>>> Namespace(xvar='12')
However this forces you to maintain the dict keys to fit the arguments name of the method add_arguments. You also do not have a simple/straight forward way of using more advance behaviors like using the action, type, choices arguments.
Also you would have to change the form of your dict to contain the various arguments you want to use. One solution would be to have the name/flag as the key of the dict in a tuple and the arguments would be a dict:
d = {("-x", "--xvar"): {"help": "Help message for x", "required": True},
("-y", "--yvar"): {"help": "Help message for y", "required": True}}
for names, args in d.iteritems():
parser.add_argument(*names, **args) # Use a similar unpacking 'magic' as the first example
parser.parse_args("-x 12 --yvar 42".split())
>>> Namespace(xvar='12', yvar='42')
EDIT
Given the comments from the OP it looks like he wants to parse values taken from a JSON file.
d = {"-x": "12", "-y": "42"}
args = []
for item in d.items():
args.extend(item)
parser.parse_args(args)
>>> Namespace(xvar='12', yvar='42')
EDIT 2
Looking at the argparse documentation this paragraph maybe somewhat relevant.
Here is defaults.json
{
"param1": "from json",
"param2": "from json"
}
and here is args.py
import argparse
from pathlib import Path
import json
json_text = Path('defaults.json').read_text()
args = argparse.Namespace(**json.loads(json_text))
parser = argparse.ArgumentParser()
parser.add_argument('--param1', default='from default')
parser.add_argument('--param2', default='from default')
parser.add_argument('--param3', default='from default')
args = parser.parse_args(namespace=args)
print(args)
running it gives the following output
python args.py --param2 'from par'
Namespace(param1='from json', param2='from par', param3='from default')
Related
I extracted Module_name from a JSON file in which a key Social is present twice
with open(filename,'r') as jsonFile:
data = json.load(jsonFile)
module = data.get('UserIntents') #get the key "UserIntents"
for bcd in module: #inside the UserIntents:
if 'ModuleName' in bcd: #extracted ModuleName
Module_Name = bcd.get('ModuleName') #extract ModuleName information
Intent_Name = bcd.get('IntentName') #extract Intent Name
data = {Module_Name: Intent_Name}
ModName.update(data)
Output of data :
print data {'ClickUrl': 'HTTP'}
print data {'AngryUser': 'HATE'}
print data {'HelloGoodBye': 'HELLO'}
print data {'Social': 'TEAM_WELLNESS'}
print data {'Social': 'TEAM_MEMBERS'}
It has a key "Social" but with different values
As of now, a dictionary: ModName is chosen to store this but it ofcourse overrides the TEAM_WELLNESS as it doesn't accept duplicates.
Now, even it is being stores as list like
data = {ModName: [Intent_Name]}
It is still overriding the previous value, how to handle such cases with same key and diffrent values?
The other way is check if Module_name has the same content twice, it will change the second occurence by appending a suffix, something like SOCIAL_1 , SOCIAL_2 to avoid duplicates but I am new to python and not sure how to handle this functionality
Current Output using Dictionary :
ModName= {'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO', 'Social': 'TEAM_MEMBERS'}
Expected Output :
ModName= {'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO', 'Social': 'TEAM_MEMBERS','Social_1':'TEAM_WELLNESS'}
This does not generate the required output. It is a suggestion of how the output might be better managed.
data = {"UserIntents": [{"FirstVariation": "How is the Alchemy team doing?", "IntentName": "TEAM_WELLNESS", "ModuleName": "Social", }, {
"FirstVariation": "I'd like to learn more about the team", "IntentName": "TEAM_MEMBERS", "ModuleName": "Social", }]}
ModName = {'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO'}
MN = 'ModuleName'
IN = 'IntentName'
UI = 'UserIntents'
for d in data[UI]:
ModName.setdefault(d[MN], []).append(d[IN])
print(ModName)
Output:
{'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO', 'Social': ['TEAM_WELLNESS', 'TEAM_MEMBERS']}
Alternative solution:
data = {"UserIntents": [{"FirstVariation": "How is the Alchemy team doing?", "IntentName": "TEAM_WELLNESS", "ModuleName": "Social", }, {
"FirstVariation": "I'd like to learn more about the team", "IntentName": "TEAM_MEMBERS", "ModuleName": "Social", }]}
ModName = {'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO'}
MN = 'ModuleName'
IN = 'IntentName'
UI = 'UserIntents'
control = {}
for d in data[UI]:
key = d[MN]
if suffix := control.setdefault(key, 0):
ModName[f'{key}_{suffix}'] = d[IN]
else:
ModName[key] = d[IN]
control[key] += 1
print(ModName)
Output:
{'ClickUrl': 'HTTP', 'AngryUser': 'HATE', 'HelloGoodBye': 'HELLO', 'Social': 'TEAM_WELLNESS', 'Social_1': 'TEAM_MEMBERS'}
after writing a python script to request some data from a server, I get the response in following structure:
{
'E_AXIS_DATA': {
'item': [
{
'AXIS': '000',
'SET': {
'item': [
{
'TUPLE_ORDINAL': '000000',
'CHANM': '0002',
'CAPTION': 'ECF',
'CHAVL': '0002',
'CHAVL_EXT': None,
'TLEVEL': '00',
'DRILLSTATE': None,
'ATTRIBUTES': None
},
{...
Apparently its not JSON.
After running following command:
results = client.service.RRW3_GET_QUERY_VIEW_DATA("/server")
df = pd.read_json(results)
i get the output meaning that the format is not being accepted as JSON;
ValueError: Invalid file path or buffer object type: <class 'zeep.objects.RRW3_GET_QUERY_VIEW_DATAResponse'>
Any help is welcome.
Thanks
Pandas has DataFrame.read_json() method that can do the trick
import pandas as pd
json_string = '{"content": "a string containing some JSON...." ... etc... }'
df = pd.load_json(json_string)
# Now you can do whatever you like with your dataframe
I have this config.py file:
# config.py
maria = dict(
corners = [1,2,3,4],
area = 2100
)
john = dict(
corners = [5,6,7,8],
area = 2400
)
and want to use parameters from it by running my main program using argsparse. somewhat like this:
# main.py
import config
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("user", help="maria or john")
args = parser.parse_args()
print(args.user)
print(config.args.user['corners'])
when I run:
pyhton3 main.py maria
I get syntax error on the 2nd print, where I would like to get [1,2,3,4].
How can I use the argument from argparse as an attribute to access the appropriate data in the config file?
IIUC:
You can use the getattr built in function in python.
The getattr(object, name[, default]):
Return the value of the named attribute of object. name must be a
string. If the string is the name of one of the object’s attributes,
the result is the value of that attribute. For example, getattr(x,
'foobar') is equivalent to x.foobar. If the named attribute does not >exist, default is returned if provided, otherwise AttributeError is raised.
Replace:
print (config.args.user['corners'])
With:
print(getattr(config, args.user)["corners"])
Avoid using executable Python code for configuration. Use something like JSON:
config.json would look like
{
"maria": {
"corners": [1,2,3,4],
"area": 2100
},
"john": {
"corners": [5,6,7,8],
"area": 2400
}
}
And your script would use
# main.py
import json
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("user", help="maria or john")
args = parser.parse_args()
with open("config.json") as f:
config = json.load(f)
print(args.user)
print(config[args.user]['corners'])
One way to go around this is to wrap your parameters in one general dict:
# config.py
params = {'maria': {'corners': [1,2,3,4], 'area': 2100},
'john': {'corners': [5,6,7,8], 'area': 2400}}
And then you can simply do in main.py:
print(config.params[args.user]['corners'])
I have an issue about how to customize OrderedDict format and convert them into a json or dictionary format(but be able to reset the key names and the structure). I have the data below:
result= OrderedDict([('index', 'cfs_fsd_00001'),
('host', 'GIISSP707'),
('source', 'D:\\usrLLSS_SS'),
('_time', '2018-11-02 14:43:30.000 EDT'),
('count', '153')])
...However, I want to change the format like this:
{
"servarname": {
"index": "cfs_fsd_00001",
"host": "GIISSP707"
},
"times": '2018-11-02 14:43:30.000 EDT',
"metricTags": {
"source": 'D:\\ddevel.log'"
},
"metricName": "serverice count",
"metricValue": 153,
"metricType": "count"
}
I will be really appreciate your help. Basically the output I got is pretty flat. But I want to customize the structure. The original structure is
OrderedDict([('index', 'cfs_fsd_00001'),('host', 'GIISSP707').....]).
The output I want to achieve is {"servarname"{"index":"cfs_fsd_00001","host":"GIISSP707"},......
You can simply reference the result dict with the respective keys that you want your target data structure to have:
{
"servarname": {
"index": result['index'],
"host": result['host']
},
"times": result['_time'],
"metricTags": {
"source": result['source']
},
"metricName": "serverice count",
"metricValue": result['count'],
"metricType": "count"
}
No sure how flexible you need for your method. I assume you have a few common keys in your OrderedDict and you want to find the metric there, then reformat them into a new dict. Here is a short function which is implemented in python 3 and I hope it could help.
from collections import OrderedDict
import json
def reformat_ordered_dict(dict_result):
"""Reconstruct the OrderedDict result into specific format
This method assumes that your input OrderedDict has the following common keys: 'index',
'host', 'source', '_time', and a potential metric whcih is subject to change (of course
you can support more metrics with minor tweak of the code). The function also re-map the
keys (for example, mapping '_time' to 'times', pack 'index' and 'source' into 'servarname'
).
:param dict_result: the OrderedDict
:return: the reformated OrderedDict
"""
common_keys = ('index', 'host', 'source', '_time')
assert all(common_key in dict_result for common_key in common_keys), (
'You have to provide all the commen keys!')
# write common keys
reformated = OrderedDict()
reformated["servarname"] = OrderedDict([
("index", dict_result['index']),
("host", dict_result['host'])
])
reformated["times"] = dict_result['_time']
reformated["metricTags"] = {"source": dict_result['source']}
# write metric
metric = None
for key in dict_result.keys():
if key not in common_keys:
metric = key
break
assert metric is not None, 'Cannot find metric in the OrderedDict!'
# don't know where you get this value. But you can customize it if needed
# for exampe if the metric name is needed here
reformated['metricName'] = "serverice count"
reformated['metricValue'] = dict_result[metric]
reformated['metricType'] = metric
return reformated
if __name__ == '__main__':
result= OrderedDict([('index', 'cfs_fsd_00001'),
('host', 'GIISSP707'),
('source', 'D:\\usrLLSS_SS'),
('_time', '2018-11-02 14:43:30.000 EDT'),
('count', '153')])
reformated = reformat_ordered_dict(result)
print(json.dumps(reformated))
I am parsing JSON that stores various code snippets and I am first building a dictionary of languages used by these snippets:
snippets = {'python': {}, 'text': {}, 'php': {}, 'js': {}}
Then when looping through the JSON I'm wanting add the information about the snippet into its own dictionary to the dictionary listed above. For example, if I had a JS snippet - the end result would be:
snippets = {'js':
{"title":"Script 1","code":"code here", "id":"123456"}
{"title":"Script 2","code":"code here", "id":"123457"}
}
Not to muddy the waters - but in PHP working on a multi-dimensional array I would just do the following (I am lookng for something similiar):
snippets['js'][] = array here
I know I saw one or two people talking about how to create a multidimensional dictionary - but can't seem to track down adding a dictionary to a dictionary within python. Thanks for the help.
This is called autovivification:
You can do it with defaultdict
def tree():
return collections.defaultdict(tree)
d = tree()
d['js']['title'] = 'Script1'
If the idea is to have lists, you can do:
d = collections.defaultdict(list)
d['js'].append({'foo': 'bar'})
d['js'].append({'other': 'thing'})
The idea for defaultdict it to create automatically the element when the key is accessed. BTW, for this simple case, you can simply do:
d = {}
d['js'] = [{'foo': 'bar'}, {'other': 'thing'}]
From
snippets = {'js':
{"title":"Script 1","code":"code here", "id":"123456"}
{"title":"Script 2","code":"code here", "id":"123457"}
}
It looks to me like you want to have a list of dictionaries. Here is some python code that should hopefully result in what you want
snippets = {'python': [], 'text': [], 'php': [], 'js': []}
snippets['js'].append({"title":"Script 1","code":"code here", "id":"123456"})
snippets['js'].append({"title":"Script 1","code":"code here", "id":"123457"})
print(snippets['js']) #[{'code': 'code here', 'id': '123456', 'title': 'Script 1'}, {'code': 'code here', 'id': '123457', 'title': 'Script 1'}]
Does that make it clear?