Develop Reference

How to create click Command using API?

I am trying to create a python click Command using the API instead of decorators. This is because I am trying to create commands dynamically from a yaml file.
parsed yaml file:
{'adhoc': {'args': ['abcd',
{'analytic-type': {'type': 'click.Choice(["prof", "fac"], '
'case_sensitive=False)'}},
{'lobplat': {'default': 'ALL',
'type': 'click.Choice(["A","B","C","D","ALL",],case_sensitive=False)'}}],
'options': [{'environment': {'alias': '-e',
'envvar': 'env',
'show_default': 'loc',
'type': 'click.Choice(["a", "b", '
'"c", "d", "e"], '
'case_sensitive=False)'}},
{'email': {'alias': '-m',
'default': 'test#test.com',
'multiple': True}},
{'runtype': {'alias': '-r',
'default': 'ADHOC',
'type': 'click.Choice(["TEST","ADHOC","SCHEDULED"], '
'case_sensitive=False)'}}],
'script': 'nohup '
'/path/to/script/script'}}
At the top level it defines a command called adhoc which has 3 parts:
Arguments (args)
Options (options)
Script (This is the function of the command)
Both argument and options have a list of different Parameters that I want to create.
Here is the class that I have written:
import click
import yaml
class Commander():
def __init__(self) -> None:
pass
def run_command(self, script):
pass
def str_to_class(self,classname):
return getattr(sys.modules[__name__], classname)
def create_args(self,arguments):
all_args = []
for arg in arguments:
if isinstance(arg, str):
all_args.append(click.Argument([arg]))
else:
attributes = arg[list(arg.keys())[0]]
print(attributes)
all_args.append(click.Argument([arg],**attributes))
return all_args
def convert_to_command(self,yaml):
for key, value in yaml.items():
name = key
script = value["script"]
options = value["options"]
args = value["args"]
click_arg = self.create_args(args)
print(click_arg)
if __name__ == "__main__":
commander = Commander()
yaml = {'adhoc': {'args': ['abcd',
{'analytic-type': {'type': 'click.Choice(["prof", "fac"], '
'case_sensitive=False)'}},
{'lobplat': {'default': 'ALL',
'type': 'click.Choice(["A","B","C","D","ALL",],case_sensitive=False)'}}],
'options': [{'environment': {'alias': '-e',
'envvar': 'env',
'show_default': 'loc',
'type': 'click.Choice(["a", "b", '
'"c", "d", "e"], '
'case_sensitive=False)'}},
{'email': {'alias': '-m',
'default': 'test#test.com',
'multiple': True}},
{'runtype': {'alias': '-r',
'default': 'ADHOC',
'type': 'click.Choice(["TEST","ADHOC","SCHEDULED"], '
'case_sensitive=False)'}}],
'script': 'nohup '
'/path/to/script/script'}}
commander.convert_to_command(yaml)
These functions are not complete. Currently I am working on writing a function to create Arguments out of the Yaml dictionary. However upon running create_command() I get the following error:
File "/project/helper/commander.py", line 111, in <module>
commander.convert_to_command(yaml)
File "/project/hassle/helper/commander.py", line 45, in convert_to_command
click_arg = self.create_args(args)
File "/project/hassle/helper/commander.py", line 32, in create_args
all_args.append(click.Argument([arg],**attributes))
File "/home/myself/miniconda3/envs/py_copier/lib/python3.7/site-packages/click/core.py", line 2950, in __init__
super().__init__(param_decls, required=required, **attrs)
File "/home/myself/miniconda3/envs/py_copier/lib/python3.7/site-packages/click/core.py", line 2073, in __init__
param_decls or (), expose_value
File "/home/myself/miniconda3/envs/py_copier/lib/python3.7/site-packages/click/core.py", line 2983, in _parse_decls
name = name.replace("-", "_").lower()
AttributeError: 'dict' object has no attribute 'replace'

Thank you for the updated code snippet!
Now, what is going on here:
all_args.append(click.Argument([arg],**attributes))
Where arg is:
{'analytic-type': {'type': 'click.Choice(["prof", "fac"], case_sensitive=False)'}}
but if you look at the documentation about the click.Argument class, you'll see the following:
class click.Argument(param_decls, required=None, **attrs)
param_decls (Sequence[str]) –
You made the sequence part, but inside the sequence you still have the dict, instead of the string(s) with the argument name(s). So you should provide click.Argument with something like this:
[list(arg.keys())[0]]
# e.g.
all_args.append(click.Argument([list(arg.keys())[0]],**attributes))
But now you have another problem:
AttributeError: 'str' object has no attribute '__name__'
which is now related to your attributes, as you pass the function as string, but click expects callable. Take a look at this question. It should resolve the issue.
PS: I've just tested it with a quick and dirty eval and it works, so when you fix the callable problem in a proper way, you are good to move on.

Trying to make a local file dictionary and search system

I am trying to make a local file searcher, which will search for files based on tags, and also will by names. i dont have any idea on how to make the searching system nor the python dictionary and searching with tags which confuse me.
files = {'samplefile1.txt', 'samplefile2.txt'}
fileName = ''
fileDiscription = 'Enter Discription here'
isMP3File = True
isMP4File = True
isTxtFile = True
isArchived = True
tags = ['sample1', 'sample2', 'favorited']
filesDictionary = {
'samplefile1.txt': {
fileName: 'coolFile1',
fileDiscription: 'cool disc.',
isMP3File: False,
isMP4File: False,
isTxtFile: True,
isArchived: False,
tags = ['sample1', 'favorited']
},
'samplefile1.txt': {
fileName: 'coolFile2',
fileDiscription: 'cool disc2',
isMP3File: False,
isMP4File: False,
isTxtFile: True,
isArchived: True,
tags = ['sample2']
},
}
so in the code above, with search function, it should show only samplefile1.txt when searched by 'sample1', or 'favorited', or samplefile2.txt if searched with 'sample2'
(also fileName is the name i was talking about in this question, not the file name on pc)
(also any idea on how to automate this 'files' dictionary adding using gui (something like how you would post stuff to twitter or smth, with ticks and message boxes))

Create a dictionary where you have each tag as a key, and the filename as a value.
Since you want to search by tag, having the tags as keys will make the search time constant.
searchDict = {
'sample1': ['samplefile1.txt'],
'favorited': ['samplefile1.txt'],
'sample2': ['samplefile2.txt']
}
then given a tag you can just get the filename inmediately
searchDict['sample1'] # will return ['samplefile1.txt']
You can then use that key to access your main dictionary files
for filename in searchDict['sample1']:
print(files[filename])
will print
{
fileName: 'coolFile1',
fileDiscription: 'cool disc.',
isMP3File: False,
isMP4File: False,
isTxtFile: True,
isArchived: False,
tags = ['sample1', 'favorited']
}
To create the searchDict, you can iterate once over your database of files, getting the tags and associating them to the filenames. It will be a costly operation if your database is big, but once done your search will run in constant time.

Read lines from a text file containing dictionaries into elements of list

I have a text file that looks like this
{'tableName': 'customer', 'type': 'VIEW'}
{'tableName': 'supplier', 'type': 'TABLE'}
{'tableName': 'owner', 'type': 'VIEW'}
I want to read it into a python program that stores it into a list of dictonaries like this
expectedOutput=[{'tableName': 'customer', 'type': 'VIEW'},{'tableName': 'supplier', 'type': 'TABLE'},{'tableName': 'owner', 'type': 'VIEW'}]
But the output I get is a list of strings
output = ["{'tableName': 'customer', 'type': 'VIEW'}",
"{'tableName': 'supplier', 'type': 'TABLE'}",
"{'tableName': 'owner', 'type': 'VIEW'}"]
The code I run is
my_file3 = open("textFiles/python.txt", "r")
data3 = my_file3.read()
output = data3.split("\n")
Can someone show me how do I store the entries inside the list as dicts and not strings.
Thank you

You can use eval but it can be dangerous (only do this if you trust the file):
my_file3 = open("textFiles/python.txt") # specifying 'r' is unnecessary
data3 = my_file3.read()
output = [eval(line) for line in data3.splitlines()] # use splitlines() rather than split('\n')
If the file contains something like __import__('shutil').rmtree('/') it could be very dangerous. Read the documentation for eval here
If you don't fully trust the file, use ast.literal_eval:
import ast
my_file3 = open("textFiles/python.txt")
data3 = my_file3.read()
output = [ast.literal_eval(line) for line in data3.splitlines()]
This removes the risk - if the file contains something like an import, it will raise a ValueError: malformed node or string. Read the documentation for ast.literal_eval here
Output:
[{'tableName': 'customer', 'type': 'VIEW'},
{'tableName': 'supplier', 'type': 'TABLE'},
{'tableName': 'owner', 'type': 'VIEW'}]

You can use the json module
import json
my_file3 = open("textFiles/python.txt", "r")
data3 = my_file3.read()
output = json.loads(str(data3.splitlines()))
print(output)
As Thonnu warned, eval is quite dangerous

String indices must be integers - Django

I have a pretty big dictionary which looks like this:
{
'startIndex': 1,
'username': 'myemail#gmail.com',
'items': [{
'id': '67022006',
'name': 'Adopt-a-Hydrant',
'kind': 'analytics#accountSummary',
'webProperties': [{
'id': 'UA-67522226-1',
'name': 'Adopt-a-Hydrant',
'websiteUrl': 'https://www.udemy.com/,
'internalWebPropertyId': '104343473',
'profiles': [{
'id': '108333146',
'name': 'Adopt a Hydrant (Udemy)',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}, {
'id': '132099908',
'name': 'Unfiltered view',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}],
'level': 'STANDARD',
'kind': 'analytics#webPropertySummary'
}]
}, {
'id': '44222959',
'name': 'A223n',
'kind': 'analytics#accountSummary',
And so on....
When I copy this dictionary on my Jupyter notebook and I run the exact same function I run on my django code it runs as expected, everything is literarily the same, in my django code I'm even printing the dictionary out then I copy it to the notebook and run it and I get what I'm expecting.
Just for more info this is the function:
google_profile = gp.google_profile # Get google_profile from DB
print(google_profile)
all_properties = []
for properties in google_profile['items']:
all_properties.append(properties)
site_selection=[]
for single_property in all_properties:
single_propery_name=single_property['name']
for single_view in single_property['webProperties'][0]['profiles']:
single_view_id = single_view['id']
single_view_name = (single_view['name'])
selections = single_propery_name + ' (View: '+single_view_name+' ID: '+single_view_id+')'
site_selection.append(selections)
print (site_selection)
So my guess is that my notebook has some sort of json parser installed or something like that? Is that possible? Why in django I can't access dictionaries the same way I can on my ipython notebooks?
EDITS
More info:
The error is at the line: for properties in google_profile['items']:
Django debug is: TypeError at /gconnect/ string indices must be integers
Local Vars are:
all_properties =[]
current_user = 'myemail#gmail.com'
google_profile = `the above dictionary`

So just to make it clear for who finds this question:
If you save a dictionary in a database django will save it as a string, so you won't be able to access it after.
To solve this you can re-convert it to a dictionary:
The answer from this post worked perfectly for me, in other words:
import json
s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
json_acceptable_string = s.replace("'", "\"")
d = json.loads(json_acceptable_string)
# d = {u'muffin': u'lolz', u'foo': u'kitty'}
There are many ways to convert a string to a dictionary, this is only one. If you stumbled in this problem you can quickly check if it's a string instead of a dictionary with:
print(type(var))
In my case I had:
<class 'str'>
before converting it with the above method and then I got
<class 'dict'>
and everything worked as supposed to

Using Argparse and Json together

I am a beginner to Python.
I wanted to know if Argparse and JSON could be used together.
Say, I have variables p,q,r
I could add them to argparse as -
parser.add_argument('-p','--param1',help='x variable', required=True)
parser.add_argument('-q','--param2',help='y variable', required=True)
parser.add_argument('-r','--param3',help='z variable', required=True)
Now suppose I wanted to read the same variables from JSON file, is it possible to do it?
So I could input the values either from command line or a JSON file.
JSON input file -
{
"testOwner": "my name",
"tests": [
"test1",
"test2",
"test3"
],
"testParameters": {
"test1": {
"param1": "0",
"param2": "20",
"param3" : "True"
},
"test2": {
"param1": "cc"
}
}
}

The args Namespace from parse_args can be transformed into a dictionary with:
argparse_dict = vars(args)
The JSON values are also in a dictionary, say json_dict. You can copy selected values from one dictionary to the other, or do a whole scale update:
argparse_dict.update(json_dict)
This way the json_dict values over write the argparse ones.
If you want to preserve both, you either need to have different argument (key) names, or the values have to be lists, which you can append or extend. That takes a bit more work, starting with using the correct nargs value in argparse.
The revised parser produces, with a test input:
In [292]: args=parser.parse_args('-p one -q two -r three'.split())
In [293]: args
Out[293]: Namespace(param1='one', param2='two', param3='three')
In [295]: args_dict = vars(args)
In [296]: args_dict
Out[296]: {'param1': 'one', 'param2': 'two', 'param3': 'three'}
The JSON string, when parsed (json.loads?) produces a dictionary like:
In [317]: json_dict
Out[317]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc'}},
'tests': ['test1', 'test2', 'test3']}
I produced this by pasting your display into my Ipython session, but I think the JSON loader produces the same thing
The argparse values could be added with:
In [318]: json_dict['testParameters']['test3']=args_dict
In [319]: json_dict
Out[319]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc'},
'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
'tests': ['test1', 'test2', 'test3']}
Here I added it as a 3rd test set, taking (by conincidence) a name from the tests list. json_dict['testParameters']['test2']=args_dict would replace the values of test2.
One way to add the args values to the undefined values of 'test2' is:
In [320]: args_dict1=args_dict.copy()
In [322]: args_dict1.update(json_dict['testParameters']['test2'])
In [324]: json_dict['testParameters']['test2']=args_dict1
In [325]: json_dict
Out[325]:
{'testOwner': 'my name',
'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
'test2': {'param1': 'cc', 'param2': 'two', 'param3': 'three'},
'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
'tests': ['test1', 'test2', 'test3']}
I used this version of update to give priority to the 'cc' value in the JSON dictionary.

Turns out to be pretty easy with the following caveats
The setup overrides values in config files with values on the command line
It only uses default values if options have not been set on the command line nor the settings file
It does not check that the settings in the config file are valid
import argparse
import json
parser = argparse.ArgumentParser()
parser.add_argument('--save_json',
help='Save settings to file in json format. Ignored in json file')
parser.add_argument('--load_json',
help='Load settings from file in json format. Command line options override values in file.')
args = parser.parse_args()
if args.load_json:
with open(args.load_json, 'rt') as f:
t_args = argparse.Namespace()
t_args.__dict__.update(json.load(f))
args = parser.parse_args(namespace=t_args)
# Optional: support for saving settings into a json file
if args.save_json:
with open(args.save_json, 'wt') as f:
json.dump(vars(args), f, indent=4)

Given that your JSON file contains a dict of the form:
d = {"name": ["-x", "--xvar"], "help": "Help message", "required": True}
After creating the parser you could "unpack" the dict like so:
parser = argparse.ArgumentParser()
parser.add_argument(*(d.pop("name")), **d)
# Put the 'name' as name/flag and then unpack the rest of
# the dict as the rest of the arguments
parser.parse_args("--xvar 12".split())
>>> Namespace(xvar='12')
However this forces you to maintain the dict keys to fit the arguments name of the method add_arguments. You also do not have a simple/straight forward way of using more advance behaviors like using the action, type, choices arguments.
Also you would have to change the form of your dict to contain the various arguments you want to use. One solution would be to have the name/flag as the key of the dict in a tuple and the arguments would be a dict:
d = {("-x", "--xvar"): {"help": "Help message for x", "required": True},
("-y", "--yvar"): {"help": "Help message for y", "required": True}}
for names, args in d.iteritems():
parser.add_argument(*names, **args) # Use a similar unpacking 'magic' as the first example
parser.parse_args("-x 12 --yvar 42".split())
>>> Namespace(xvar='12', yvar='42')
EDIT
Given the comments from the OP it looks like he wants to parse values taken from a JSON file.
d = {"-x": "12", "-y": "42"}
args = []
for item in d.items():
args.extend(item)
parser.parse_args(args)
>>> Namespace(xvar='12', yvar='42')
EDIT 2
Looking at the argparse documentation this paragraph maybe somewhat relevant.

Here is defaults.json
{
"param1": "from json",
"param2": "from json"
}
and here is args.py
import argparse
from pathlib import Path
import json
json_text = Path('defaults.json').read_text()
args = argparse.Namespace(**json.loads(json_text))
parser = argparse.ArgumentParser()
parser.add_argument('--param1', default='from default')
parser.add_argument('--param2', default='from default')
parser.add_argument('--param3', default='from default')
args = parser.parse_args(namespace=args)
print(args)
running it gives the following output
python args.py --param2 'from par'
Namespace(param1='from json', param2='from par', param3='from default')