I have two custom commands at my setup.py file: create_tables and drop_tables:
class create_tables(command):
description = 'create DB tables'
user_options = [
('database=', 'd', 'which database configuration use'),
('reset', 'r', 'reset all data previously'),
]
def initialize_options(self):
command.initialize_options(self)
self.reset = False
def run(self):
if self.reset:
self.run_command('drop_tables')
else:
command.run(self)
from vk_relations import models
models.create_tables()
print 'Tables were created successfully'
class drop_tables(command):
description = 'drop all created DB tables'
user_options = [
('database=', 'd', 'which database configuration use'),
]
def run(self):
command.run(self)
answer = raw_input('Are you sure you want to clear all VK Relations data? (y/n): ')
if 'y' == answer:
from vk_relations import models
models.drop_tables()
print 'Tables were dropped successfully'
elif 'n' == answer:
quit()
else:
sys.exit()
Command $ setup.py create_tables -r -dmain should run command drop_tables and create new tables at main database, but run_command method doesn't allow to provide options to the command. How to specify option database for drop_tables inside create_tables command?
Right now I've used this hack:
cmd_obj = self.distribution.get_command_obj('drop_tables')
cmd_obj.database = self.database
self.run_command('drop_tables')
The "proper" solution
Setting attributes on the objects will fail for the pre-defined targets such as build.
The closest you can get here to a proper solution is following:
class drop_tables(command): # <-- Note this should come from drop_tables command
def finalize_options(self):
self.set_undefined_options("create_tables", ("database", "database"))
This is the approach used to inherit arguments from build to build_py and other subcommands.
On the build command
I do not fancy the circular reference the authors of the distutils package introduced in the build command.
The execution order there is following: the build command calls the build_py subcommand. The subcommand goes back to the build command and gets the parameters that are left undefined. This makes tight coupling because both commands need to know about each other.
Also if another aggregate command were to be added the ambiguity would be introduced - the build_py would have two sources of parameters to inherit.
The approach that would reduce the coupling should be different.
If the build command is an aggregate command and therefore it should handle all the parameter passing to its subcommands.
class build(command):
...
def finalize_options(self):
for cmd_name in self.get_sub_commands():
cmd_obj = self.distribution.get_command_obj(cmd_name)
cmd_obj.set_undefined_options("build", ("build_lib", "build_lib"), ...)
Now there's no need to pass command by name and we can use instance instead. This would also resolve the infinite recursion in set_undefined_options > ensure_finalized > finalize_options > set_undefined_options
The proper solution
Given the current state of affairs the better solutions for you problem would be:
class create_tables(command):
def run(self):
cmd_obj = self.distribution.get_command_obj("drop_tables")
cmd_obj.set_undefined_options("create_tables", ("database", "database"))
self.run_command("drop_tables")
Related
I am looking for some advice to avoid having to instantiate a class twice; this is more of a design pattern question. I am creating an application using the Python Click library.
I have a Settings class that first loads all initial default settings into a dictionary (hard-coded into the application), then loads all settings overrides (if specified) from a TOML file on the user's computer into a dictionary, and then finally merges the two and makes them available as attributes of the class instance (settings.<something>).
For most of these settings, I also want to be able to specify a command-line flag. The priority then becomes:
Command-line flag. If not specified, then fallback to...
User setting in TOML file. If not specified, then finally fallback to...
Application default
In order to achieve this result, I am finding that, when using Click's decorators, I have to do something like this:
import click
from myapp import Settings
settings = Settings()
pass_settings = click.make_pass_decorator(Settings, ensure=True)
#click.command()
#click.help_option('-h', '--help')
#click.option(
'-s', '--disk-size',
default=settings.instance_disk_size,
help="Disk size",
show_default=True,
type=int
)
#click.option(
'-t', '--disk-type',
default=settings.instance_disk_type,
help="Disk type",
show_default=True,
type=click.Choice(['pd-standard', 'pd-ssd'])
)
#pass_settings
def create(settings, disk_size, disk_type):
print(disk_size)
print(disk_type)
Why twice?
The settings = Settings() line is needed to provide the #click.option decorators with the default value. The default value could either come from the user override TOML file (if present), or from the application default.
The click.make_pass_decorator seems to be the recommended way for interleaved commands; it's even mentioned in their documentation. Inside of the function, in addition to the CLI parameters passed, I also sometimes needs to reference other attributes in the Settings class.
My question is, which is better? Is there a way to use the pass_settings decorator in the other click.option decorators? Or should I ditch using click.make_pass_decorator entirely?
One way to approach the problem of not wanting to instantiate Settings twice, is to inherit from click.Option, and insert the settings instance into
the context directly like:
Custom Class:
def build_settings_option_class(settings_instance):
def set_default(default_name):
class Cls(click.Option):
def __init__(self, *args, **kwargs):
kwargs['default'] = getattr(settings_instance, default_name)
super(Cls, self).__init__(*args, **kwargs)
def handle_parse_result(self, ctx, opts, args):
obj = ctx.find_object(type(settings_instance))
if obj is None:
ctx.obj = settings_instance
return super(Cls, self).handle_parse_result(ctx, opts, args)
return Cls
return set_default
Using Custom Class:
To use the custom class, pass the cls parameter to #click.option() decorator like:
# instantiate settings
settings = Settings()
# get the setting option builder
settings_option_cls = build_settings_option_class(settings)
# decorate with an option with an appropraie option name
#click.option("--an_option", cls=settings_option_cls('default_setting_name'))
How does this work?
This works because click is a well designed OO framework. The #click.option() decorator usually instantiates a
click.Option object but allows this behavior to be over ridden with the cls parameter. So it is a relatively
easy matter to inherit from click.Option in our own class and over ride the desired methods.
In this case we use a couple of closures to capture the Settings instance and parameter name. In the returned
class we over ride click.Option.handle_parse_result() to allow us to insert the setting object into the context.
This allows the pass_settings decorator to find the settings in the context, and thus it will not need to create a new instance.
Test Code:
import click
class Settings(object):
def __init__(self):
self.instance_disk_size = 100
self.instance_disk_type = 'pd-ssd'
settings = Settings()
settings_option_cls = build_settings_option_class(settings)
pass_settings = click.make_pass_decorator(Settings)
#click.command()
#click.help_option('-h', '--help')
#click.option(
'-s', '--disk-size',
cls=settings_option_cls('instance_disk_size'),
help="Disk size",
show_default=True,
type=int
)
#click.option(
'-t', '--disk-type',
cls=settings_option_cls('instance_disk_type'),
help="Disk type",
show_default=True,
type=click.Choice(['pd-standard', 'pd-ssd'])
)
#pass_settings
def create(settings, disk_size, disk_type):
print(disk_size)
print(disk_type)
if __name__ == "__main__":
commands = (
'-t pd-standard -s 200',
'-t pd-standard',
'-s 200',
'',
'--help',
)
import sys, time
time.sleep(1)
print('Click Version: {}'.format(click.__version__))
print('Python Version: {}'.format(sys.version))
for cmd in commands:
try:
time.sleep(0.1)
print('-----------')
print('> ' + cmd)
time.sleep(0.1)
create(cmd.split())
except BaseException as exc:
if str(exc) != '0' and \
not isinstance(exc, (click.ClickException, SystemExit)):
raise
Test Results:
Click Version: 6.7
Python Version: 3.6.2 (default, Jul 17 2017, 23:14:31)
[GCC 5.4.0 20160609]
-----------
> -t pd-standard -s 200
200
pd-standard
-----------
> -t pd-standard
100
pd-standard
-----------
> -s 200
200
pd-ssd
-----------
>
100
pd-ssd
-----------
> --help
Usage: test.py [OPTIONS]
Options:
-h, --help Show this message and exit.
-s, --disk-size INTEGER Disk size [default: 100]
-t, --disk-type [pd-standard|pd-ssd]
Disk type [default: pd-ssd]
Differing opinion
Instead of modifying the click invocation and using dynamic class construction, expose the default settings as a class attribute for the Settings class. IE:
#click.option(
'-t', '--disk-type',
default=settings.instance_disk_type,
help="Disk type",
show_default=True,
type=click.Choice(['pd-standard', 'pd-ssd'])
)
becomes
#click.option(
'-t', '--disk-type',
default=Settings.defaults.instance_disk_type,
help="Disk type",
show_default=True,
type=click.Choice(['pd-standard', 'pd-ssd'])
)
This is likely cleaner and makes the semantics (meaning) of your code much clearer than using a class constructor s in the accepted answer.
In fact, the Settings.defaults could well be an instance of Settings. It doesn't mater that you're instantiating twice, as this isn't really the issue here, rather that your client/consumer code for the Settings object has to perform the instantiation. If that's done in the Settings class, it remains a clean API and doesn't require the caller to instantiate twice.
If you have a look at my lidtk repository, especially the classifiers, you can see that the following files are almost identical (current version in case this is fixed in future):
cld2_mod.py
langdetect_mod.py
langid_mod.py
text_cat.py
They all inherit from lidtk.LIDClassifier and they all have the commands
Usage: lidtk <<module name>> [OPTIONS] COMMAND [ARGS]...
Use the <<module name>> language classifier.
Options:
--help Show this message and exit.
Commands:
get_languages
predict
print_languages
wili
wili_k
wili_unk
Is it possible to de-duplicate the click-code? I would like to use inheritance to de-duplicate the code.
Glancing just shortly over your repo I think what you want is something like this:
import click
def group_factory(group, name, params):
"""This creates a subcommand 'name' under the existing click command
group 'group' (which can be the main group), with sub-sub-commands
'cmd1' and 'cmd2'. 'params' could be something to set the context
for this group.
"""
#group.group(name=name)
def entry_point():
pass
#entry_point.command()
#click.option("--foo")
def cmd1(foo):
print("CMD1", name, params, foo)
#entry_point.command()
#click.option("--bar")
def cmd2(bar):
print("CMD2", name, params, bar)
return entry_point
You can either use the return value of group_factory as a main entry point in a set of different scripts:
if __name__ == "__main__":
ep = group_factory(click, "", "something")
ep()
... or you can use group_factory to repeatedly build the same sub-command hierarchy under some top-level command under different names (and with different params):
#click.group()
def cli():
pass
group_factory(cli, "a", 1)
group_factory(cli, "b", 2)
group_factory(cli, "c", 3)
I am developing ryu app. The app is basically a python script. The ryu apps are invoked by ryu-manager like this
ryu-manager {filename}
There are certain parameters that are taken by ryu-manager. I want to know if there is a way i could pass arguments to my file?
argparse module of python to parse command line options is there but am not sure it will work as all arguments I provide are used by ryu-manager not my script.
Any help would be appreciated.
I haven't tried this, but:
"Ryu currently uses oslo.config.cfg for command-line parsing.
(ryu/contrib/oslo/config).
There are several examples in the tree. ryu/app/tunnel_port_updater.py"
from
http://comments.gmane.org/gmane.network.ryu.devel/2709
see also
https://github.com/osrg/ryu/blob/master/ryu/app/tunnel_port_updater.py
The Ryu 'getting started' page simply suggests:
ryu-manager [--flagfile <path to configuration file>] [generic/application specific options…]
http://www.osrg.net/ryu/_sources/getting_started.txt
Doing so is a 4-step process. I'll show an example where you read parameters and then print them, but you could assign them to variables or do whatever you would like to by referencing this process.
Create a .conf file (e.g. params.conf)
#Example Conf File
[DEFAULT]
param1_int = 42
param2_str = "You read my data :)"
param3_list = 1,2,3
param4_float = 3.14
Add the following code to your __init__ method. I did this to the simple_switch_13.py which comes with ryu:
from ryu import cfg
:
:
class SimpleSwitch13(app_manager.RyuApp):
def __init__(self, *args, **kwargs):
super(SimpleSwitch13, self).__init__(*args, **kwargs)
:
CONF = cfg.CONF
CONF.register_opts([
cfg.IntOpt('param1_int', default=0, help = ('The ultimate answer')),
cfg.StrOpt('param2_str', default='default', help = ('A string')),
cfg.ListOpt('param3_list', default = None, help = ('A list of numbers')),
cfg.FloatOpt('param4_float', default = 0.0, help = ('Pi? Yummy.'))])
print 'param1_int = {}'.format(CONF.param1_int))
print 'param2_str = {}'.format(CONF.param2_str))
print 'param3_list = {}'.format(CONF.param3_list))
print 'param4_float = {}'.format(CONF.param4_float))
Run Script
ryu-manager paramtest.py --config-file [PATH/TO/FILE/params.conf]
Profit
I referenced the following when putting together my answer, they can provide more detail (such as the oslo.config stuff, which I had never heard of prior to running into this issue).
More info on oslo.config: http://www.giantflyingsaucer.com/blog/?p=4822
Ryu email chain on this issue: https://sourceforge.net/p/ryu/mailman/message/33410077/
I have not found a way to pass arguments to a ryu controller. One way that I have used to get around this is to pass arguments as an environment variable. For example, I have a program which invokes ryu-manager and needs to pass a parameter to the app. I do this as follows: ARG=value ryu-manager app.py
Inside the setup.py script I need to create some temporary files for the installation. The natural place to put them would be the "build/" directory.
Is there a way to retrieve its path that works if installing via pypi, from source, easy_install, pip, ...?
Thanks a lot!
By default distutils create build/ in current working dir, but it can be changed by argument --build-base. Seems like distutils parses it when executing setup and parsed argument does not accessible from outside, but you can cut it yourself:
import sys
build_base_long = [arg[12:].strip("= ") for arg in sys.argv if arg.startswith("--build-base")]
build_base_short = [arg[2:].strip(" ") for arg in sys.argv if arg.startswith("-b")]
build_base_arg = build_base_long or build_base_short
if build_base_arg:
build_base = build_base_arg[0]
else:
build_base = "."
This naive version of parser still shorter than optparse's version with proper error handling for unknown flags. Also you can use argparse's parser, which have try_parse method.
distutils/setuptools provide an abstract Command class that users can use to add custom commands to their package's setup process. This is the same class that built-in setup commands like build and install are subclasses of.
Every class that is a subclass of the abstract Command class must implement the initialize_options, finalize_options, and run methods. The "options" these method names refer to are class attributes that are derived from command-line arguments provided by the user (they can also have default values). The initialize_options method is where a class's options are defined, the finalize_options method is where a class's option values are assigned, and the run method is where a class's option values are used to perform the function of the command.
Since command-line arguments may affect more than one command, some command classes may share options with other command classes. For example, all the distutils/setuptools build commands (build, build_py, build_clib, build_ext, and build_scripts) and the install command need to know where the build directory is. Instead of having every one of these command classes define and parse the same command-line arguments into the same options, the build command, which is the first of all these commands to be executed, defines and parses the command-line arguments and options, and all the other classes get the option values from the build command in their finalize_options method.
For example, the build class defines the build_base and build_lib options in its initialize_options method and then computes their values from the command-line arguments in its finalize_options method. The install classes also defines the build_base and build_lib options in its initialize_options method but it gets the values for these options from the build command in its finalize_options method.
You can use the same pattern to add a custom sub-command to the build command as follows (it would be similar for install)
import setuptools
from distutils.command.build import build
class BuildSomething(setuptools.Command):
def initialize_options(self):
# define the command's options
self.build_base = None
self.build_lib = None
def finalize_options(self):
# get the option values from the build command
self.set_undefined_options('build',
('build_base', 'build_base'),
('build_lib', 'build_lib'))
def run(self):
# do something with the option values
print(self.build_base) # defaults to 'build'
print(self.build_lib)
build_something_command = 'build_something'
class Build(build):
def has_something(self):
# update this to check if your build should run
return True
sub_commands = [(build_something_command, has_something)] + build.sub_commands
COMMAND_CLASS = {
build_something_command: BuildSomething, # custom command
'build': Build # override distutils/setuptools build command
}
setuptools.setup(cmdclass=COMMAND_CLASS)
Alternatively, you could just subclass one of the distutils/setuptools classes if you just want to extend its functionality and it already has the options you need
import setuptools
from setuptools.command.build_py import build_py
class BuildPy(build_py):
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
# do something with the option values
print(self.build_lib) # inherited from build_py
build_py.run(self) # make sure the regular build_py still runs
COMMAND_CLASS = {
'build_py': BuildPy # override distutils/setuptools build_py command
}
setuptools.setup(cmdclass=COMMAND_CLASS)
Unfortunately, none of this is very well documented anywhere. I learned most of it from reading the distutils and setuptools source code. Any of the build*.py and install*.py files in either repository's command directory is informative. The abstract Command class is defined in distutils.
perhaps something like this? works in my case with python 3.8
...
from distutils.command.build import get_platform
import sys
import os
...
def configuration(parent_package='', top_path=None):
config = Configuration('', parent_package, top_path)
# add xxx library
config.add_library('xxx',['xxx/src/fil1.F90',
'xxx/src/file2.F90',
'xxx/src/file3.F90'],
language='f90')
# check for the temporary build directory option
_tempopt = None
_chkopt = ('-t','--build-temp')
for _opt in _chkopt:
if _opt in sys.argv:
_i = sys.argv.index(_opt)
if _i < len(sys.argv)-1:
_tempopt = sys.argv[_i+1]
break
# check for the base directory option
_buildopt = 'build'
_chkopt = ('-b','--build-base')
for _opt in _chkopt:
if _opt in sys.argv:
_i = sys.argv.index(_opt)
if _i < len(sys.argv)-1:
_buildopt = sys.argv[_i+1]
break
if _tempopt is None:
# works with python3 (check distutils/command/build.py)
platform_specifier = ".%s-%d.%d" % (get_platform(), *sys.version_info[:2])
_tempopt = '%s%stemp%s'%(_buildopt,os.sep,platform_specifier)
# add yyy module (wraps fortran code in xxx library)
config.add_extension('fastpost',sources=['yyy/src/fastpost.f90'],
f2py_options=['--quiet'],
libraries=['xxx'])
# to access the mod files produced from fortran modules comppilaton, add
# the temp build directory to the include directories of the configuration
config.add_include_dirs(_tempopt)
return config
setup(name="pysimpp",
version="0.0.1",
description="xxx",
author="xxx",
author_email="xxx#yyy",
configuration=configuration,)
I've been thinking about ways to automatically setup configuration in my Python applications.
I usually use the following type of approach:
'''config.py'''
class Config(object):
MAGIC_NUMBER = 44
DEBUG = True
class Development(Config):
LOG_LEVEL = 'DEBUG'
class Production(Config):
DEBUG = False
REPORT_EMAIL_TO = ["ceo#example.com", "chief_ass_kicker#example.com"]
Typically, when I'm running the app in different ways I could do something like:
from config import Development, Production
do_something():
if self.conf.DEBUG:
pass
def __init__(self, config='Development'):
if config == "production":
self.conf = Production
else:
self.conf = Development
I like working like this because it makes sense, however I'm wondering if I can somehow integrate this into my git workflow too.
A lot of my applications have separate scripts, or modules that can be run alone, thus there isn't always a monolithic application to inherit configurations from some root location.
It would be cool if a lot of these scripts and seperate modules could check what branch is currently checked out and make their default configuration decisions based upon that, e.g., by looking for a class in config.py that shares the same name as the name of the currently checked out branch.
Is that possible, and what's the cleanest way to achieve it?
Is it a good/bad idea?
I'd prefer spinlok's method, but yes, you can do pretty much anything you want in your __init__, e.g.:
import inspect, subprocess, sys
def __init__(self, config='via_git'):
if config == 'via_git':
gitsays = subprocess.check_output(['git', 'symbolic-ref', 'HEAD'])
cbranch = gitsays.rstrip('\n').replace('refs/heads/', '', 1)
# now you know which branch you're on...
tbranch = cbranch.title() # foo -> Foo, for class name conventions
classes = dict(inspect.getmembers(sys.modules[__name__], inspect.isclass)
if tbranch in classes:
print 'automatically using', tbranch
self.conf = classes[tbranch]
else:
print 'on branch', cbranch, 'so falling back to Production'
self.conf = Production
elif config == 'production':
self.conf = Production
else:
self.conf = Development
This is, um, "slightly tested" (python 2.7). Note that check_output will raise an exception if git can't get a symbolic ref, and this also depends on your working directory. You can of course use other subprocess functions (to provide a different cwd for instance).