I have a fairly standard Python test (full sources are here):
from absl.testing import absltest
[...]
class BellTest(absltest.TestCase):
def test_bell(self):
[...]
and the corresponding entries in the BUILD file, where the dependencies are part of the BUILD file:
py_test(
name = "bell_test",
size = "small",
srcs = ["bell_test.py"],
python_version = "PY3",
srcs_version = "PY3",
deps = [
":tensor",
":state",
":ops",
":bell",
],
)
Without problems I can 'run' this via
bazel run bell_test
[...] comes out Ok.
However, I cannot 'test' it
bazel test bell_test
[...] FAILED
The log file tells me that it cannot find the dependency to absl.testing. This is puzzling, given that it works with 'run'. This also works on Linux and MacOS without problems.
I tried to add all kinds of ways to add a dependency to absl / testing, but to no avail. Pointers would be greatly appreciated.
Side note: It would be great if bazel would print the path to the log file with Windows backslashes!
I write a module that wraps functionality of an external binary.
For example, I wrap ls program into a python module my_wrapper.py
import my_wrapper
print my_wrapper.ls('some_directory/')
# list files in some_directory
and in my_wrapper.py I do:
# my_wrapper.py
PATH_TO_LS = '/bin/ls'
def ls(path):
proc = subprocess.Popen([PATH_TO_LS, path], ...)
...
return paths
(of course, I do not wrap ls but some other binary)
The binary might be installed with an arbitrary location, like /usr/bin/, /opt/ or even at the same place as the python script (./binaries/)
Question:
What would be the cleanest (from the user perspective) way to set the path to the binary?
Should the user specify my_wrapper.PATH_TO_LS = ... or invoke some my_wrapper.set_binary_path(path) at the beginning of his script?
Maybe it would be better to specify it in env, and the wrapper would find it with os.environ?
If the wrapper is distributed as egg, can I require during the installation, that the executable is already present in the system (see below)?
egg example:
# setup.py
setup(
name='my_wrapper',
requires_binaries=['the_binary'] # <--- require that the binary is already
# installed and on visible
# on execution path
)
or
easy_install my_wrapper BINARY_PATH=/usr/local/bin/the_binary
Create a "configuration object" with sane defaults. Allow the consumer to modify the values as appropriate. Accept a configuration object instance to your functions, taking the one you created by default.
I would like to get scons to call nosetests with a list of directories. What would be the best way to do this?
If you need to analyze return code of external application (if you calling tests, for example), you need to use Command() + python subprocess module.
If you using only Command you can't get return code of application.
For example:
if 'test' in COMMAND_LINE_TARGETS:
runTestsCmd = env.Command('runTests', None, Action(runTests, "Running tests"))
AlwaysBuild(runTestsCmd)
Alias('test', runTestsCmd)
runTests function example:
def runTests(target = None, source = None, env = None) :
# fill args
retCode = subprocess.call(args, env = env['ENV'], cwd = cwd, shell = True)
Exit(retCode)
Also, you can set additional dependencies for runTestsCmd.
Depends(runTestsCmd, [appAndLibsToBuild])
I dont know the nose framework, but there are two ways to execute external applications with SCons (there are other ways to do it with python, but no need to mention those) as follows:
Execute() - executes always while analyzing the SConscript files
Command() - acts like a target and only executes according to its dependencies
I would think you would want to use the Command() option to only launch the unit tests if one of the related dependencies changed.
Regarding the list of directories, then you can use some python programming, like this:
dirs = ['dir1', 'dir2', 'dir3']
for dir in dirs:
cmd = 'theScriptToExecute $SOURCE $TARGET'
env.Command(target = 'whatever', source = dir, action = cmd)
building libraries with waf is nice and I like the lib<targetname> naming scheme. But when I use is with boost::python, I'd like to get rid of it: I'd like the librarie's name to be like the target name. This is just a simple rename, I know, but: Can I tell waf to leave out putting lib before the target name (alternatively: specify an own name which stays untouched)?
Ok, got it. This feature can be enabled by using the python tool, found here: http://docs.waf.googlecode.com/git/apidocs_16/tools/python.html#module-waflib.Tools.python
The main point is calling conf.init_pyext() and in the build directive for the shared library specifying features='pyext':
def options(opt):
opt.load('python')
def configure(conf):
conf.load('python')
conf.check_python_version((2,4,2))
conf.check_python_headers()
def build(bld):
bld.shlib(
features = 'pyext',
source = "mymodule.cpp",
target = "myfoo",
use = "PYTHON BOOST_PYTHON")
Now, in the build directory there is a shared library called myfoo.so which can directly be imported.
How can I pass a user-defined parameter both from the command line and setup.cfg configuration file to distutils' setup.py script?
I want to write a setup.py script, which accepts my package specific parameters. For example:
python setup.py install -foo myfoo
As Setuptools/Distuils are horribly documented, I had problems finding the answer to this myself. But eventually I stumbled across this example. Also, this similar question was helpful. Basically, a custom command with an option would look like:
from distutils.core import setup, Command
class InstallCommand(Command):
description = "Installs the foo."
user_options = [
('foo=', None, 'Specify the foo to bar.'),
]
def initialize_options(self):
self.foo = None
def finalize_options(self):
assert self.foo in (None, 'myFoo', 'myFoo2'), 'Invalid foo!'
def run(self):
install_all_the_things()
setup(
...,
cmdclass={
'install': InstallCommand,
}
)
Here is a very simple solution, all you have to do is filter out sys.argv and handle it yourself before you call to distutils setup(..).
Something like this:
if "--foo" in sys.argv:
do_foo_stuff()
sys.argv.remove("--foo")
...
setup(..)
The documentation on how to do this with distutils is terrible, eventually I came across this one: the hitchhikers guide to packaging, which uses sdist and its user_options.
I find the extending distutils reference not particularly helpful.
Although this looks like the "proper" way of doing it with distutils (at least the only one that I could find that is vaguely documented). I could not find anything on --with and --without switches mentioned in the other answer.
The problem with this distutils solution is that it is just way too involved for what I am looking for (which may also be the case for you).
Adding dozens of lines and subclassing sdist is just wrong for me.
Yes, it's 2015 and the documentation for adding commands and options in both setuptools and distutils is still largely missing.
After a few frustrating hours I figured out the following code for adding a custom option to the install command of setup.py:
from setuptools.command.install import install
class InstallCommand(install):
user_options = install.user_options + [
('custom_option=', None, 'Path to something')
]
def initialize_options(self):
install.initialize_options(self)
self.custom_option = None
def finalize_options(self):
#print('The custom option for install is ', self.custom_option)
install.finalize_options(self)
def run(self):
global my_custom_option
my_custom_option = self.custom_option
install.run(self) # OR: install.do_egg_install(self)
It's worth to mention that install.run() checks if it's called "natively" or had been patched:
if not self._called_from_setup(inspect.currentframe()):
orig.install.run(self)
else:
self.do_egg_install()
At this point you register your command with setup:
setup(
cmdclass={
'install': InstallCommand,
},
:
You can't really pass custom parameters to the script. However the following things are possible and could solve your problem:
optional features can be enabled using --with-featurename, standard features can be disabled using --without-featurename. [AFAIR this requires setuptools]
you can use environment variables, these however require to be set on windows whereas prefixing them works on linux/ OS X (FOO=bar python setup.py).
you can extend distutils with your own cmd_classes which can implement new features. They are also chainable, so you can use that to change variables in your script. (python setup.py foo install) will execute the foo command before it executes install.
Hope that helps somehow. Generally speaking I would suggest providing a bit more information what exactly your extra parameter should do, maybe there is a better solution available.
I successfully used a workaround to use a solution similar to totaam's suggestion. I ended up popping my extra arguments from the sys.argv list:
import sys
from distutils.core import setup
foo = 0
if '--foo' in sys.argv:
index = sys.argv.index('--foo')
sys.argv.pop(index) # Removes the '--foo'
foo = sys.argv.pop(index) # Returns the element after the '--foo'
# The foo is now ready to use for the setup
setup(...)
Some extra validation could be added to ensure the inputs are good, but this is how I did it
A quick and easy way similar to that given by totaam would be to use argparse to grab the -foo argument and leave the remaining arguments for the call to distutils.setup(). Using argparse for this would be better than iterating through sys.argv manually imho. For instance, add this at the beginning of your setup.py:
argparser = argparse.ArgumentParser(add_help=False)
argparser.add_argument('--foo', help='required foo argument', required=True)
args, unknown = argparser.parse_known_args()
sys.argv = [sys.argv[0]] + unknown
The add_help=False argument means that you can still get the regular setup.py help using -h (provided --foo is given).
Perhaps you are an unseasoned programmer like me that still struggled after reading all the answers above. Thus, you might find another example potentially helpful (and to address the comments in previous answers about entering the command line arguments):
class RunClientCommand(Command):
"""
A command class to runs the client GUI.
"""
description = "runs client gui"
# The format is (long option, short option, description).
user_options = [
('socket=', None, 'The socket of the server to connect (e.g. '127.0.0.1:8000')',
]
def initialize_options(self):
"""
Sets the default value for the server socket.
The method is responsible for setting default values for
all the options that the command supports.
Option dependencies should not be set here.
"""
self.socket = '127.0.0.1:8000'
def finalize_options(self):
"""
Overriding a required abstract method.
The method is responsible for setting and checking the
final values and option dependencies for all the options
just before the method run is executed.
In practice, this is where the values are assigned and verified.
"""
pass
def run(self):
"""
Semantically, runs 'python src/client/view.py SERVER_SOCKET' on the
command line.
"""
print(self.socket)
errno = subprocess.call([sys.executable, 'src/client/view.py ' + self.socket])
if errno != 0:
raise SystemExit("Unable to run client GUI!")
setup(
# Some other omitted details
cmdclass={
'runClient': RunClientCommand,
},
The above is tested and from some code I wrote. I have also included slightly more detailed docstrings to make things easier to understand.
As for the command line: python setup.py runClient --socket=127.0.0.1:7777. A quick double check using print statements shows that indeed the correct argument is picked up by the run method.
Other resources I found useful (more and more examples):
Custom distutils commands
https://seasonofcode.com/posts/how-to-add-custom-build-steps-and-commands-to-setuppy.html
To be fully compatible with both python setup.py install and pip install . you need to use environment variables because pip option --install-option= is bugged:
pip --install-option leaks across lines
Determine what should be done about --(install|global)-option with Wheels
pip not naming abi3 wheels correctly
This is a full example not using the --install-option:
import os
environment_variable_name = 'MY_ENVIRONMENT_VARIABLE'
environment_variable_value = os.environ.get( environment_variable_name, None )
if environment_variable_value is not None:
sys.stderr.write( "Using '%s=%s' environment variable!\n" % (
environment_variable_name, environment_variable_value ) )
setup(
name = 'packagename',
version = '1.0.0',
...
)
Then, you can run it like this on Linux:
MY_ENVIRONMENT_VARIABLE=1 pip install .
MY_ENVIRONMENT_VARIABLE=1 pip install -e .
MY_ENVIRONMENT_VARIABLE=1 python setup.py install
MY_ENVIRONMENT_VARIABLE=1 python setup.py develop
But, if you are on Windows, run it like this:
set "MY_ENVIRONMENT_VARIABLE=1" && pip install .
set "MY_ENVIRONMENT_VARIABLE=1" && pip install -e .
set "MY_ENVIRONMENT_VARIABLE=1" && python setup.py install
set "MY_ENVIRONMENT_VARIABLE=1" && python setup.py develop
References:
How to obtain arguments passed to setup.py from pip with '--install-option'?
Passing command line arguments to pip install
Passing the library path as a command line argument to setup.py