I have the following configuration with hydra:
data_files: null
theta: 0.2
use_weights: true
I can override this as:
python -m apps.test_hydra data.data_files=x
However, when I try to specify a list as:
python -m apps.test_hydra data.data_files=[x,y]
Then I get the following error:
zsh: no matches found: data.data_files=[x,y]
I am not sure if this is some issue with the zsh shell or hydra issue
Wrap the string in quotes to avoid [ being interpreted as a glob expression.
python -m apps.test_hydra 'data.data_files=[x,y]'
On a hydra github issue, they use the following syntax. You might have to do something similar.
$ python -m apps.test_hydra 'data#data_files=[x, y]'
Use " \ " before special characters.
python -m apps.test_hydra data.data_files=\[x,y\]
I am trying to profile a reasonably sized python project using cprofile.
I am usually invoking my python app as a module something like:
python -m cabon.apps.detector -i input_image.png
Now howo can I use cprofile from the command line. I see that the cProfile itself is invoked as a module with -m and how can this be combined with my python app which is also invoked as a module?
The files cProfile and profile can also be invoked as a script to profile another script. For example:
python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py)
As mentioned in the documentation, we can specify that a module is being profiled by using additional -m. First for the cProfile and then the actual module:
python -m cProfile -m cabon.apps.detector -i input_image.png
Problem
I'm using python 3.6.6 on Fedora 28. I have a project structure as follows :
test/__init__.py
test/signal.py
test/notsignal.py
If I run $ python3 -m doctest -v test/signal.py
I get:
10 items had no tests:
signal
signal.Handlers
signal.ItimerError
signal.Sigmasks
signal.Signals
signal._enum_to_int
signal._int_to_enum
signal.struct_siginfo
signal.struct_siginfo.__reduce__
signal.struct_siginfo.__repr__
0 tests in 10 items.
0 passed and 0 failed.
Test passed.
which, to me, clearly shows that doctest is trying to run on the built-in signal module. By renaming the file I was able to run docset. Am I missing something or is this a bug?
To reproduce
You may use the following shell script.
#!/bin/sh
mkdir -p test
touch test/__init__.py
echo -e ""'"'""'"'""'"'"\n>>> _ = print(f'Doctest at {__name__} was run.')\n"'"'""'"'""'"'"" > test/signal.py
cp test/signal.py test/notsignal.py
python3 -m doctest -v test/signal.py
python3 -m doctest -v test/notsignal.py
If you look at the doctest source, you can see that doctest tries to import the modules that you pass to it.
It's very likely that the standard library's signal module has already been imported:
$ python -c 'import sys;import doctest;print("signal" in sys.modules)'
True
When doctest tries the import the Python interpreter finds that there is already a module named "signal" in sys.modules and returns that rather than your signal module.
Perhaps this is a bug - maybe doctest could be smarter about how it imports - but in practice I think the best course of action is to rename your module. In general, having modules with the same names as standard library modules almost always causes problems.
According to this answer you can import pip from within a Python script and use it to install a module. Is it possible to do this with conda install?
The conda documentation only shows examples from the command line but I'm looking for code that can be executed from within a Python script.
Yes, I could execute shell commands from within the script but I am trying to avoid this as it is basically assuming that conda cannot be imported and its functions called.
You can use conda.cli.main. For example, this installs numpy:
import conda.cli
conda.cli.main('conda', 'install', '-y', 'numpy')
Use the -y argument to avoid interactive questions:
-y, --yes Do not ask for confirmation.
I was looking at the latest Conda Python API and noticed that there are actually only 2 public modules with “very long-term stability”:
conda.cli.python_api
conda.api
For your question, I would work with the first:
NOTE: run_command() below will always add a -y/--yes option (i.e. it will not ask for confirmation)
import conda.cli.python_api as Conda
import sys
###################################################################################################
# The below is roughly equivalent to:
# conda install -y 'args-go-here' 'no-whitespace-splitting-occurs' 'square-brackets-optional'
(stdout_str, stderr_str, return_code_int) = Conda.run_command(
Conda.Commands.INSTALL, # alternatively, you can just say "install"
# ...it's probably safer long-term to use the Commands class though
# Commands include:
# CLEAN,CONFIG,CREATE,INFO,INSTALL,HELP,LIST,REMOVE,SEARCH,UPDATE,RUN
[ 'args-go-here', 'no-whitespace-splitting-occurs', 'square-brackets-optional' ],
use_exception_handler=True, # Defaults to False, use that if you want to handle your own exceptions
stdout=sys.stdout, # Defaults to being returned as a str (stdout_str)
stderr=sys.stderr, # Also defaults to being returned as str (stderr_str)
search_path=Conda.SEARCH_PATH # this is the default; adding only for illustrative purposes
)
###################################################################################################
The nice thing about using the above is that it solves a problem that occurs (mentioned in the comments above) when using conda.cli.main():
...conda tried to interpret the comand line arguments instead of the arguments of conda.cli.main(), so using conda.cli.main() like this might not work for some things.
The other question in the comments above was:
How [to install a package] when the channel is not the default?
import conda.cli.python_api as Conda
import sys
###################################################################################################
# Either:
# conda install -y -c <CHANNEL> <PACKAGE>
# Or (>= conda 4.6)
# conda install -y <CHANNEL>::<PACKAGE>
(stdout_str, stderr_str, return_code_int) = Conda.run_command(
Conda.Commands.INSTALL,
'-c', '<CHANNEL>',
'<PACKAGE>'
use_exception_handler=True, stdout=sys.stdout, stderr=sys.stderr
)
###################################################################################################
Having worked with conda from Python scripts for a while now, I think calling conda with the subprocess module works the best overall. In Python 3.7+, you could do something like this:
import json
from subprocess import run
def conda_list(environment):
proc = run(["conda", "list", "--json", "--name", environment],
text=True, capture_output=True)
return json.loads(proc.stdout)
def conda_install(environment, *package):
proc = run(["conda", "install", "--quiet", "--name", environment] + packages,
text=True, capture_output=True)
return json.loads(proc.stdout)
As I pointed out in a comment, conda.cli.main() was not intended for external use. It parses sys.argv directly, so if you try to use it in your own script with your own command line arguments, they will get fed to conda.cli.main() as well.
#YenForYang's answer suggesting conda.cli.python_api is better because this is a publicly documented API for calling conda commands. However, I have found that it still has rough edges. conda builds up internal state as it executes a command (e.g. caches). The way conda is usually used and usually tested is as a command line program. In that case, this internal state is discarded at the end of the conda command. With conda.cli.python_api, you can execute several conda commands within a single process. In this case, the internal state carries over and can sometimes lead to unexpected results (e.g. the cache becomes outdated as commands are performed). Of course, it should be possible for conda to handle this internal state directly. My point is just that using conda this way is not the main focus of the developers. If you want the most reliable method, use conda the way the developers intend it to be used -- as its own process.
conda is a fairly slow command, so I don't think one should worry about the performance impact of calling a subprocess. As I noted in another comment, pip is a similar tool to conda and explicitly states in its documentation that it should be called as a subprocess, not imported into Python.
I found that conda.cli.python_api and conda.api are limited, in the sense that, they both don't have the option to execute commands like this:
conda export env > requirements.txt
So instead I used subprocess with the flag shell=True to get the job done.
subprocess.run(f"conda env export --name {env} > {file_path_from_history}",shell=True)
where env is the name of the env to be saved to requirements.txt.
The simpler thing that i tried and worked for me was :
import os
try:
import graphviz
except:
print ("graphviz not found, Installing graphviz ")
os.system("conda install -c anaconda graphviz")
import graphviz
And make sure you run your script as admin.
Try this:
!conda install xyzpackage
Please remember this has to be done within the Python script not the OS prompt.
Or else you could try the following:
import sys
from conda.cli import main
sys.exit(main())
try:
import conda
from conda.cli import main
sys.argv = ['conda'] + list(args)
main()
This might be an easy question but I don't know the name of what I'm trying to do, so I don't know how to search for it.
Basically when I'm in terminal (linux command line) and I type
$ python do_something.py stuff
I want to get the stuff to mean something for my script. So two questions:
How is this called?
How can I do it?
The simplest way is for the do_something.py script to import sys and access the "stuff" command-line argument as sys.argv(1). There are many fancier ways, of course.
What you're asking for is called argument parsing.
To do this the proper way, you should definitively use argparse.
It's a neat and yet very powerful library to make argument parsing more efficient. Plus, it makes your scripts manage arguments the proper Linux way, by default.
Basic example:
import argparse
parser = argparse.ArgumentParser(description='My argparse program')
parser.add_argument('--verbose',
action='store_true',
help='sets output to verbose' )
args = parser.parse_args()
if args.verbose:
print("~ Verbose!")
else:
print("~ Not so verbose")
Then you can do cool stuff like:
$ python3 myscript.py --verbose
~ Verbose!
And what's even cooler, it provides an automatic --help (or -h) argument:
$ python3 myscript.py --help
usage: myscript.py [-h] [--verbose]
My argparse program
optional arguments:
-h, --help show this help message and exit
--verbose sets output to verbose
This is the kind of library that allows you to quite easily do complicated stuff like:
./myscript.py --password=no -c -o --keep_moving --name="Robert"
Here is a link to a nice tutorial from which the above example has been freely adapted.