nosetest --with-profile --profile-stats-file output
The output can't read by runsnake, because nosetest uses hotshot, if I want to generate a file that can be read with runsnake, I need to convert it so:
st = hotshot.stats.load('output')
st.dump_stats('output_new')
Could I run the test with cProfile directly for read with runsnake?
Evolving on the answer of #squid, you can use a nose plugin called nose-cprof to replace the nose default profiler, hotshot, with cProfile.
To install it:
pip install nose-cprof
Then call nose like this:
nosetests --with-cprofile
It should generate a cProfile output file, that you can then analyze with tools like runsnakerun.
#cihanpesend's answer didn't quite work for me (cProfile couldn't find 'nosetests'), but I did have success on Linux using:
python -m cProfile -o profile.out `which nosetests` .
The resulting output works fine in runsnake.
(Presumably on Windows you could replace which nosetests with the hard-coded path to your nosetests top-level python script.)
I think you are right that the output from nosetests' hotshot profiler is not compatible with runsnake. Certainly the two don't play nice together out of the box for me either.
I don't have info about nosetest except it is python project. So;
python -m cProfile -o outputfile nosetest
Then,
runsnake outputfile
RunSnakeRun is extremly useful to visualize profiler.
Note: to run runsnake, you must install wx and numpy.
update: from omikron's comment; runsnakerun can not support python3 profile output. (i didn't try)
Or you can try nose-cprof plugin: https://github.com/msherry/nose-cprof
It replaces hotshot with cProfile
With pyprof2calltree:
$ pip install pyprof2calltree
$ nosetests --with-cprofile --profile-stats=profile.out tests/
$ pyprof2calltree -i profile.out -k
With xdot:
$ sudo apt install xdot
$ gprof2dot -f pstats profile.out | dot -Tpng -o profile.png
Related
I have below project structure in Pycharm.
Project Folder: PythonTutorial
Package: pytestpackage
Python Files: test_conftest_demo1.py, test_conftest_demo2.py
I'm trying to run the above 2 python files having almost similar name using pytest from command prompt with the below command. But I'm facing the below issue. Please help me on the same.
Note: I'm using windows 10 operating system .
Command Used:
py.test -s -v test_conftest_demo*.py
use the -k option to specify substring matching.
$ pytest -s -v -k "test_conftest_demo"
When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py
Trying to install easy_install (IRONIC) and it is not wanting to cooperate with me.
I am running the following command in terminal with python2.7
curl https://bootstrap.pypa.io/ez_setup.py -o - | python
returns me -->
-bash: https://bootstrap.pypa.io/ez_setup.py: No such file or directory
is it a simple fix?
This should work:
curl https://bootstrap.pypa.io/ez_setup.py -s | python
Unless redirected otherwise, cURL will output the received body to STDOUT, so you do not have to pass it -o - (which really should not do anything according to the docs). You should pass it -s in order to suppress the progress meter and any errors. Then pipe it directly into python. Given no other arguments, python will determine STDIN as the source of the script to execute.
I've been usually installed python packages through pip.
For Google App Engine, I need to install packages to another target directory.
I've tried:
pip install -I flask-restful --target ./lib
but it fails with:
must supply either home or prefix/exec-prefix -- not both
How can I get this to work?
Are you using OS X and Homebrew? The Homebrew python page https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md calls out a known issue with pip and a work around.
Worked for me.
You can make this "empty prefix" the default by adding a
~/.pydistutils.cfg file with the following contents:
[install]
prefix=
Edit: The Homebrew page was later changed to recommend passing --prefix on the command line, as discussed in the comments below. Here is the last version which contained that text. Unfortunately this only works for sdists, not wheels.
The issue was reported to pip, which later fixed it for --user. That's probably why the section has now been removed from the Homebrew page. However, the problem still occurs when using --target as in the question above.
I believe there is a simpler solution to this problem (Homebrew's Python on macOS) that won't break your normal pip operations.
All you have to do is to create a setup.cfg file at the root directory of your project, usually where your main __init__.py or executable py file is. So if the root folder of your project is: /path/to/my/project/, create a setup.cfg file in there and put the magic words inside:
[install]
prefix=
OK, now you sould be able to run pip's commands for that folder:
pip install package -t /path/to/my/project/
This command will run gracefully for that folder only. Just copy setup.cfg to whatever other projects you might have. No need to write a .pydistutils.cfg on your home directory.
After you are done installing the modules, you may remove setup.cfg.
On OSX(mac), assuming a project folder called /var/myproject
cd /var/myproject
Create a file called setup.cfg and add
[install]
prefix=
Run pip install <packagename> -t .
Another solution* for Homebrew users is simply to use a virtualenv.
Of course, that may remove the need for the target directory anyway - but even if it doesn't, I've found --target works by default (as in, without creating/modifying a config file) when in a virtual environment.
*I say solution; perhaps it's just another motivation to meticulously use venvs...
I hit errors with the other recommendations around --install-option="--prefix=lib". The only thing I found that worked is using PYTHONUSERBASE as described here.
export PYTHONUSERBASE=lib
pip install -I flask-restful --user
this is not exactly the same as --target, but it does the trick for me in any case.
As other mentioned, this is known bug with pip & python installed with homebrew.
If you create ~/.pydistutils.cfg file with "empty prefix" instruction it will fix this problem but it will break normal pip operations.
Until this bug is officially addressed, one of the options would be to create your own bash script that would handle this case:
#!/bin/bash
name=''
target=''
while getopts 'n:t:' flag; do
case "${flag}" in
n) name="${OPTARG}" ;;
t) target="${OPTARG}" ;;
esac
done
if [ -z "$target" ];
then
echo "Target parameter must be provided"
exit 1
fi
if [ -z "$name" ];
then
echo "Name parameter must be provided"
exit 1
fi
# current workaround for homebrew bug
file=$HOME'/.pydistutils.cfg'
touch $file
/bin/cat <<EOM >$file
[install]
prefix=
EOM
# end of current workaround for homebrew bug
pip install -I $name --target $target
# current workaround for homebrew bug
rm -rf $file
# end of current workaround for homebrew bug
This script wraps your command and:
accepts name and target parameters
checks if those parameters are empty
creates ~/.pydistutils.cfg file with "empty prefix" instruction in it
executes your pip command with provided parameters
removes ~/.pydistutils.cfg file
This script can be changed and adapted to address your needs but you get idea. And it allows you to run your command without braking pip. Hope it helps :)
If you're using virtualenv*, it might be a good idea to double check which pip you're using.
If you see something like /usr/local/bin/pip you've broken out of your environment. Reactivating your virtualenv will fix this:
VirtualEnv: $ source bin/activate
VirtualFish: $ vf activate [environ]
*: I use virtualfish, but I assume this tip is relevant to both.
I have a similar issue.
I use the --system flag to avoid the error as I decribe here on other thread where I explain the specific case of my situation.
I post this here expecting that can help anyone facing the same problem.
I am working on a tdd project in python and I am looking for a quick way to run all unit tests in my t/ directory. In perl this is easy:
$ prove -lvr t/
I am looking for the python equivalent. It does not seem that nose has this functionality. I rolled a command line statement to do something like this:
for x in `find t/ | grep py`; do echo $x && python $x ; done
But this lacks flags like -l (include the local lib dir) and -v (verbose). Does this or does this not exist in python? I want a one liner like this:
$ pyprove -lvr t/
You can do this by running python -m unittest discover -s t/, which will discover your unit tests and run them for you. It looks like there's a verbose flag, but I don't see a flag to include the local lib dir.
Try nose. Add -v for verbosity, -w to specify search directories (ref: usage).
nosetests -v -w t/