I've been making a python project using pipenv, and I want to be able to run it in a terminal from any location on my (linux) system.
Specifically, say I have the following directory structure:
/home
/project
Pipfile
main.py
/other_dir
I would like to be able to make an alias that allows me to call main.py like so:
/home/other_dir$ alias_to_my_proyect --some args
and run it in the virtual env, having the same behaviour as
/home/project$ pipenv run python main.py
But in another directory.
If it weren't a pipenv project, I'd just use a shebang a the start of the file and then add an alias to it in my .bashrc, but I want to use pipenv's virtual environment, but I cant find a way to do this with pipenv.
If you want to use a specific python environment for your script you will need to point it to the interpreter of that environment. On Mac the default is that pipenv installs all virtualenvs to /Users/<user_name>/.local/share/virtualenvs/ however that can be set to different locations as described in the manual:
Pipenv automatically honors the WORKON_HOME environment variable, if you have it set — so you can tell pipenv to store your virtual environments wherever you want, e.g.:
export WORKON_HOME=~/.venvs
In addition, you can also have Pipenv stick the virtualenv in project/.venv by setting the PIPENV_VENV_IN_PROJECT environment variable.
You can find out where the exact location of the virtualenv is with pipenv --venv inside your project folder. It returns something like /Users/reedef/.local/share/virtualenvs/project-BpR9WgCa. The interpreter is in ./bin/python of that location.
If we assume that you did not set any environment variable and you are using Mac than that means that you can write a script:
#!/usr/bin/env sh
/Users/reedef/.local/share/virtualenvs/project-BpR9WgCa/bin/python /home/project/main.py
and place it somewhere in your $PATH, e.g. /usr/local/bin/my_fancy_main to let it run in that specific environment.
Note: as mentioned by #Jon in the comments, -BpR9WgCa at the end of the path is stable as it is made from the project path:
hash = hashlib.sha256(location.encode()).digest()[:6]
It should be the same as long as the project path hasn't changed.
You can just use
#!/usr/bin/env pipenv-shebang
in your script after you install my pipenv-shebang package:
pip install pipenv-shebang
You should use the standard setuptools library to write a setup.py file. In particular you can write an entry_points section that names your main script:
entry_points={
'console_scripts': [
'alias_to_my_project = project.main.main'
]
}
Once you've done this, you can activate and install your package into your virtual environment
pipenv install -e .
# or without pipenv
. ~/vpy/bin/activate
pip install -e .
This will create a wrapper script in $VIRTUAL_ENV/bin/alias_to_my_project that loads the project.main Python module and calls its main function.
The wrapper script knows about the virtual environment and can be called directly without specifically activating the virtual environment. So you can do something like
ln -s $VIRTUAL_ENV/bin/alias_to_my_project $HOME/bin/alias_to_my_project
PATH=$HOME/bin:$PATH
and it will always be available.
Related
I'm a newbie to python(3), but not to programming in general.
I'd like to distribute a git repo with myprogram consisting of these files:
requirements.txt
myprogram.py
lib/modulea.py
lib/moduleb.py
My question is: What is the best-practice and least surprising way to let users run myprogram.py using the dependencies in requirements.txt? So that after git clone, and some idiomatic installation command(s), ./myprogram.py or /some/path/to/myprogram.py "just works" without having to first set magical venv or python3 environment variables?
I want to be able to run it using the #! shebang so that /path/to/myprogram.py and double-clicking it from the file manager GUI does the correct thing.
I already know I can create a wrapper.sh or make a clever shebang line. But I'm looking for the best-practice approach, since I'm new to python.
More details
I'm guessing that users would
git clone $url workdir
cd workdir
python3 -m venv .
./bin/pip install -r requirements.txt
And from now on this uses the modules from requirements.txt:
./myprogram.py
If I knew that the project directory was always /home/peter/workdir, I could start the myprogram.py with:
#!/home/peter/workdir/bin/python3
but I'd like to avoid hard-coding the project directory in myprogram.py.
This also seems to work in my tiny demo, but clearly this is brittle and not best-practice, but it illustrates what I'm trying to do:
#!/usr/bin/env python3
import os
import sys
print(os.path.join(os.path.dirname(__file__), 'lib', 'python3.10', 'site-packages'))
I'm sure I could come up with some home-grown shebang line that works, but what is the idiomatic way to do this in python3?
Again: After pip install, I absolutely refuse to have to to set any environment variables or call any setup code in future shells before running myprogram.py. (Unless that strongly conflicts with "idiomatic", which I hope isn't the case)...
Expanding #sinoroc's comment into an answer:
I've looked at https://packaging.python.org/en/latest/tutorials/packaging-projects/ and also at "entrypoints", and this is the smallest example I can think of. Create an empty directory with these two files:
pyproject.toml:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "example_module_pmorch"
version = "0.0.1"
[project.scripts]
runme = "example_module_pmorch:cli_main"
src/example_module_pmorch/__init__.py:
def cli_main():
print("I'm the entrypoint")
Now if I run this:
$ python3 -m venv .
# Adding -e during development is optional
$ ./bin/pip install .
Then ./bin/runme does the right thing and prints I'm the entrypoint.
I do not see why you would need to hardcode anything. From your last snippet it looks like you are forcing the Python import system to include the target directory of the virtual environment you first create.
Based on your explanation, it seems you are using venv as your virtual environment manager. So long as your users install the dependencies in the virtual environment, and then activate the virtual environment before running the script, the dependencies will be ready for your module/script to use them.
This line you share: ./bin/pip install -r requirements.txt manually uses the package manager of the virtual environment you create with python3 -m venv .. Instead, you would want your user to create the environment (python3 -m venv example-env), activate the environment (source example-env/bin/activate) and then run the pip install command: python3 -m pip install -r requirements.txt.
The user of the package has to make sure that the environment is active before running the script.
I am trying to install a python package basicsr through requirements.txt file, and I would like to set an environment variable before doing so.
The usual pip installation goes like this :
BASICSR_EXT=True pip install basicsr
and I would like to translate this in the requirements.txt file.
Can anyone help?
I don't think it's possible in requirements.txt.
If you're using virtualenv (you should), try this tool for virtual environment management: https://virtualenvwrapper.readthedocs.io/en/latest/
It includes scripts, such as postactivate: https://virtualenvwrapper.readthedocs.io/en/latest/scripts.html?highlight=postactivate#postactivate, that are run when the virtual environment is activated.
The local $VIRTUAL_ENV/bin/postactivate script is sourced after the new environment is enabled. $VIRTUAL_ENV refers to the new environment at the time the script runs.
Basically put this in the $VIRTUAL_ENV/bin/postactivate file:
BASICSR_EXT=True
I'm developing my master thesis on a university's server, so I have my account and I can log in and do all the stuff I want if I remain inside /home/myname/.
I'm developing some python scripts and now I want to integrate python with the octave module, which is not currently installed on the system, and , of course, I cannot do anything with sudo apt-get install .
How can I overcome this problem without asking to my teacher?
thank you all,
Fabio
Please don't copy python and pip. You should use a virtualenv to install project-specific packages. This is particularly useful in your use-case where you can't install things at the system level. Even if you could, virtualenvs are recommended so the dependencies of each project are isolated.
Here is a quick primer that should get you going.
Create the virtualenv
virtualenv ~/project/env
Activate the virtualenv
source ~/project/env/bin/activate
This will modify your bash prompt by placing the name of your virtualenv in parenthesis to indicate that your virtualenv is activated.
(env) hostname:current_folder user$
Install Packages into the virtualenv
pip install -r requirements.txt
Use the virtualenv
python script.py
Use virtualenv by default in a script
script.py
#!~/project/env/bin/python
print('hello world!')
Then from the command line
chmod ugo+x script.py
./script.py
hello world!
Deactivate the virtualenv
deactivate
Make yourself a local copy of python and pip, then you can install whatever modules you want and not have to worry about getting a sysadmin to help you.
There are some good instructions here
Go here to get the link to the version of python you need and substitute it in the instructions above.
In your .bashrc add alias and path to your local copy - you may need to modify this for your own situation:
alias python="~/bin/python"
PATH=~/.local/bin:~/bin:$PATH
For the PATH - when you install local copies of modules through pip they by default go to ~/.local - change this if you prefer.
Begin your scripts with:
#/usr/bin/env python
so they use your preferred python version
I'm creating a Django app that requires me to use python2.7.6 . My system has python3.4.1 installed so I have to use a virtualenv with python2.7 installed. I installed such a virtualenv using Pycharm and named it django_python_2.7 but when I activate it in the terminal and run "python", it still shows that it's using system's python3.4.1:
here is what I did:
Activate the environment:
source django_python_2.7/bin/activate
Run python, and it shows:
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21) ---> this is the system level python and not the one installed in virtualenv
However, when I run which python, it shows the correct path that points to virtualenv's python version:
/Users/calvinmwhu/....../django_python_2.7/bin/python
When I explicitly run the python version installed in that virtualenv:
django_python_2.7/bin/python
it shows the correct version:
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
I have no idea what's going on. I'm developing this app in Pycharm IDE but I really like executing commands in the terminal . But in the terminal the virtualenv is not using the correct version of python..Why does running a simple "python" command in the virtualenv still default to the system's python ?
Could anyone provide some hints? Is it necessary to change the PATH variable to make it contain the path to the virtualenv's python?
If you want to change the PYTHONPATH used in a virtualenv, you can add the following line to your virtualenv's django_python_2.7/bin/activate file
export PYTHONPATH="/path/to/python"
export OLD_PYTHONPATH="$PYTHONPATH"
To restore to its original value on deactivate, you could add following line to your django_python_2.7/bin/postdeactivate script.
export PYTHONPATH="$OLD_PYTHONPATH"
Otherwise, create new env using
virtualenv -p /usr/bin/python2.7 django_python_2.7
I discovered the same problem...
and like #skyline75489 mentioned:
I forgot that i had stated an alias to my python3 executable a time ago.
I found it in my .bash files in my home directory and removed it.
Everything worked out fine again with my virtual environment.
If you changed the path to your venv or ranamed any of the parents folders of your venv directory, then this will break the configured paths, if that is case you have two options:
recreating it
Create a requirements.txt file using: pip freeze > requirements.txt
Delete the venv directory: rm -r old-vnev/
Create a new virtualenv with correct name: python -m venv new-venv
Activate new virtualenv: source new-venv/bin/activate
Install packages from requirements.txt: pip install -r requirements.txt
Another simpler way
search for all occurences of the string old/path/to/your/venv/
replace them with correct/path/to/your/venv/
after that source new-venv/bin/activate will work as intended again.
Hope this help!
In case it helps anyone else: if you changed the path to your venv folder (such as changing the parent folder), this will happen. This was my issue.
Recreating your virtualenv will fix it, as you should hopefully have a requirements.txt created to rebuild your virtualenv.
This might have even been the root cause for OP.
Double check your paths. I had an issue like this recently where running which python from within the activated virtualenv would still return the default system version (/usr/bin/python). However, if I ran the scripts specifying the binaries directly (./venv/bin/python, etc) from within the virtualenv, it worked as expected so it appeared all the dependencies had been installed correctly.
The issue was that I had moved the parent virtualenv directory after building everything. This meant all the virtualenv paths pointed to the original location which was no longer valid, and python correctly defaulted to the default system binary.
I use a bash script like this:
$ source venv/bin/activate
$ alias vpython=$VIRTUAL_ENV/bin/python3
and use vpython when wanting to use the python executable within the virtual environment. A nice way to check which executable you are actually using within python is the following:
>>> import sys
>>> print(f'executable \033[0;33;40m{sys.executable}\033[0m')
In my situation after system update symbolic link from the virtualenv was somehow broken and it switched to default system python version. The solution was to replace symbolic link by the correct one.
Deactivate virtual env if you are inside by:
deactivate
Change virtualenv python symbolic link:
ln -s /your/wanted/python/bin/python /your/virtualenv/bin/python
Start virtualenv again and it should use correct python version.
If you are not sure where is your python, then you can localise it by:
which python3
I had a similar problem. But I had it because I had moved my env folder to another place. So, if you did so, just go to activate file in bin folder and change VIRTUAL_ENV="CurrentPathToYourEnvFolder" (it's 40th line in file)
I'm using Python 3.4 on Windows.
I have created a virtual environment with
python c:\Python34\Tools\Scripts\pyvenv.py foo
Then activated it
foo\Scripts\activate.bat
And installed several libraries in it.
Question: Is it safe to rename the venv folder "foo" to "bar"?
I.e. once i have renamed foo to bar, then activate it with
bar\Scripts\activate.bat
will it still work?
What could make problems are any environment variable settings and absolute paths.
To be able to do that you should use virtualenv. From the docs:
Normally environments are tied to a specific path. That means that you cannot move an environment around or copy it to another computer. You can fix up an environment to make it relocatable with the command:
$ virtualenv --relocatable ENV
This will make some of the files created by setuptools or distribute use relative paths, and will change all the scripts to use activate_this.py instead of using the location of the Python interpreter to select the environment.
Note: you must run this after you've installed any packages into the environment. If you make an environment relocatable, then install a new package, you must run virtualenv --relocatable again.
But be aware:
The --relocatable option currently has a number of issues, and is not guaranteed to work in all circumstances. It is possible that the option will be deprecated in a future version of virtualenv.