I am trying to build a minimal docker image, capable of nothing more but running the Python interpreter. During development I start from alpine, the final image will be build FROM scratch. I am working on a Linux Mint machine.
I am able to compile the Python compiler and install it into my current working directory like this:
cd Python-3.8.7
./configure --prefix=$PWD/../python
make install
However, I did not find out how to tweak the --prefix settings correctly so that the created shebangs will later work in the docker container.
First line of ./python/pip3 contains the absolute path of my host and reads
#!/home/orion/minimal_py_image/Python-3.8.7/../python/bin/python3.8
But it should read
#!/python/bin/python3.8
because /python is the location under which the Python interpreter will be found in the docker image.
How can I trick the make install script so that the final destination will be /python/bin?
I would like to keep the build contained in the current directory i.e. not using the folder /python on the host where I do the compilation.
Additional Information
Probably not directly relevant for this question but as reference: Here is the Dockerfile I am trying to get working:
FROM alpine
COPY python /python
COPY lib64/* /lib64/
ENV LD_LIBRARY_PATH=/usr/lib64/:/lib64/:/python/lib
ENV PATH="${PATH}:/python/bin"
I am capable of running Python already with docker run -it mini python3 -c "print('hello from python')" but pip is not working yet due-to the wrong shebang.
A common convention in Autoconf-based build systems is to support a Make variable DESTDIR. When you run make install, if DESTDIR is set, it actually installs into the configured directory under DESTDIR, but still built with the original path. You can then create an archive of the target directory, or in a Docker context, use that directory as the build context.
cd Python-3.8.7
# Use the final install target as the --prefix
./configure --prefix=/python
# Installs into e.g. ../python/python/bin/python3.8
make install DESTDIR=../python
cd ../python
tar cvzf ../python.tar.gz .
You can see this variable referenced in the Python Makefile in many places.
I have a virtualenv located at /home/user/virtualenvs/Environment. Now I need this environment at another PC. So I installed virtualenv-clone and used it to clone /Environment. Then I copied it to the other PC via USB. I can activate it with source activate, but when I try to start the python interpreter with sudo ./Environment/bin/python I get
./bin/python: 1: ./bin/python: Syntax Error: "(" unexpected
Executing it without sudo gives me an error telling me that there is an error in the binaries format.
But how can this be? I just copied it. Or is there a better way to do this? I can not just use pip freeze because there are some packages in /Environment/lib/python2.7/site-packages/ which I wrote myself and I need to copy them, too. As I understand it pip freeze just creates a list of packages which pip then downloads and installs.
Do the following steps on the source machine:
workon [environment_name]
pip freeze > requirements.txt
copy requirements.txt to other PC
On the other PC:
create a virtual environment using mkvirtualenv [environment_name]
workon [environment_name]
pip install -r requirements.txt
You should be done.
Other Resources:
How to Copy/Clone a Virtual Environment from Server to Local Machine
Pip Freeze Not Applicable For You?
Scenario: you have libraries installed on your current system that are very hard to migrate using pip freeze and am talking really hard, because you have to download and install the wheels manually such as gdal, fiona, rasterio, and then even doing so still causes the project to crash because possibly they were installed in the wrong order or the dependencies were wrong and so on.
This is the experience I had when I was brought on board a project.
For such a case, when you finally get the environment right you basically don't want to go through the same hell again when you move your project to a new machine. Which I did, multiple times. Until finally I found a solution.
Now, disclaimer before I move on:
I don't advocate for this method as the best, but it was the best for my case at the time.
I also cannot guarantee it will work when switching between different OSes as I have only tried it between Windows machine. In fact I don't expect it to work when you move from Windows to other OSs as the structure of the virtualenv folder from Unix-based OS is different from that of Windows.
Finally, the best way to do all of this is to use Docker. My plan is to eventually do so. I have just never used Docker for a non-web-app project before and I needed a quick fix as my computer broke down and the project could not be delayed. I will update this thread when I can once I apply Docker to the project.
THE HACK
So this is what I did:
Install the same base Python on your new machine. If you have 3.9 on the old, install 3.9 on the new one and so on. Keep note of where the executable can be located, usually something like C:\Users\User\Appdata\Local\Programs\Python\PythonXX
Compress your virtual env folder, copy it into the project directory
inside your new machine. Extract all files there
Using text editor of your choice, or preferably IDE, use the 'Search
in all files' feature to look for all occurrences of references to
your old machine paths: C:\Users*your-old-username*
Replace these with your new references. For my case I had to do it in
the following files inside the virtual env folder: pyvenv.cfg, Scripts/activate, Scripts/activate.bat, Scripts/activate.fish and Scripts/activate.nu.
And that's it!
Good luck everyone.
I think what occurs is that you just copy the symbolic links in the source file to the target machine as binary files(no longer links). You should copy it using rsync -l to copy to keep those links.
Usually I use virtualenv to create a new environment, then I go to the environment where I want to copy from, copy all the folders and paste it into the environment folder I just created, but most importantly when asking if you want to replace the Destination files, choose to skip these files. This way you keep your settings.
At least for me, this has worked very well.
I hope it works for you too.
I share my experience.
Suppose another PC does not install Python
Python version: 3.7.3
Platform: Platform: Windows 10, 7 (64bit)
The following is working for me.
Step:
download Windows embeddable zip file
download get-pip.py (because the embeddable zip file does not provide pip)
[Optional] install tkinter, see this article: Python embeddable zip: install Tkinter
Choose a packaging method (I use NSIS: https://nsis.sourceforge.io/Download)
folder artictures:
- main.nsi
- InstallData
- contains: Step1 & Step2
- YourSitePackages # I mean that packages you do not intend to publish to PyPI.
- LICENSE
- MANIFEST.in
- README.rst
- ...
- requirements.txt
- setup.py
The abbreviated content about main.nsi is as follows:
!define InstallDirPath "$PROGRAMFILES\ENV_PYTHON37_X64"
!define EnvScriptsPath "${InstallDirPath}\Scripts"
...
CreateDirectory "${InstallDirPath}" # Make sure the directory exists before the writing of Uninstaller. Otherwise, it may not write correctly!
SetOutPath "${InstallDirPath}"
SetOverwrite on
File /nonfatal /r "InstallData\*.*"
SetOutPath "${InstallDirPath}\temp"
SetOverwrite on
File /nonfatal /r "YourSitePackages\*.*"
nsExec::ExecToStack '"${InstallDirPath}\python.exe" "${InstallDirPath}\get-pip.py"' # install pip
nsExec::ExecToStack '"${InstallDirPath}\Scripts\pip.exe" install "${InstallDirPath}\temp\."' # install you library. same as: `pip install .`
RMDir /r "${InstallDirPath}\temp" # remove source folder.
...
/*
Push ${EnvScriptsPath} # Be Careful about the length of the HKLM.Path. it is recommended to write it to the HKCU.Path, it is difficult for the user path to exceed the length limit
Call AddToPath # https://nsis.sourceforge.io/Path_Manipulation
*/
hope someone will benefit from this.
This question is not a duplicate.
It pertains not just to renaming a virtual environment, but to actually moving it to a different directory, including, potentially, a different user's directory.
This is not the same as merely renaming a virtual environment, especially to people unfamiliar with virtualenvs.
If I create a virtualenv, and I move it to a different folder, will it still work?
$ virtualenv -p /usr/bin/python3 /home/me/Env/my-python-venv
$ source Env/my-python-venv/bin/activate
(my-python-venv) $
...later that day, the virtual environment MOVED...
(my-python-venv) $ deactivate
$ mkdir -p /home/me/PeskyPartyPEnvs
$ mv /home/me/Env/my-python-venv /home/me/PeskyPartyPEnvs/
Question:
Will this work?
$ source /home/me/PeskyPartyPEnvs/my-python-venv/bin/activate
(my-python-venv) $ /home/me/PeskyPartyPEnvs/my-python-venv/bin/pip3 install foaas
I mean this as less of a question about the wisdom of trying this (unless that wisdom is humorous, of course), and more about whether it's possible. I really want to know whether it's possible to do in Python 3, or whether I just have to suck it up and clone it.
Can I just mv a virtualenv like that without sadness? I do want to avoid sadness.
Yes. It is possible to move it on the same platform. You can use --relocatable on an existing environment.
From --help:
--relocatable -- Make an EXISTING virtualenv environment relocatable.
This fixes up scripts and makes all .pth files relative.
HOWEVER, this does NOT seem to change the activate script, and rather only changes the pip* and easy_install* scripts. In the activate script, the $VIRTUAL_ENV environment variable hardcoded as the original /path/to/original/venv. The $VIRTUAL_ENV variable is used to set the PATH of your active environment too, so it must be changed based on the new location in order to call python and pip etc. without absolute path.
To fix this issue, you can change the $VIRTUAL_ENV environment variable in the activate script (for example using sed), and everything should be good to go.
An example of usage:
$ cd ~/first
$ virtualenv my-venv
$ grep 'VIRTUAL_ENV=' my-venv/bin/activate
VIRTUAL_ENV="/home/username/first/my-venv"
$ virtualenv --relocatable my-venv
Making script my-venv/bin/easy_install relative
Making script my-venv/bin/easy_install-2.7 relative
Making script my-venv/bin/pip relative
Making script my-venv/bin/pip2 relative
Making script my-venv/bin/pip2.7 relative
### Note that `activate` has not been touched
$ mkdir ~/second
$ mv my-venv ~/second
$ cd ~/second
$ grep 'VIRTUAL_ENV=' my-venv/bin/activate
VIRTUAL_ENV=/home/username/first/my-venv
### (This variable hasn't been changed, it still refers to the old, now non-existent directory!)
$ sed -i -e 's|username/first|username/second|' my-venv/bin/activate
## sed can be used to change the path.
## Note that the `-i` (in place) flag won't work on all machines.
$ source my-venv/bin/activate
(my-venv) $ pip install foass
...
(my-venv) $ python
[...]
> import foass
Hooray, now you can install things and load them into your newly located virtual environment.
For Python 3.3+ (with new venv built-in module)
Short Answer (regardless of version):
There's no clean, direct way to move a virtual environment
Just recreate, it's easy!!
Long Answer:
As of Python v3.3, the virtualenv package has become a built-in module named venv.
The --relocatable option mentioned in other answers has not been included in venv, and currently there is no good, safe way that I'm aware of to either rename or relocate a Python virtual environment.
However, it is fairly simple to recreate a virtual environment, with all its currently installed packages. See this answer, or see the section below. During the process you can recreate the new environment in whatever location and with whatever name you desire.
In the answer linked above, he mentions some 3rd party packages which may support direct renames or moves. If you are settled on pursuing a way to move a virtual environment, you could look into if those work with venv as well.
Note: In that answer, it is focused on virtualenv, rather than venv. See next section for how to translate.
venv vs. older virtualenv command syntax
The command to use venv is:
python -m venv
rather than just virtualenv, which installs as a command in the original package. Where "python" refers to however you run your python executable, which could be a variety of things, such as:
python
py or py -3.7 or similar (the Python Launcher for Windows for Python 3.3+ and bundled with Python for Windows, or the py package that can be installed separately for Linux [and MacOS?])
python3 (convention for linux environments that dual install python 2 and 3)
If you are having issues, use the absolute path to the python executable you want to run: e.g. c:\program files\python37\python.exe
If you are unsure which version is being run, you can always python --version to find out.
How to recreate a virtual environment
Creating/recreating a virtual environment is easy and should become second nature after you work with them for a bit. This process mirrors what you would do to distribute your script as a package (with it's dependencies) in the first half, and then what someone would do to install your script/package for further development.
First, get an updated list of what is in the virtual environment. With it active, get the Python version it uses and save out the list of dependencies to a file.
Use python --version with the virtual environment activated to see what version of Python it is using.
This is for clarity - you may want to update the Python version for various reasons - at least to the latest patch version
For example, if the existing venv is using Python v3.7.4, but now v3.7.6 is out - use v3.7.6 instead, which should including only non-breaking security and bug fixes.
Use python -m pip freeze > requirements.txt to create the list of current package dependencies and put them into the requirements.txt file. This command works in Linux or the Git Bash for sure - not 100% sure about Powershell or Command Line in Windows.
Now create a new virtual environment and then add the dependencies from the old one.
Make your new venv.
Make sure you are using the correct version of python that you want to install to the venv.
If you want it to be exactly the same Python version:
While in the old venv, type "python --version", then make sure you create the new venv with that version of the python command.
For the new venv folder entry in the command:
Either add an absolute or relative path to the desired final folder location.
Use python -m venv my_new_venv to create a new virtual environment in the current working directory in a new my_new_venv folder.
The name of the venv folder will be the name of the venv (what shows up in the prompt when it is activated).
Install your dependencies from the requirements.txt file.
python -m pip install -r requirements.txt
You might need to reinstall local packages that are in development mode.
Note, if you ever need to see the specific location a package is installed to, use:
python -m pip list -v
The -v or "verbose" option will add some extra information about each package that is installed, including the path it is installed in. This is useful to make sure you are keeping virtual, user, and system installed packages straight.
At this point you can just delete the old venv folder and all contents. I recommend using a GUI for that - file deletions are often permanent from the linux command line, and a small typo can be bad news.
BUT ALAS:
No, you can't simply mv. There are workarounds, but it might be easier to reinstall.
(my-python-venv)$ /home/me/PeskyPartyPEnvs/pip3 install foaas
zsh: /home/me/PeskyPartyPEnvs/pip3: bad interpreter: /home/me/Env/my-python-venv/bin/python3: no such file or directory
(my-python-venv)$ deactivate
$
... presses enter a lot in frustration, and the following works
$
$
$ pip3 search foaas
Except it is not from my-python-venv, ergo sadness.
Want to mv your virtualenv and use it, otherwise unmodified?
Short Answer:
Well, ya can't.
The --relocatable argument to virtualenv appears to allow you to do this.
YES, YOU CAN! (In windows)
The workaround is easy, just move your virtual environment anywhere then edit activate.bat inside scripts\:
Move to the virtual environment to the desired directory
Right-click and edit activate.bat located at venv_folder\scripts.
Change VIRTUAL_ENV variable from:
set VIRTUAL_ENV=C:\old_directory\venv_name
into
set VIRTUAL_ENV=C:\new_directory\venv_name
Save the edited batch file, and thats it!
NOTE: My solution should work and save windows users setting up new virtual environments, I doubt this will work in other operating system since .bat is from MS-DOS
Yes, this should be possible if you haven't done anything that depends on the current directory of the virtualenv.
However, if you have the choice, the best thing to do is to create new virtualenv and start using the new virtualenv instead. This is the safest choice and least likely to cause issues later.
The documentation does mention that:
Each virtualenv has path information hard-coded into it,
For example, if you have run setvirtualenvproject then it won't be able to switch to the right directory after you run workon ... so in that case you'd need to fix that manually.
In general a virtualenv is little more than a directory with the necessary Python interpreter files plus packages that you need.
Using answers of this and other threads about similar topic, I've made a bash script that, located and executed within the virtualenv directory itself, will help with your virtualenv moves.
After doing virtualenv --relocatable yourenv you'll need to change your VIRTUAL_ENV variable every time you move the directory, so if you don't wan't to change it manually, use this.
#!/bin/bash \n
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
EXISTING=$(grep 'VIRTUAL_ENV=' bin/activate)
NEWDIR=VIRTUAL_ENV=\"$DIR\"
sed -i -e "s|$EXISTING|$NEWDIR|" bin/activate
source bin/activate
I hope it helps.
I wrote a venv-move script.
The first argument is the path to the venv. It deletes any __pycache__ under that path.
It detects the old path, and replaces it with the current path, after confirming. It seems to work okay, even when moving to a different machine of the same type.
It would make sense to re-write this in Python, but the program would be longer.
#!/bin/bash -eu
venv=$1
old=`perl -ne '/VIRTUAL_ENV="(.*?)"/ && print "$1\n"' "$venv/bin/activate"`
new=$PWD/$venv
find "$venv" -name __pycache__ | xargs rm -rf --
files=`fgrep -r "$old" "$venv" -l`
echo "replace $old with $new in:"
echo "$files"
read -p "[yn] ? " YN
[ "$YN" = y ]
sed -i "s:$old:$new:g" $files
TL;DR
virtualenv-clone is included part of virtualenvwrapper
virtualenv-clone /path/to/old/venv /path/to/new/venv
Alternatively
You could also try cpvirtualenv
cpvirtualenv /path/to/old/venv /path/to/new/venv
But cpvirtualenv expects the /path/to/old/venv to be existing inside $WORKON_HOME and if it isn't it fails. Since this calls virtualenv-clone you may as well use that instead; to avoid errors like
mark#Desktop:~/venvs$ cpvirtualenv ./random/ $WORKON_HOME/random
Copying random as /home/mark/.virtualenvs/venvs/random...
Usage: virtualenv-clone [options] /path/to/existing/venv /path/to/cloned/venv
virtualenv-clone: error: src dir '/home/mark/.virtualenvs/venvs/random' does not exist
Warning as per virtualenvwrapper documentation
Copying virtual environments is not well supported. Each virtualenv
has path information hard-coded into it, and there may be cases where
the copy code does not know it needs to update a particular file. Use
with caution.
What does it actually do ?
As per virtualenv-clone PyPi page
A script for cloning a non-relocatable virtualenv.
Virtualenv provides a way to make virtualenv's relocatable which could
then be copied as we wanted. However making a virtualenv relocatable
this way breaks the no-site-packages isolation of the virtualenv as
well as other aspects that come with relative paths and /usr/bin/env
shebangs that may be undesirable.
Also, the .pth and .egg-link rewriting doesn't seem to work as
intended. This attempts to overcome these issues and provide a way to
easily clone an existing virtualenv.
It performs the following:
copies sys.argv[1] dir to sys.argv[2]
updates the hardcoded VIRTUAL_ENV variable in the activate script to
the new repo location. (--relocatable doesn't touch this)
updates the shebangs of the various scripts in bin to the new Python
if they pointed to the old Python. (version numbering is retained.)
it can also change /usr/bin/env python shebangs to be absolute too,
though this functionality is not exposed at present.
checks sys.path of the cloned virtualenv and if any of the paths are
from the old environment it finds any .pth or .egg link files within
sys.path located in the new environment and makes sure any absolute
paths to the old environment are updated to the new environment.
finally it double checks sys.path again and will fail if there are
still paths from the old environment present.
NOTE: This script requires Python 2.7 or 3.4+
I have created my program using virtual env. It is working in my project folder fine. Now i need to take this program and release it to the production environment that is supposed to be accessible by everybody.So this program should be runnable as is or it might be incorporated into other programs as a step. How am i supposed to deploy it? Zip the whole project folder? Is it possible to do without requiring clients to copy it and then unzip and run? Or the only way is to create a commonly accessible script that automates unzipping of the thing and configuring virtual env and then running it or there is a smarter way?
More complicated scenario is when it supposed to be used as library. How to deploy it so others could specify it as their dependency and pick it up? Seems like the only way is to create your own PyPi-like local repository - is that correct?
Thanks!
So here is what i have found:
If we have a project A as API:
create a folder where you will store the wheels (~/wheelhouse)
using pip config specify this folder as one to find links in http://www.pip-installer.org/en/latest/configuration.html
i have:
[global]
[install]
no-index = yes
find-links = /home/users/me/wheelhouse
Make sure the wheel package is installed.
In your project create setup.py file that will allow for the wheel creation and execute
python setup.py bdist_wheel
copy the generated wheel to the wheelhouse so it has:
~/wheelhouse/projectA-0.1-py33-none-any.whl
Now we want to create a project that uses that projectA API - project B
we are creating a separate folder for this project and then create a virtual environment for it.
mkdir projectB; cd projectB
virtualenv projectB_env
source projectB_env/bin/activate
pip install projectA
Now if you run python console in this folder you will be able to import the classes from the projectA! One problem solved!
Now you have finished the development of projectB and you need to run it.
For that purpose I'd recommend to use Pex (twitter.common.python) library. Pex now supports (v0.5.1) wheels lookup as dependencies. I'm feeding it the content of requirements.txt file to resolve dependencies. So as the result you will get the executable lightweight archived virtualenv that will have everything necessary for the project to run.
This should get you started:
http://docs.python.org/2/distutils/
http://guide.python-distribute.org/
http://pythonhosted.org/setuptools/
I'm new to pyramid and paster, just reading the docs for now. I use virtualenv and inside the virtualenv dir I want to start a pyramid project. The problem is that I would like for paster to not create a dir with the project name, and instead put all the scaffold files on the current dir (the venv root).
I thought about just not using paster but I still wouldn't know how to point to my app on development.ini "use" option.
I could also have my virtualenv on an entirely different place of my filesystem, but that seems weird to me (maybe virtualenvwrapper could make it easier). Any other way to do this?
It is confusing at first but your code really doesn't need to be in your virtual environment directory at all. Actually it's better not to put your code inside your environment, as you might want to use different environments with the same code, for example to test your code with different versions of Python or different versions of a library.
virtualenvwrapper does put all your environments in a single place. virtualenvwrapper is a convenient tool on top of virtualenv but you don't need it to put your code and your environments in different places. Maybe you should get a bit more comfortable with virtualenv itself before starting to use virtualenvwrapper.
You should let paster create a directory with the project name. This is the directory that you will commit in version control (eg. git, mercurial...). You don't want to commit the directory containing the virtual environment.
This is really just bike shedding because how you create the project and the virtualenv are irrelevant and you can place either of them anywhere, including within each other.
However, if you really want to, you can paster create -t pyramid_starter -o .. <current_directory_name> to create the project within the current directory.
To create a new project:
cd ~/work/my_repo
virtualenv --no-site-packages env
env/bin/pip install pyramid
env/bin/paster create -t pyramid_starter -o .. my_repo
git init
echo 'env' > .gitignore
git add .
I'll usually do this when setting up a new machine:
cd ~/work
git clone /path/to/<my repo>.git
cd my_repo
virtualenv --no-site-packages env
env/bin/pip install -e . # equivalent to env/bin/python setup.py develop
Using the setup I just mentioned, you'd want to add the env directory to your .gitignore file.