Develop Reference

Force venv to put python's symlink in identical locations on macOS and Windows

I am developing a program inside a Python virtual environment. Running the command
python -m venv .venv
on Windows, puts the python symlink in .venv\Scripts\ while
python3 -m venv .venv
on macOS puts the symlinks on .venv/bin/. I'm using Visual Studio Code for development and there is a .vscode\\settings.json which needs to point towards the executable's symlinks. For example
{
"python.pythonPath": ".venv\\path\\to\\python",
}
How can I force the venv to put the symlinks in identical locations regardless of the platform?
P.S.1. The .venv folder is ignored by git and is included in the .gitignore file, as AFIK it is not a good practice to ship it with the codebase.
P.S.2. To avoid XY-problem, the final goal is to have an identical development environment on different platforms. Whatever the solution, regardless of the package manager, Visual Studio Code should be able to find the executable's symlinks on different platforms.
P.S.3. I need to find a way to change the __VENV_BIN_NAME__ parameter.
P.S.4. From here, one possible solution might be to use "python.pythonPath": "${env:PYTHON_INSTALL_LOC}", instead.
P.S.5. I asked another question here.

Two things. One, you can't have a virtual environment use bin/ on Windows (see the rejected idea).
Two, the Python extension is running an experiment to get rid of python.pythonPath which will solve your conflict. See the May release blog post for details and how to opt in.

Why does it matter where each OS puts the venv python? If you've correctly made sure that .venv is not in the code base but making sure it's in .gitignore than it doesn't matter if someone who loads up your code even need venv. It's good practice and guarantees that whoever has your code on whichever OS their python, with any dependencies your program needs, is going to be used.
If you are using multiple OS to develop than, create a workspace or project .vscode settings file with those specific settings to the python you want to use but also make sure that .vscode directory in your project is in your .gitignore. Then set up separate .vscode settings on different machines for those OS-specific settings.

Add relative paths to configuration file virtualenv_path_extensions.pth

I'm attempting to find a set of steps necessary to make a virtual environment of python 3.6 on windows relocatable.
1st I created a virtual environment on virtualenv 15.1.0 with the following command:
virtualenv
--always-copy
-a "path\to\project\dir"
-r "path\to\requirements.txt"
venv_name
After this, I run the following command to use the built in 'make paths relative ' functionality of virtualenv:
virtualenv --relocatable venv_name
Part of my requirements.txt is pypiwin32 library which, at least when installed via pip, wont work until the:
python Scripts/pywin32_postinstall.py -install
script is run (See here for details).
At this point, if I search the venv directory for clues of hardcoding, I see them in scripts\activate.bat, which I can make relative by changing this:
set "VIRTUAL_ENV=C:\path\to\venv"
into this:
pushd %~dp0..
set VIRTUAL_ENV=%CD%
popd
There are some other other places where I had to make slight adjustments to make them relative (I used the search in folder feature of sublime with my username as the search parameter - it brought up all the path\to\username\then\some\more style lines in the directory.
There are 2 hardcoded paths which are not so simple:
1. "path\to\venv\Lib\orig-prefix.txt"
I understand that orig-prefix.txt is a record of which is the source python installation on which the venv was based and so cannot really be relative but may need to be left blank if moving the venv to another machine (it's absence may crash the python launcher but its emptiness is fine.)
2. "path\to\venv\Lib\site-packages\virtualenv_path_extensions.pth"
This is trickier. As it is a hard-coded path which is then added to sys.path as a location to look for modules, when I move the venv to another machine where this path doesn't exist, the module load will fail.
Is there a way I can add relative paths to the configuration files such as virtualenv_path_extensions.pth

Normally environments are tied to a specific path. That means that you cannot move an environment around or copy it to another computer. You can fix up an environment to make it relocatable with the command:
$ virtualenv --relocatable ENV
This will make some of the files created by setuptools use relative paths, and will change all the scripts to use activate_this.py instead of using the location of the Python interpreter to select the environment.
Note: scripts which have been made relocatable will only work if the virtualenv is activated, specifically the python executable from the virtualenv must be the first one on the system PATH. Also note that the activate scripts are not currently made relocatable by virtualenv --relocatable.
Note: you must run this after you’ve installed any packages into the environment. If you make an environment relocatable, then install a new package, you must run virtualenv --relocatable again.
Also, this does not make your packages cross-platform. You can move the directory around, but it can only be used on other similar computers. Some known environmental differences that can cause incompatibilities: a different version of Python, when one platform uses UCS2 for its internal unicode representation and another uses UCS4 (a compile-time option), obvious platform changes like Windows vs. Linux, or Intel vs. ARM, and if you have libraries that bind to C libraries on the system, if those C libraries are located somewhere different (either different versions, or a different filesystem layout).

virtualenvwrapper: when does `workon` changes to the project directory?

I use virtualenvwrapper to manage my environments. I create my projects with the -a <path-to-project> argument and PROJECT_HOME is not set, because my projects don't share a common path. VIRTUALENVWRAPPER_WORKON_CD is set to 1 though and a valid .project file exist in the virtual environments.
When I use the workon command, it only changes the working directory to the path of the project sometimes, while at other times, the directory stays the same, despite the environment being activated correctly.
So when and how does the directory change trough workon happens? And are there things I have to do / set for it to work?

I have a small solution albeit kinda hacky
Go to the folder which this variable, $VIRTUALENVWRAPPER_HOOK_DIR is pointing to. You might need to be in a virtual environment to see that variable, but usually it has the same value as $WORKON_HOME
In this folder you should see a script called postactivate
Add this as the last line in the script: cdproject
Now what this does is that every time you type workon <project_name>, this script will run after the virtual environment is activated, and the directory you were working in will switch to the directory for that project.
For more life-cycle hooks, see here!
NOTE
I didn't fully test this to make sure it works no matter how the project was created, but to make sure it works, I recommend creating your virtual environments with mkvirtualenv -a <env_name> or if the environment exists and with the environment activated, go to it's project folder and run setvirtualenvproject. Now the next time you try to do workon ..., the script will kick in and take you to your project folder
Addendum
Since this is a shell script, you can do some fancy tweaks than just that one line. For example you might want to only do something if the activated project has a certain pattern, or part of a certain group of projects. Checkout some of the $VIRTUALENVWRAPPER_* variables to see what other useful information you can get

setvirtualenvproject is now called setprojectdir. To use the current directory write setprojectdir .

Using PYTHONPATH to use a virtualenv

I have a virtualenv in a structure like this:
venv/
src/
project_files
I want to run a makefile (which calls out to Python) in the project_files, but I want to run it from a virtual environment. Because of the way my deployment orchestration works, I can't simply do a source venv/bin/activate.
Instead, I've tried to export PYTHONPATH={project_path}/venv/bin/python2.7. When I try to run the makefile, however, the python scripts aren't finding the dependencies installed in the virtualenv. Am I missing something obvious?

The PYTHONPATH environmenbt variable is not used to select the path of the Python executable - which executable is selected depends, as in all other cases, on the shell's PATH environment variable. PYTHONPATH is used to augment the search list of directories (sys.path in Python) in which Python will look for modules to satisfy imports.
Since the interpreter puts certain directories on sys.path before it actions PYTHONPATH precisely to ensure that replacement modules with standard names do not shadow the standard library names. So any standard library module will be imported from the library associated with the interpreter it was installed with (unless you do some manual furkling, which I wouldn't recommend).
venv/bin/activate does a lot of stuff that needs to be handled in the calling shell's namespace, which can make tailoring code rather difficult if you can't find a way to source the script..

You can actually just call the Python interpreter in your virtual environment. So, in your Makefile, instead of calling python, call venv/bin/python.

To run a command in a virtualenv, you could use vex utility:
$ vex venv make
You could also check, whether make PYTHON=venv/bin/python would be enough in your case.
PYTHONPATH adjusts sys.path list. It doesn't change python binary. Don't use it here.

Virtual Environments with Python

Hi I've been reading a lot about virtual environments but I don't seem to get one thing.
So I have my path like this:
../my_app/
../my_app/app.py
..etc
Where should I put the virtual environment?
Inside my_appas /my_app/venv/bin,include,lib?
or at the same level as my_app
/my_app/
/venv/
I don't understand if the location matters or if by using activate it will reference it instead of going to the main environment.
I hope this question makes sense.
Thanks!

I recommend utilizing the root directory which virtualenv creates as the root directory for your source files. Virtual-envirments are designed to be tied to a project, not shared between different projects. Example, lets say I write a lot of code in my ~/school directory. I'd cd to ~/school, do virtualenv ENV. Now I have an ENV directory in which to keep my source files and dependencies for the project. So you can create a ~/school/ENV/source folder in which to keep all your source folders. And all your virtual-environment files are close to your program, easily accessible in the ENV directory.
EDIT:
To address one part of your question: As long as you keep track of your environment, and you source bin/activate before running your python programs and installing dependencies with pip, you can install your virtual environment anywhere.

I don't understand if the location matters or if by using activate it
will reference it instead of going to the main environment.
It doesn't matter, as activate will take care of the paths correctly, but you shouldn't put your virtual environment in your source, as it is not part of your application (and its not portable). Instead, create a file with your requirements and put that under your source control.
You should put it in any directory other than your source code. The activate script will make sure the paths point to the right places.
Here is an example:
$ virtualenv /home/somedir/envs/myenv
... # some output
$ source /home/somedir/envs/myenv/bin/activate
(myenv) $ mkdir /home/somedir/projects
(myenv) $ cd /home/somedir/projects
(myenv) projects $
As you can see, the virtual environment is in the envs directory and is called myenv. Your source is in /home/somedir/projects. Type deactivate to exit your virtual environment.