I have a Python project which defines an entry_point in its setup.py like so:
entry_points={'console_scripts': ['foo=foo.main:run']}
When installed globally via Pip (from PyPI) and run with foo the application's sys.path doesn't use the current working directories virtualenv environment, as expected. Is it possible to write my program in such a way that it can use packages from an active virtual environment?
Related
I would like to make a relocatable environment. So I need to use relative paths in the package installations. For this I just create a Conda Environment like this:
conda create --prefix env python=3.6.5
activate .\env
And then I have installed the needed packages as usual with
pip install package_name
The problem comes when I want to install my own package. I have created a structure like this and I have followed this tutorial:
some_root_dir/
|-- setup.py
|-- python_files
|-- |-- runall.py
|-- |-- test0.py
And the content of the setup.py is this:
from setuptools import setup
setup(
name='my_app',
version='0.1',
description='My app',
keywords="app csv some other word",
url='https://www.my_domain.com/',
author='My name',
author_email='email#email_domain.com',
license='MIT',
packages=['my_package'],
zip_safe=False,
)
But after the installation with:
cd some_root_dir
pip install .
and moving it to another location, the paths that are appearing in the application are the ones where I did the pip install .
I have been looking for information here, but I did not find anything useful.
Main Steps I want to do
Create a conda environment and install some packages with pip or conda, my own python package included
Copy the environment folder to another computer
Run the application in this computer where conda and python are not installed. If I use the python.exe included in the folder python should know where the packages are installed and how to import them.
Questions
How can I use relative paths in the environment packages?
Is this doable? Or am I doing anything wrong?
Which are the best practices to achieve what I want?
Are the relocatable environments possible?
Note: I am using Windows 10 and Miniconda 3.
Virtualenv
The equivalent on virtualenv would be this:
virtualenv --relocatable env_folder
But it is an experimental feature
Update (August 7, 2018)
Actually what I want is what #interfect says in his comment, the issue is here. So relocatable environments on conda are not possible yet
I think that relocatable environments depend on the installed packages. They should be implemented with relative paths and avoiding hardcoded paths. All the paths that are used in the source code of the package should be inside the own package. So if you install well done package you won´t have any problem to relocate the environment in other folder or computer.
As you will need to add all the folders inside the package you will need to modify the arguments of the setup. Add these two parameters in order to add folders to the final package. If you don´t do this the folders won´t copied to the site-packages folder within the environment (the final destination when you install the package with pip):
packages=[
'main_folder',
'main_folder.folder_with_python_files',
'main_folder.other_folder_with_python_files',
],
package_data={
'main_folder': [
'static/css/*.*',
'templates/*.*',
],
},
Environments, Package Manager and Paths
I have tried to build the environment on Windows with Virtualenv, but I had some problems building a basic environment:
There was a dll library missing: VCRUNTIME140.dll
The runpy module was missing as well. This is used to run commands with the -m parameter: python -m ...
Other packages dependencies were not installed when I used pip such as zipfile
So I came back to Conda Environments again, but I have built the environment with package manager pip, instead of conda, because the packages were much lighter in my case.
Therefore, my recommendation is to install the packages with pip. If any of them are giving problems after the relocation, we should check if there is any harcoded path and change it directly. Though the best solution would be to modify the original source code and install the customised package.
Some python scripts in the environment had the absolute path on the header with #!.
#!C:\absolute\path\to\python.exe
I just removed them because if I call any script with the python.exe that is currently inside the environment those headers are ignored
Update
Also conda-pack can be useful. I haven´t tried it yet
conda-pack is a command line tool for creating relocatable conda environments. This is useful for deploying code in a consistent environment, potentially in a location where python/conda isn’t already installed.
If you turn your package into a conda-package (trivial if you are using pip already), you just conda install your packages on the new machine and everything will be relocated at install time.
That includes any compiled libraries let alone the paths in scripts. Conda will modify everything so it just works no matter where you install it.
I am trying to install flask and a few other modules for Python2.
When I try to install them using command pip install flask, it installs these for Python3.
This has created major issues because things like django are not compatible with Python3.
When I want to run a program using Python2, I cant use any of these modules.
Question
How do I use pip to install modules into a specified version of Python?
Try:
python2.7 -m pip install flask
Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally.
Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.
Currently, there are two common tools for creating Python virtual environments:
venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.
virtualenv needs to be installed separately, but supports Python 2.6+ and Python 3.3+, and pip, setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).
The basic usage is like so:
Using virtualenv:
virtualenv <DIR>
source <DIR>/bin/activate
Using venv:
python3 -m venv <DIR>
source <DIR>/bin/activate
For more information, see the virtualenv docs or the venv docs.
Managing multiple virtual environments directly can become tedious, so the dependency management tutorial introduces a higher level tool, Pipenv, that automatically manages a separate virtual environment for each project and application that you work on.
https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments
I have a pytest test, let's call it test.py. I used to run this test outside of virtualenv; now I'm trying to run it inside a virtualenv sandbox.
The project is structured like this:
~/project/test # where test.py and all virtualenv files live
~/project/mylibrary
test.py imports from mylibrary. In the past, this worked because I have the code in ~/project/mylibrary installed into /usr/lib/python2.7/dist-packages/mylibrary.
I can't run virtualenv with the --system-site-packages flag. I also can't move the code from ~/project/mylibrary into the ~/project/test folder. How can I get access to the code in mylibrary inside my virtualenv?
You don't need to do anything special - as long as you are working inside a virtualenv, python setup.py install will automatically install packages into
$VIRTUAL_ENV/lib/python2.7/site-packages
rather than your system-wide
/usr/lib/python2.7/dist-packages
directory.
In general it's better to use pip install mylibrary/, since this way you can neatly uninstall the package using pip uninstall mylibrary.
If you're installing a working copy of some code that you're developing, it might be a good idea to install it in "editable" mode using pip install -e mylibrary/, which creates a link to your source directory so that your installed module gets updated as you edit the code.
The easiest way would be to add the directory containing the library to your sys.path
Having windows 10, pycharm version 4.5.4, python v3.4, django v1.8 I imported a django project existed in virtualenv directory into pycharm. Thus far I've installed several packages through the same virtualenv (using pip) like django-registration-redux etc, all being recognizable by pycharm. I installed pymongo in the same way as all the other packages but pycharm cannot recognize it although the package exists in the site-packages of the virtualenv directory and I can import it using the django manage.py shell.
I tried installing the pymongo globally and this way pycharm could recognize the package.
My question being: what pycharm doesn't recognize the pymongo package that was installed in virtualenv and how can I fix it?
Please check in your virtual environment if that package is really installed:
$ pip freeze
pymongo==<version of installed pymongo>
if it is really installed in your Virtual Environment check PyCharm settings. (Configuring Project interpreter)
Settings => Project: => Project Interpreter
Your Project interpreter should be listed there and selected for current project. If no - select it.
Also all packages installed in virtual environment will be listed there.
I am installing virtualenv and it seems to access the system site packages before accessing the local site packages. Ipython is required by some other programs so it was automatically installed. This only happened recently and now it finds that version instead of the one found locally in the environment.
How do I tell the environment to use local packages within the environment before global packages? Can you set the Path variable for within the environment?
Ended up being an error with previously had set the PYTHON_PATH variable in .bashrc so this was looking in the system built directories before looking locally. Kind of defeating the purpose of virtual_env.
If you are using distribute + pip to manage dependencies simply run pip -l freeze > requirements.txt, this creates a dependency list of all your local packages. Next remove the current virtualenv; rerun the virtualenv command and specify the --no-site-packages option. Activate your new environment and finally pip install -r requirements.txt to download all the dependencies from the requirements file.