I would like to make a relocatable environment. So I need to use relative paths in the package installations. For this I just create a Conda Environment like this:
conda create --prefix env python=3.6.5
activate .\env
And then I have installed the needed packages as usual with
pip install package_name
The problem comes when I want to install my own package. I have created a structure like this and I have followed this tutorial:
some_root_dir/
|-- setup.py
|-- python_files
|-- |-- runall.py
|-- |-- test0.py
And the content of the setup.py is this:
from setuptools import setup
setup(
name='my_app',
version='0.1',
description='My app',
keywords="app csv some other word",
url='https://www.my_domain.com/',
author='My name',
author_email='email#email_domain.com',
license='MIT',
packages=['my_package'],
zip_safe=False,
)
But after the installation with:
cd some_root_dir
pip install .
and moving it to another location, the paths that are appearing in the application are the ones where I did the pip install .
I have been looking for information here, but I did not find anything useful.
Main Steps I want to do
Create a conda environment and install some packages with pip or conda, my own python package included
Copy the environment folder to another computer
Run the application in this computer where conda and python are not installed. If I use the python.exe included in the folder python should know where the packages are installed and how to import them.
Questions
How can I use relative paths in the environment packages?
Is this doable? Or am I doing anything wrong?
Which are the best practices to achieve what I want?
Are the relocatable environments possible?
Note: I am using Windows 10 and Miniconda 3.
Virtualenv
The equivalent on virtualenv would be this:
virtualenv --relocatable env_folder
But it is an experimental feature
Update (August 7, 2018)
Actually what I want is what #interfect says in his comment, the issue is here. So relocatable environments on conda are not possible yet
I think that relocatable environments depend on the installed packages. They should be implemented with relative paths and avoiding hardcoded paths. All the paths that are used in the source code of the package should be inside the own package. So if you install well done package you won´t have any problem to relocate the environment in other folder or computer.
As you will need to add all the folders inside the package you will need to modify the arguments of the setup. Add these two parameters in order to add folders to the final package. If you don´t do this the folders won´t copied to the site-packages folder within the environment (the final destination when you install the package with pip):
packages=[
'main_folder',
'main_folder.folder_with_python_files',
'main_folder.other_folder_with_python_files',
],
package_data={
'main_folder': [
'static/css/*.*',
'templates/*.*',
],
},
Environments, Package Manager and Paths
I have tried to build the environment on Windows with Virtualenv, but I had some problems building a basic environment:
There was a dll library missing: VCRUNTIME140.dll
The runpy module was missing as well. This is used to run commands with the -m parameter: python -m ...
Other packages dependencies were not installed when I used pip such as zipfile
So I came back to Conda Environments again, but I have built the environment with package manager pip, instead of conda, because the packages were much lighter in my case.
Therefore, my recommendation is to install the packages with pip. If any of them are giving problems after the relocation, we should check if there is any harcoded path and change it directly. Though the best solution would be to modify the original source code and install the customised package.
Some python scripts in the environment had the absolute path on the header with #!.
#!C:\absolute\path\to\python.exe
I just removed them because if I call any script with the python.exe that is currently inside the environment those headers are ignored
Update
Also conda-pack can be useful. I haven´t tried it yet
conda-pack is a command line tool for creating relocatable conda environments. This is useful for deploying code in a consistent environment, potentially in a location where python/conda isn’t already installed.
If you turn your package into a conda-package (trivial if you are using pip already), you just conda install your packages on the new machine and everything will be relocated at install time.
That includes any compiled libraries let alone the paths in scripts. Conda will modify everything so it just works no matter where you install it.
Related
I use Visual Studio Code with flake8 linter but I don't know how to make a virtual environment with an access to globally installed package (flake8).
virtualenv env --system-site-packages doesn't work like I would like because when I put a command pip freeze it shows me all global packages.
You can set the "python.linting.flake8Path" setting to point to your global install of flake8.
As for keeping your dev tools separate from project's dependencies, pretty much all packaging tools for Python have a way to separate the two when listing dependencies. You can also keep a requirements.txt file that maintains your project's dependencies and a dev-requirements.txt that contains:
-r requirements.txt
flake8
and that can be what developers of your code install from.
Running command line:
virtualenv --system-site-packages venv
I'm expecting venv folder venv\Lib\site-packages to contain all the necessary library from the projects that are located in:
C:\Users\XXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\
But it's not the case, only a few are installed.
Example, my program currently use pdfminer which is in
C:\Users\XXXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\
I want it to be included in venv\Lib\site-packages but it is not copied.
Any advice?
--system-site-packages doesn't copy packages, it just allows python from the virtualenv to access packages in C:\Users\XXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\.
There is no way to copy packages because they could depend on their installation directory. If you want these packages in the virtualenv don't use --system-site-packages and install all packages in the virtualenv.
A virtualenv environment is the same as if you have just installed a new version of Python. It has no packages other than the standard packages provided with Python. If you want other packages, you have to install them with 'pip' or however you'd do it with the native Python version that you are using.
So in general, just do pip install <packagename>.
If you find yourself often wanting to create virtualenvs with a standard set of base packages, then put together a requirements.txt file listing all of the packages you want to install as a base, and do pip install -r requirements.txt inside a new virtualenv, right after you create it.
One nice thing about a virtualenv is that it's all yours. Your user owns 100% of it, unlike the base Python version that is owned by the system. To install new packages into the base Python version, you often have to have root access (sudo privileges). With virtualenvs, you don't need special permissions (in fact, you'll get all screwed up if you use sudo in a virtualenv) to install all the packages you want. Everything you do lives within your own home directory. Another neat thing is that when you are done with a virtualenv, you just throw away the root directory that contains it.
If its not mandatory to use virtualenv, I would suggest to go with Anaconda. That'll pretty much help your concern.
Conda as a package manager helps you find and install packages. By default quite a few packages are already installed, so as to set you up quickly for your project. To check the list of packages installed in terminal, type: conda list to obtain the packages installed using conda.
If you need a package that requires a different version of Python, you do not need to switch to a different environment manager, because conda is also an environment manager.
With just a few commands, you can set up a totally separate environment to run that different version of Python, while continuing to run your usual version of Python in your normal environment
I have a python project that has a few dependencies (defined under install_requires in setup.py). My ops people requires a package to be self contained and only depend on a python installation. The litmus test would be that they're able to get a zip-file and then unzip and run it without an internet connection.
Is there an easy way to package an install including dependencies? It is acceptable if I have to build on the OS/architecture that it will eventually be run on.
For what it's worth, I've tried both setup.py build and setup.py sdist, but they don't seem to fit the bill since they do not include dependencies. I've also considered virtualenv (which could be installed if absolutely necessary), but that has hard coded paths which makes it less than ideal.
There are a few nuances to how pip works. Unfortunately, using --prefix vendor to store all the dependencies of the project doesn't work if any of those dependencies, or dependencies of dependencies are installed into a place where pip can find them. It will skip those dependencies and just install the rest to your vendor folder.
In the past I've used virtualenv's --no-site-packages option to solve this issue. At one company we would ship the whole virtualenv, which includes the python binary. In the interest of only shipping the dependencies, you can combine using a virtualenv with the --prefix switch on pip to give yourself a clean environment that installs to the right place.
I'll provide an example script that creates a temporary virtualenv, activates it, then installs the dependencies to a local vendor folder. This is handy if you are running in CI.
#!/bin/bash
tempdir=$(mktemp -d -t project.XXX) # create a temporary directory
trap "rm -rf $tempdir" EXIT # ensure it is cleaned up
# create the virtualenv and exclude packages outside of it
virtualenv --python=$(which python2.7) --no-site-packages $tempdir/venv
# activate the virtualenv
source $tempdir/venv/bin/activate
# install the dependencies as above
pip install -r requirements.txt --prefix=vendor
In most cases you should be able to "vendor" all the dependencies. It's basically a crude version of virtualenv.
For example look at how the requests package includes chardet and urllib3 in its own source tree. Here's an example script that should do the initial downloading and copying for you: https://gist.github.com/proppy/1136723
Once you have the dependencies installed, you can reference them with from .some.namespace import dependency_name to make sure that you're using your local versions.
It's possible to do this with recent versions of pip (I'm using 8.1.2). On the build machine:
pip install -r requirements.txt --prefix vendor
Then run it:
PYTHONPATH=vendor/lib/python2.7/site-packages python yourapp.py
(This is basically an expansion of #valentjedi comment. Thanks!)
let's say you have python module app.py with dependencies in requirements.txt file.
first, install all your dependencies in appdeps folder.
python -m pip install -r requirements.txt --target=./appdeps
then in your app.py module add this dependency folder to the pythonpath
# app.py
import sys
sys.path.append('appdeps')
# rest of your module normally
#...
this will work the same way as if you were running this script from venv with all the dependencies installed inside ;>
I built a python3 package called gender_univ using the Anaconda conda build command. I uploaded the package to the Anaconda cloud and then installed it into my conda environment. Though the package shows up in the list of installed packages when I type conda list, whenever I try to access the package using import gender_univ I get the error no module named gender_univ.
I want to understand why I can't seem to import a package that is apparently installed in my conda virtual environment? Any suggestions.
First, I would check that you are using the correct python (i.e. which python and confirm that it is the python in your conda environment). Next, you can check if your package is in the site-packages directory of that same python.
The most likely thing, I'd bet, is that the conda package doesn't include everything correctly. Are you sure that you have a build.sh (or bld.bat if you are on windows) and a setup.py? Did you try expanding your built conda package and looking for your python classes in there?
If you expand your built conda package, probably something like gender_univ-VERSION-py35_0.tar.bz2, you should see a lib/python3.5/site-packages/gender_univ directory (i.e. python package). Do you? If not, then the failure is with your building.
I've needed to deal with this for some time, but never really figured out what the most pythonic way of importing/setting up PYTHONPATH for custom modules is. I know I can use virtualenv to manage it, I know I can set it inside of scripts, or through pth files, but none of these seem very clean and pythonic to me, so I'm guessing I'm missing something.
Almost always, all custom modules I'm interested in are contained in the git directory I've cloned down that has whatever script I'm running, if that simplifies things.
I'm guessing virtualenv is the answer, but figured I'd ask in case I'm missing anything.
EDIT: To clarify, this is only a question about custom modules. I'm already using pip for modules from PyPI.
You can use pip to install packages that are not on PyPI also. You just need an URI endpoint and a valid python package:
Examples:
$ pip install https://github.com/pypa/pip/archive/develop.zip#egg=pip
$ pip install git+https://github.com/pypa/pip.git#egg=pip
$ pip install git+git://github.com/pypa/pip.git#egg=pip
$ pip install /path/to/pip.tar.gz
$ pip install .
Read more at https://pip-installer.org/en/latest/usage.html#pip-install
virtualenv is a good start.
There are also package managers like pip and easy_install that manage third party modules.
In code you can use:
import sys
sys.path.append('/path/to/customModule')
Virtualenv is the way to go with this.
pip install virtualenv
Then make a folder to setup your environments. Inside that folder:
virtualenv <new_env_name>
That'll create a new folder in that directory, inside that there's a bin folder, run source on activate in that bin folder. You can then do pip install and it will only install it for that environment.
If you're cloning a git repo that you also want to be able to peruse the code easily (like if you're also working on that repo) clone it into your work_dir and then symlink or alias the package folder into the site-package directory inside that virtualenv's lib directory. Otherwise, if it's packaged correctly if you do python setup.py install it should install it right for that virtualenv.