I wrote some Python code I would like to package to be easily installed from pip and conda.
The code runs in a conda environment containing all its dependencies.
For both pip and conda it seems that I need to write a setup.py file with a install_requires variable to set the dependencies. For pip I also need a requirement.txt with these dependencies.
conda list --export gives me a way to export the environment with everything including libraries, cython and ipython:
alabaster=0.7.3=py27_0
babel=1.3=py27_0
backports_abc=0.4=py27_0
cairo=1.12.18=3
cffi=0.9.2=py27_0
cython=0.23.4=py27_0
decorator=4.0.4=py27_0
docutils=0.12=py27_0
flake8=2.3.0=py27_0
fontconfig=2.11.1=3
...
but how can I get only the Python packages my code depends on? Shall I go through all my imports? In that case how would I manage the dependencies of the dependencies?
Related
I've had to install a package with pip in a conda environment to get it to work for my application (link).
The package works fine. However, every time I modify the virtual environment in any way, conda tries to install the "missing" package - which would effectively result in downgrading it.
Question: is there a way to mark the pip package as 'manually installed' in the conda venv (e.g. in the same way apt-mark would handle it)? The intention is to get miniconda to leave it alone while still handling the remaining dependencies for the desired additional package.
The pip installed package indeed shows up when typing conda list, with Channel "pypi".
Can give any additional information if needed.
Thanks in advance for any help.
I would like to make a relocatable environment. So I need to use relative paths in the package installations. For this I just create a Conda Environment like this:
conda create --prefix env python=3.6.5
activate .\env
And then I have installed the needed packages as usual with
pip install package_name
The problem comes when I want to install my own package. I have created a structure like this and I have followed this tutorial:
some_root_dir/
|-- setup.py
|-- python_files
|-- |-- runall.py
|-- |-- test0.py
And the content of the setup.py is this:
from setuptools import setup
setup(
name='my_app',
version='0.1',
description='My app',
keywords="app csv some other word",
url='https://www.my_domain.com/',
author='My name',
author_email='email#email_domain.com',
license='MIT',
packages=['my_package'],
zip_safe=False,
)
But after the installation with:
cd some_root_dir
pip install .
and moving it to another location, the paths that are appearing in the application are the ones where I did the pip install .
I have been looking for information here, but I did not find anything useful.
Main Steps I want to do
Create a conda environment and install some packages with pip or conda, my own python package included
Copy the environment folder to another computer
Run the application in this computer where conda and python are not installed. If I use the python.exe included in the folder python should know where the packages are installed and how to import them.
Questions
How can I use relative paths in the environment packages?
Is this doable? Or am I doing anything wrong?
Which are the best practices to achieve what I want?
Are the relocatable environments possible?
Note: I am using Windows 10 and Miniconda 3.
Virtualenv
The equivalent on virtualenv would be this:
virtualenv --relocatable env_folder
But it is an experimental feature
Update (August 7, 2018)
Actually what I want is what #interfect says in his comment, the issue is here. So relocatable environments on conda are not possible yet
I think that relocatable environments depend on the installed packages. They should be implemented with relative paths and avoiding hardcoded paths. All the paths that are used in the source code of the package should be inside the own package. So if you install well done package you won´t have any problem to relocate the environment in other folder or computer.
As you will need to add all the folders inside the package you will need to modify the arguments of the setup. Add these two parameters in order to add folders to the final package. If you don´t do this the folders won´t copied to the site-packages folder within the environment (the final destination when you install the package with pip):
packages=[
'main_folder',
'main_folder.folder_with_python_files',
'main_folder.other_folder_with_python_files',
],
package_data={
'main_folder': [
'static/css/*.*',
'templates/*.*',
],
},
Environments, Package Manager and Paths
I have tried to build the environment on Windows with Virtualenv, but I had some problems building a basic environment:
There was a dll library missing: VCRUNTIME140.dll
The runpy module was missing as well. This is used to run commands with the -m parameter: python -m ...
Other packages dependencies were not installed when I used pip such as zipfile
So I came back to Conda Environments again, but I have built the environment with package manager pip, instead of conda, because the packages were much lighter in my case.
Therefore, my recommendation is to install the packages with pip. If any of them are giving problems after the relocation, we should check if there is any harcoded path and change it directly. Though the best solution would be to modify the original source code and install the customised package.
Some python scripts in the environment had the absolute path on the header with #!.
#!C:\absolute\path\to\python.exe
I just removed them because if I call any script with the python.exe that is currently inside the environment those headers are ignored
Update
Also conda-pack can be useful. I haven´t tried it yet
conda-pack is a command line tool for creating relocatable conda environments. This is useful for deploying code in a consistent environment, potentially in a location where python/conda isn’t already installed.
If you turn your package into a conda-package (trivial if you are using pip already), you just conda install your packages on the new machine and everything will be relocated at install time.
That includes any compiled libraries let alone the paths in scripts. Conda will modify everything so it just works no matter where you install it.
I'm working on packaging up a suite of tools that can be installed in different environments, and I've run into many problems with dependencies, which are an issue since this package will be installed in air-gapped environments.
The package will be installed via Anaconda, and I have provided the installation script. In order to create the package, I ran the following command:
conda metapackage toolkit_bundle 0.0.1 --dependencies r-essentials tensorflow gensim spacy r-ggplot2 r-plotly r-dplyr r-rjson r-tm r-reshape2 r-shiny r-sparklyr r-slam r-nlp r-cluster r-ggvis r-plyr r-tidyr r-zoo r-magrittr r-xtable r-htmlwidgets r-formattable r-highcharter --summary "Toolkit Bundle"
This produced a tar.bzip2 file that I held on to and tried to install via the conda command
conda install toolkit_bundle.tar.bz2
The command seemed to run successfully, but I was unsuccessful in importing the modules in Python. I also tried creating a virtual conda environment and importing the package.
conda create -n myenv toolkit_bundle-0.0.1.tar.bz2
There was no error, but none of the modules were able to be imported either.
Am I missing a step in this process, or is my thought process flawed?
Update:
It looks like my thinking was pretty flawed. A quick skim of the conda metapackage command documentation revealed the following:
Tool for building conda metapackages. A metapackage is a package with no files, only metadata. They are typically used to collect several packages together into a single package via dependencies.
So my initial understanding was incorrect, and the package only contains metadata. Are there any other ideas for creating packages with dependencies resolved that can be installed in an air gapped environment?
I think you want to look at the command conda build for making packages, which just requires writing an appropriate meta.yaml file containing the dependencies, along with some other build parameters. There is good documentation for doing so on the conda website: https://conda.io/docs/user-guide/tasks/build-packages and there is a repo of examples.
If you have a working PIP package, you can also auto-generate a conda package recipe using conda skeleton.
Once you have built a set of packages locally, you can use the --use-local option to conda install to install from your local repo, with no need for an internet connection (as long as the packages for all the dependencies are in your local repo).
I was able to download the packages I needed via the pypi website, and after determining the dependencies, I manually downloaded them and wrote a script to install them in the required order.
I am working on an open source python project which depends on a lot of non-python packages (perl, r, ...). Consequently, they use conda to install the dependencies, as they can't be simply installed with pip.
You can run $ conda install --channel bioconda <awesome_package> to install the stable build of the package. I want to install it in the development mode. For a purely python project, I'd do something like this:
pull the source code from github
$ cd path/to/awesome_package && pip install -e .
run tests
modify the code
run tests
etc
In the step 2 above, the pip command installs all I need to work. It uses setup.py script to do its job, and requirements.txt together with requirements_dev.txt to install the dependencies. And, I don't need to rebuild/reinstall anything when I change something in the source code.
How do I do the same with conda instead of pip? How do you provide a list of requirements (python and non-python) to conda?
The most relevant thing I found is this: http://conda.pydata.org/docs/building/bpp.html
However, with this approach I'd need to rebuild and reinstall the package locally very time I want to run the tests, which is something I'd like to avoid.
I am only going to change the python source code of the package I'm working on.
TL;DR: how to build a package with conda in a dev mode from the source code?
Any help is much appreciated.
I am working on a project where they are using Ansible to run several conda installs. I need to install two additional packages from github that have dependencies that are already covered by the existing conda installs with the second package having a dependency on the first.
Using the Ansible code below, I can get the first package to install without reinstalling the dependencies.
- name: install mypackage
shell: /home/myname/envs/myproject/bin/pip install --install-option="--prefix=/home/myname/envs/myproject" --egg https://github.com/myname/mypackage/archive/my_branch.zip
This gets me 95% of the way there, however, when I try to install the second package, it doesn't recognize the first package as having been installed and fails.
I am new to this and I have been throwing things up against the wall but I'm not able to install the first package in such a way where:
It recognizes the existing conda installs
The second package identifies the first one
From what I can understand from your task you are using a venv to install the packages, that's good. I don't understand why, though, you are using the shell module to handle the install.. This not good.
You can handle all this with ansible' pip module :
- name: "Install mypackage"
pip:
virtualenv: /home/{{ lookup('env','USER') }}/envs/myproject/
name: "{{ item }}"
with_items:
- "https://github.com/myname/mypackage1/archive/my_branch.zip"
- "https://github.com/myname/mypackage2/archive/my_branch.zip"
This should install correctly the packages in the order you require, without the hassle of having to work your way through shell output.
Note that you can mix normal python packages with eggs etc..
As an alternative to virtualenv you can use executable.
Have a look at the docs
I believe the question is how to use ansible to pip install packages within a conda environment. Noting that it is perfectly possible to use pip install within a conda environment, which is particularly useful in cases where the desired package does not exist on the conda repositories and cannot be installed with conda install.
The goal is thus to use the environment created by conda, and not a virtualenv (for which, btw, ansible's pip module provides specific parameters).
I have managed to do so by using ansible's pip module and pointing the pip executable to the one installed within the desired conda environment.
See code below, notice usage of the executable variable:
- name: Install pip packages WITHIN a designated conda environment
pip:
name: some_package_name
executable: "/home/[username]/[anaconda3]/envs/[conda_env_name]/bin/pip"
# ^-- Of course you will need to ensure the correct path.
This will pip install the packages inside the designated conda environment.