def get_pkgs():
pkgs = []
for importer, modname, ispkg in \
pkgutil.walk_packages(path=None, onerror=lambda x: None):
pkgs.append(modname)
return pkgs
Above code snippet gives me all python packages in distribution. Can someone suggest me a way to get all packages out of these used by a python application ?
Snakefood is nice, as already answered by Sven Marnach, but this gives all file dependencies and I'm not sure if it will just tell you what packages are used, versus every single file dependency.
For packages available in the Python Package Index, you could use virtualenv (and pip, which comes with it) to get a simple list of required/used packages.
For example, assuming you also have the excellent virtualenvwrapper tools installed (highly recommended), below is a sequence that shows what the package requirements are for pylint:
$ mkvirtualenv pylint_dep_check --no-site-packages
New python executable in pylint_dep_check/bin/python
Installing setuptools............done.
$ pip freeze #Note the wsgiref 'bug' where it always shows up
wsgiref==0.1.2
$ workon pylint_dep_check
(pylint_dep_check) $ pip install pylint
(... snipped lengthy install text ...)
(pylint_dep_check) $ pip freeze
logilab-astng==0.21.1
logilab-common==0.55.0
pylint==0.23.0
unittest2==0.5.1
wsgiref==0.1.2
Noe the all-important use of the --no-site-packages virtualenv creation option which (surprise!) ensures that your virtualenv is completely fresh and has none of the site-packages from your distribution installed. This way it is clear what is needed for the app you installed.
If this is an application that you have developed, a nice way to keep track of your dependencies (and an excellent/clean way to work) is to set up the application in a clean virtualenv (created using the --no-site-packages option again) and then again use pip freeze to figure out what packages you've installed to make it work.
The ability to start a "fresh" python installation with the --no-site-packages option is extremely useful. I do it for all applications and to test out packages I'm interested in without cluttering my workspace(s).
If you aren't using virtualenv and pip yet, get on it already. Here is a good introduction:
http://mathematism.com/2009/07/30/presentation-pip-and-virtualenv/
The simplest way is to use the Python dependency analysis tool snakefood.
Related
I am trying to install a Python package with this command
pip install <name of package>
I'm getting permission errors and I'm not sure why. I could run it with sudo, but someone told me that was a bad idea, and I should use a virtualenv instead.
What is a virtualenv? What does it do for me?
Running with the system Python and libraries limits you to one specific Python version, chosen by your OS provider. Trying to run all Python applications on one Python installation makes it likely that version conflicts will occur among the collection of libraries. It's also possible that changes to the system Python will break other OS features that depend on it.
Virtual environments, or "virtualenvs" are lightweight, self-contained Python installations, designed to be set up with a minimum of fuss, and to "just work" without requiring extensive configuration or specialized knowledge.
virtualenv avoids the need to install Python packages globally. When a virtualenv is active, pip will install packages within the environment, which does not affect the base Python installation in any way.
In Python 3.3 or later, you can create a virtualenv as follows:
$ python3 -m venv ENV_DIR
For Windows, you should replace python3 with the full path to python.exe:
>C:\Python34\python.exe -m venv ENV_DIR
(This is a typical Python installation; your system may vary.)
In older versions of Python, including Python 2, one of the following commands should work in most cases:
$ virtualenv ENV_DIR
$ venv ENV_DIR
$ pyvenv ENV_DIR
$ pyvenv3 ENV_DIR
ENV_DIR should be a non-existent directory. The directory can have any name, but to keep these instructions simple, I will assume you have created your virtualenv in a directory called venv (e.g. with python3 -m venv ./venv).
To work in your virtualenv, you activate it:
$ . ./venv/bin/activate
(venv)$
Or use this if you have a windows system:
$ venv\Scripts\activate
The (venv) in the shell prompt lets you know which virtualenv you have activated, but you can turn this feature off if you do not like it. You can run all the usual Python commands, and they will be local to your virtualenv:
(venv)$ pip install requests numpy
[...]
(venv)$ python
[...]
>>> import requests
>>> import numpy as np
>>>
python will run the version of Python that you installed into your virtualenv, so (for example) you don't have to type python3 to get Python 3. The Python that it runs will have access to all the standard library modules and all the packages you installed into the virtualenv, but (by default) none of the packages installed in the system-wide site-packages directory.
This last rule is important: by restricting your virtualenv to only use locally-installed packages, you can ensure that you control exactly which dependencies your project is using, even if some new system-wide package gets installed or updated next week. If you like, you can get a listing of your installed packages:
(venv)$ pip freeze
requests==2.13.0
numpy==1.12.0
(venv)$
pip can also parse this format and install from it, and it will install the same versions, even if updates have been released in the meantime:
(venv)$ pip freeze >requirements.txt
(some-other-venv)$ pip install -r requirements.txt
[...]
(some-other-venv)$ python
>>> import requests
>>> import numpy as np
>>>
You can get out of the virtualenv by deactivating it:
(venv)$ deactivate
$ python
[...]
>>> import requests
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'requests'
You can create as many virtualenvs as you like, and they won't interfere with each other, nor with your system packages. A virtualenv is "just" a directory with a bunch of binaries and scripts under it, so you can remove a virtualenv the same way you remove any directory (rm -r venv on Unix). If the virtualenv is activated when you remove it, you may confuse your shell, so it's probably a good idea to deactivate first in that case.
Some times you are not given root privileges and you might end up not being able to use sudo. Many other times, it's not advisable to use sudo to install packages as it might overwrite some package which might be in use by some other applications.
Virtualenv can help you create a separate environment where you don't need root privileges as well as be able to tailor the environment according to your need. It consists of self-contained python installation which only interacts with your specific created environment.
So basically, it gives you a bit of freedom as well as avoid damaging (or modifying) the root environment which might be hosting many old functionalities of old applications.
Installation is pretty easy too.
Installing packages with sudo pip will install packages globally, which may break some system tools.
By install globally it means you will install your packages in place like /usr/lib/python2.7/site-package so if some packages need a previous version of your python packages, this action may break it.
virtualenv allows you to avoid installing Python packages globally by making an isolated python environment. That means it will install packages just in your desire project folder.
On mac and linux
Install
python3 -m pip install --user virtualenv
Creating a Virtual Env: Go to your desired project folder
python3 -m virtualenv env
Activating a virtualenv: In your desired project folder
source env/bin/activate
After activating you can install your packages using pip.
For more information about using it in Windows:
How to use virtualenv in Windows
I am going to break your question into two parts.
What is a virtualenv?
Python has its own way of downloading, storing, and resolving site packages. But Python can not differentiate between different versions in the site-package directory. Packages will be installed in one of the directories, whose name can be found by running the site.getsitepackages() commands.
>>> import site
>>> site.getsitepackages()
This means package_v2.0.1 and package_v3.0.1 have to be in the same directory with the same name i.e. package, which is obviously not possible. Now, you may ask why we would need the same package with different versions on our system. This is because multiple projects may require different versions of Python packages or even different Python versions themselves. So there was a need to have something which will resolve these conflicts and Virtualenv came into the picture to solve this issue.
What does it do for me?
It isolates the environment for Python projects so that each project can have its own dependencies.
I have written a Python utility script that relies on the Debian package python3-apt:
import apt
...
def get_packages():
cache = apt.Cache()
for pkg in cache:
if pkg.installed and pkg.name in PACKAGE_LIST:
yield pkg.name
I am now expanding the script into a project, with the eventual intent of making it available on PyPI and/or as a Debian package itself.
I use virtualenvs to isolate my Python development environments. What package name (or path) do I need to add to my virtualenv so that I can call import apt from within that environment?
So far I have tried:
apt on PyPI. Strange old release.
vext. Does not currently support apt.
other things on PyPI that start with "apt". None of them are a simple intermediary for python3-apt.
You can achieve this with pipenv as follows (similar instructions should work for other venv managers):
pipenv --site-packages # see note 1
PIP_IGNORE_INSTALLED=1 pipenv install # see note 2
You are more likely to run this as:
pipenv --site-packages
PIP_IGNORE_INSTALLED=1 pipenv install -e . --dev
# treats codebase as a package, also installs dev dependencies
Note 1: We must access system packages (aka site packages) so that we can import apt.
Note 2: ...but we prefer virtualenv packages to system packages. See
https://pipenv.pypa.io/en/latest/advanced/#working-with-platform-provided-python-components for details.
Comments:
This means that all other system packages not defined in your Pipfile are also available in your venv. You must remember that they aren't necessarily available to other developers using the same codebase. If you have a basic CI environment, it should catch this.
This approach will work for other packages not supported by vext.
Running command line:
virtualenv --system-site-packages venv
I'm expecting venv folder venv\Lib\site-packages to contain all the necessary library from the projects that are located in:
C:\Users\XXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\
But it's not the case, only a few are installed.
Example, my program currently use pdfminer which is in
C:\Users\XXXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\
I want it to be included in venv\Lib\site-packages but it is not copied.
Any advice?
--system-site-packages doesn't copy packages, it just allows python from the virtualenv to access packages in C:\Users\XXX\AppData\Local\Programs\Python\Python36\Lib\site-packages\.
There is no way to copy packages because they could depend on their installation directory. If you want these packages in the virtualenv don't use --system-site-packages and install all packages in the virtualenv.
A virtualenv environment is the same as if you have just installed a new version of Python. It has no packages other than the standard packages provided with Python. If you want other packages, you have to install them with 'pip' or however you'd do it with the native Python version that you are using.
So in general, just do pip install <packagename>.
If you find yourself often wanting to create virtualenvs with a standard set of base packages, then put together a requirements.txt file listing all of the packages you want to install as a base, and do pip install -r requirements.txt inside a new virtualenv, right after you create it.
One nice thing about a virtualenv is that it's all yours. Your user owns 100% of it, unlike the base Python version that is owned by the system. To install new packages into the base Python version, you often have to have root access (sudo privileges). With virtualenvs, you don't need special permissions (in fact, you'll get all screwed up if you use sudo in a virtualenv) to install all the packages you want. Everything you do lives within your own home directory. Another neat thing is that when you are done with a virtualenv, you just throw away the root directory that contains it.
If its not mandatory to use virtualenv, I would suggest to go with Anaconda. That'll pretty much help your concern.
Conda as a package manager helps you find and install packages. By default quite a few packages are already installed, so as to set you up quickly for your project. To check the list of packages installed in terminal, type: conda list to obtain the packages installed using conda.
If you need a package that requires a different version of Python, you do not need to switch to a different environment manager, because conda is also an environment manager.
With just a few commands, you can set up a totally separate environment to run that different version of Python, while continuing to run your usual version of Python in your normal environment
I am trying to install a Python package with this command
pip install <name of package>
I'm getting permission errors and I'm not sure why. I could run it with sudo, but someone told me that was a bad idea, and I should use a virtualenv instead.
What is a virtualenv? What does it do for me?
Running with the system Python and libraries limits you to one specific Python version, chosen by your OS provider. Trying to run all Python applications on one Python installation makes it likely that version conflicts will occur among the collection of libraries. It's also possible that changes to the system Python will break other OS features that depend on it.
Virtual environments, or "virtualenvs" are lightweight, self-contained Python installations, designed to be set up with a minimum of fuss, and to "just work" without requiring extensive configuration or specialized knowledge.
virtualenv avoids the need to install Python packages globally. When a virtualenv is active, pip will install packages within the environment, which does not affect the base Python installation in any way.
In Python 3.3 or later, you can create a virtualenv as follows:
$ python3 -m venv ENV_DIR
For Windows, you should replace python3 with the full path to python.exe:
>C:\Python34\python.exe -m venv ENV_DIR
(This is a typical Python installation; your system may vary.)
In older versions of Python, including Python 2, one of the following commands should work in most cases:
$ virtualenv ENV_DIR
$ venv ENV_DIR
$ pyvenv ENV_DIR
$ pyvenv3 ENV_DIR
ENV_DIR should be a non-existent directory. The directory can have any name, but to keep these instructions simple, I will assume you have created your virtualenv in a directory called venv (e.g. with python3 -m venv ./venv).
To work in your virtualenv, you activate it:
$ . ./venv/bin/activate
(venv)$
Or use this if you have a windows system:
$ venv\Scripts\activate
The (venv) in the shell prompt lets you know which virtualenv you have activated, but you can turn this feature off if you do not like it. You can run all the usual Python commands, and they will be local to your virtualenv:
(venv)$ pip install requests numpy
[...]
(venv)$ python
[...]
>>> import requests
>>> import numpy as np
>>>
python will run the version of Python that you installed into your virtualenv, so (for example) you don't have to type python3 to get Python 3. The Python that it runs will have access to all the standard library modules and all the packages you installed into the virtualenv, but (by default) none of the packages installed in the system-wide site-packages directory.
This last rule is important: by restricting your virtualenv to only use locally-installed packages, you can ensure that you control exactly which dependencies your project is using, even if some new system-wide package gets installed or updated next week. If you like, you can get a listing of your installed packages:
(venv)$ pip freeze
requests==2.13.0
numpy==1.12.0
(venv)$
pip can also parse this format and install from it, and it will install the same versions, even if updates have been released in the meantime:
(venv)$ pip freeze >requirements.txt
(some-other-venv)$ pip install -r requirements.txt
[...]
(some-other-venv)$ python
>>> import requests
>>> import numpy as np
>>>
You can get out of the virtualenv by deactivating it:
(venv)$ deactivate
$ python
[...]
>>> import requests
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'requests'
You can create as many virtualenvs as you like, and they won't interfere with each other, nor with your system packages. A virtualenv is "just" a directory with a bunch of binaries and scripts under it, so you can remove a virtualenv the same way you remove any directory (rm -r venv on Unix). If the virtualenv is activated when you remove it, you may confuse your shell, so it's probably a good idea to deactivate first in that case.
Some times you are not given root privileges and you might end up not being able to use sudo. Many other times, it's not advisable to use sudo to install packages as it might overwrite some package which might be in use by some other applications.
Virtualenv can help you create a separate environment where you don't need root privileges as well as be able to tailor the environment according to your need. It consists of self-contained python installation which only interacts with your specific created environment.
So basically, it gives you a bit of freedom as well as avoid damaging (or modifying) the root environment which might be hosting many old functionalities of old applications.
Installation is pretty easy too.
Installing packages with sudo pip will install packages globally, which may break some system tools.
By install globally it means you will install your packages in place like /usr/lib/python2.7/site-package so if some packages need a previous version of your python packages, this action may break it.
virtualenv allows you to avoid installing Python packages globally by making an isolated python environment. That means it will install packages just in your desire project folder.
On mac and linux
Install
python3 -m pip install --user virtualenv
Creating a Virtual Env: Go to your desired project folder
python3 -m virtualenv env
Activating a virtualenv: In your desired project folder
source env/bin/activate
After activating you can install your packages using pip.
For more information about using it in Windows:
How to use virtualenv in Windows
I am going to break your question into two parts.
What is a virtualenv?
Python has its own way of downloading, storing, and resolving site packages. But Python can not differentiate between different versions in the site-package directory. Packages will be installed in one of the directories, whose name can be found by running the site.getsitepackages() commands.
>>> import site
>>> site.getsitepackages()
This means package_v2.0.1 and package_v3.0.1 have to be in the same directory with the same name i.e. package, which is obviously not possible. Now, you may ask why we would need the same package with different versions on our system. This is because multiple projects may require different versions of Python packages or even different Python versions themselves. So there was a need to have something which will resolve these conflicts and Virtualenv came into the picture to solve this issue.
What does it do for me?
It isolates the environment for Python projects so that each project can have its own dependencies.
I have a python project that has a few dependencies (defined under install_requires in setup.py). My ops people requires a package to be self contained and only depend on a python installation. The litmus test would be that they're able to get a zip-file and then unzip and run it without an internet connection.
Is there an easy way to package an install including dependencies? It is acceptable if I have to build on the OS/architecture that it will eventually be run on.
For what it's worth, I've tried both setup.py build and setup.py sdist, but they don't seem to fit the bill since they do not include dependencies. I've also considered virtualenv (which could be installed if absolutely necessary), but that has hard coded paths which makes it less than ideal.
There are a few nuances to how pip works. Unfortunately, using --prefix vendor to store all the dependencies of the project doesn't work if any of those dependencies, or dependencies of dependencies are installed into a place where pip can find them. It will skip those dependencies and just install the rest to your vendor folder.
In the past I've used virtualenv's --no-site-packages option to solve this issue. At one company we would ship the whole virtualenv, which includes the python binary. In the interest of only shipping the dependencies, you can combine using a virtualenv with the --prefix switch on pip to give yourself a clean environment that installs to the right place.
I'll provide an example script that creates a temporary virtualenv, activates it, then installs the dependencies to a local vendor folder. This is handy if you are running in CI.
#!/bin/bash
tempdir=$(mktemp -d -t project.XXX) # create a temporary directory
trap "rm -rf $tempdir" EXIT # ensure it is cleaned up
# create the virtualenv and exclude packages outside of it
virtualenv --python=$(which python2.7) --no-site-packages $tempdir/venv
# activate the virtualenv
source $tempdir/venv/bin/activate
# install the dependencies as above
pip install -r requirements.txt --prefix=vendor
In most cases you should be able to "vendor" all the dependencies. It's basically a crude version of virtualenv.
For example look at how the requests package includes chardet and urllib3 in its own source tree. Here's an example script that should do the initial downloading and copying for you: https://gist.github.com/proppy/1136723
Once you have the dependencies installed, you can reference them with from .some.namespace import dependency_name to make sure that you're using your local versions.
It's possible to do this with recent versions of pip (I'm using 8.1.2). On the build machine:
pip install -r requirements.txt --prefix vendor
Then run it:
PYTHONPATH=vendor/lib/python2.7/site-packages python yourapp.py
(This is basically an expansion of #valentjedi comment. Thanks!)
let's say you have python module app.py with dependencies in requirements.txt file.
first, install all your dependencies in appdeps folder.
python -m pip install -r requirements.txt --target=./appdeps
then in your app.py module add this dependency folder to the pythonpath
# app.py
import sys
sys.path.append('appdeps')
# rest of your module normally
#...
this will work the same way as if you were running this script from venv with all the dependencies installed inside ;>