I have a python library that has a dependency on importlib. importlib is in the standard library in Python 2.7, but is a third-party package for older pythons. I typically keep my dependencies in a pip-style requirements.txt. Of course, if I put importlib in here, it will fail if installed on 2.7. How can I conditionally install importlib only if it's not available in the standard lib?
I don't think this is possible with pip and a single requirements file. I can think of two options I'd choose from:
Multiple requirements files
Create a base.txt file that contains most of your packages:
# base.txt
somelib1
somelib2
And create a requirements file for python 2.6:
# py26.txt
-r base.txt
importlib
and one for 2.7:
# py27.txt
-r base.txt
Requirements in setup.py
If your library has a setup.py file, you can check the version of python, or just check if the library already exists, like this:
# setup.py
from setuptools import setup
install_requires = ['somelib1', 'somelib2']
try:
import importlib
except ImportError:
install_requires.append('importlib')
setup(
...
install_requires=install_requires,
...
)
Related
I use python click package and setuptools to create a simple command line.
And I work in a pipenv virtualenv.
My working directory is like this：
jkt/scripts/app.py
And my setup.py is like this:
from setuptools import setup, find_packages
setup(
name='jkt',
version='0.1',
packages=find_packages(),
include_package_data=True,
entry_points='''
[console_scripts]
jktool=jkt.scripts.app:my_function
''',
)
Then I run the command
pip install --editable .
And run jktool to execute my_function but I get the error:
ModuleNotFoundError No module named 'jkt'.
But when the app.py in jkt directory I can run my function
setup(
name='app',
version='0.1',
py_modules=['app'],
entry_points='''
[console_scripts]
app=app:jktools
''',
)
After I run pip install -e . I can use app command to run my function.
As I mentioned, I can't reproduce your error (Python 3.7 with modern pip seems to work just fine), but there are a couple things that could potentially be going wrong on older versions.
Since it doesn't look like you put __init__.py files in your subdirectories, find_packages doesn't actually find any packages at all (python3 -c 'from setuptools import find_packages; print(find_packages()) prints the empty list, []). You can fix this in one of three ways:
Create empty __init__.py files to explicitly mark those folders as package folders; on a UNIX-like system, touch jkt/__init__.py and touch jkt/scripts/__init__.py is enough to create them
Python 3.3+ only: (also requires modern setuptools so pip install --upgrade setuptools might be necessary) Replace your use of find_packages with find_namespace_packages (which recognizes Python 3 era implicit namespace packages).
Just get rid of find_packages entirely and list the packages directly, e.g. replace packages=find_packages(), with packages=['jkt', 'jkt.scripts'],
Options #2 only works on Python 3.3+, so if your package is intended to work on older versions of Python, go with option #1 or #3.
I am trying to use setup.py to install a Python package that is kept in a git repository, which we'll call my_dependency. In my_package, I have a setup.py file with:
setup(
...
install_requires=[
...
'my_dependency=VERSION'
],
dependency_links=['git+https://...my_dependency.git#egg=my_dependency-VERSION',]
)
When I run my setup file (python setup.py develop), the dependency appears to install; it shows up as my_dependency==VERSION when I run pip freeze. However, when I start a python session and call import my_dependency, I get ImportError: No module named my_dependency.
I don't know if this is possibly the source of the problem, but when running setup.py, I get a warning:
Processing dependencies for my_package==0.1
Searching for my_dependency==VERSION
Doing git clone from https://.../my_dependency.git to /var/folders/.../.../T/easy_install-_rWjyp/my_dependency.git
Best match: my_dependency VERSION
Processing my_dependency.git
Writing /var/folders/.../my_dependency.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/.../my_dependency.git/egg-dist-tmp-UMiNdL
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Copying my_dependency-VERSION-py2.7.egg to /.../my_package/venv/lib/python2.7/site-packages
Adding my_dependency VERSION to easy-install.pth file
However, I am able to use the package if I install it through pip, like this: pip install -e git+https://.../my_dependency.git#egg=my_dependency-VERSION
For reference, the dependency package structure looks like this:
my_dependency/
my_dependency/
__init__.py
setup.py
And its setup.py contains this:
from setuptools import setup
setup(
name='my_dependency',
version='VERSION',
description='...',
author='...',
url='https://...',
license='MIT',
install_requires=[
'numpy',
],
zip_safe=False,
)
The solution was (in retrospect) pretty silly. My dependency package was missing this line in its setup.py:
packages=['my_dependency'],
That meant the package was correctly building and installing, but it wasn't actually including the code in the package. This became apparent when I looked at the SOURCES.txt in the egg-info: it didn't include any of the Python source files in the package.
I'm developing a Python application and in the process of branching off a release. I've got a PyPI server set up on a company server and I've copied a source distribution of my package onto it.
I checked that the package was being hosted on the server and then tried installing it on my local development machine. I ended up with this output:
$ pip3 install --trusted-host 172.16.1.92 -i http://172.16.1.92:5001/simple/ <my-package>
Collecting <my-package>
Downloading http://172.16.1.92:5001/packages/<my-package>-0.2.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\setup.py", line 9, in <module>
import appdirs
ModuleNotFoundError: No module named 'appdirs'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\
The reason is that I'm trying to import a third-party library appdirs in my setup.py, which is necessary for me to compute the data_files argument to setup():
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
import os
from collections import defaultdict
import appdirs
from <my-package>.version import __version__ as <my-package>_version
APP_NAME = '<my-app>'
APP_AUTHOR = '<company>'
SYSTEM_COMPONENT_PLUGIN_DIR = os.path.join(appdirs.user_data_dir(APP_NAME, APP_AUTHOR), 'components')
# ...
setup(
# ...
data_files=component_files,
)
However, I don't have appdirs installed on my local dev machine and I don't expect the end users to have it either.
Is it acceptable to rely on third-party libraries like this in setup.py, and if so what is the recommended approach to using them? Is there a way I can ensure appdirs gets installed before it's imported in setup.py, or should I just document that appdirs is a required package to install my package?
I'm ignoring licensing issues in this answer. You definetly need to take these into account before you really do a release.
Is it acceptable to rely on third-party libraries like this in setup.py
Yes, it is acceptable but generally these should be minimized, especially if these are modules which have no obvious use for the end-user. Noone likes to have packages they don't need or use.
what is the recommended approach to using them?
There are basically 3 options:
Bootstrap them (for example use pip to programmatically install packages). For example setuptools provides an ez_setup.py file that can be used to bootstrap setuptools. Maybe that can be customized to download and install appdirs.
Include them (especially if it's a small package) in your project. For example appdirs is basically just a single file module. Pretty easy to copy and maintain in your project. Be very careful with licensing issues when you do that!
Fail gracefully when it's not possible to import them and let the user install them. For example:
try:
import appdirs
except ImportError:
raise ImportError('this package requires "appdirs" to be installed. '
'Install it first: "pip install appdirs".')
You could use pip to install the package programmatically if the import fails:
try:
import appdirs
except ImportError:
import pip
pip.main(['install', 'appdirs'])
import appdirs
In some circumstances you may need to use importlib or __import__ to import the package after pip.main or referesh the PATH variable. It could also be worthwhile to include a verification if the user really wants to install that package before installing it.
I used a lot of the examples from "Installing python module within code" and I haven't personally tried used this in setup.py files but it looks like it could be a solution for your question.
You can mention install_requires with the dependencies list. Please check the python packaging guide here. Also you can provide a requirements.txt file so that it can be run at once using "pip install -r"
I have previously created a python package and uploaded it to pypi. The package depends upon 2 other packages defined within the setup.py file:
from setuptools import setup
from dominos.version import Version
def readme():
with open('README.rst') as file:
return file.read()
setup(name='dominos',
version=Version('0.0.1').number,
author='Tomas Basham',
url='https://github.com/tomasbasham/dominos',
license='MIT',
packages=['dominos'],
install_requires=[
'ratelimit',
'requests'
],
include_package_data=True,
zip_safe=False)
As both of these were already installed within my virtualenv this package would run fine.
Now trying to consume this package within another python application (and within a separate virtualenv) I have defined the following requirements.txt file:
dominos==0.0.1
geocoder==1.13.0
For reference dominos is the package I uploaded to pypi. Now running pip install --no-cache-dir -r requirements.txt fails because dependencies of dominos are missing:
ImportError: No module named ratelimit
Surely pip should be resolving these dependencies since I have defined them in the setup.py file of dominos. Clarity on this would be great.
I have read a bunch of threads on setuptools here.
A lot of people seem not to like it very much.
But I need to install MySQL-python-1.2.3. and when I do that I get this error:
MySQL-python-1.2.3 X$ python setup.py cleanTraceback (most recent call last):
File "setup.py", line 5, in <module>
from setuptools import setup, Extension
ImportError: No module named setuptools
So it seems I need setuptools and that it is assumed that it is installed.
On the setuptools python homepage it says:
Setuptools will install itself using the matching version of Python (e.g. python2.4), and will place the easy_install executable in the default location for installing Python scripts (as determined by the standard distutils configuration files, or by the Python installation).
Does this mean it will replace any default easy install from python?
If so I dont want to use it.
If so can I install MySQL-python-1.2.3 without setupttools?
Thanks
You should use virtualenv and pip.
Virtualenv automatically creates a setuptools version within the new environment, so the default one is intact.
You may want to read how the packaging and installing works: 1, 2