Develop Reference

How to *install* extra package data using setuptools?

This looks an awful lot like this question, but it isn't quite.
The answers to that question cover how to get extra data into a source distribution and binary distribution, but don't actually address how to get package data installed when included in a source distribution, or how to get them installed in a wheel file.
Here's an example of what I mean (python 3.7.4):
% ls
MANIFEST.in README.txt foopackage setup.py venv
% cat setup.py
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
setup(
name='foopackage',
version='0.0.1',
description='A Package Full of Foo',
packages=find_packages(),
package_data={
'foopackage': [
'README.txt',
],
},
include_package_data=True,
)
% cat MANIFEST.in
include README.txt
% . venv/bin/activate
(venv)
% python setup.py sdist bdist_wheel
[...]
% ls dist
foopackage-0.0.1-py3-none-any.whl foopackage-0.0.1.tar.gz
(venv)
% unzip -v dist/foopackage-0.0.1-py3-none-any.whl| grep README.txt
(venv)
% tar tvzf dist/foopackage-0.0.1.tar.gz| grep README.txt
-rw-r--r-- 0 matt staff 0 8 Nov 17:21 foopackage-0.0.1/README.txt
(venv)
% deactivate
% cd ../foo
% . py/bin/activate
(py)
% pip install ../foopackage/dist/foopackage-0.0.1.tar.gz
Processing ../foopackage/dist/foopackage-0.0.1.tar.gz
Installing collected packages: foopackage
Found existing installation: foopackage 0.0.1
Uninstalling foopackage-0.0.1:
Successfully uninstalled foopackage-0.0.1
Running setup.py install for foopackage ... done
Successfully installed foopackage-0.0.1
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(py)
% find . -name README.txt
(py)
%
The README.txt file isn't included in the wheel, and despite being included in the sdist file, it is not installed when that package is installed.
I can't seem to find the magic incantation to get setuptools to actually install this file somewhere. What am I missing?
I should probably point out that the real package I'm building has to be py27 and py34+ compatible, in case that constrains the solution in any way.

The problem is in
package_data={
'foopackage': [
'README.txt',
],
},
With this code you declare that your foopackage contains a file README.txt (file patterns in package_data are treated relative to the packages they belong to). But it is not! The file is in the root folder, not in the foopackage/ subfolder. This is also the reason why the file is not included into your wheel(s).
By listing the file in MANIFEST.in you forced setuptools to include the file into source dist but setuptools doesn't know what to do with the file — it expects the file to be inside foopackage/.
There are 2 ways to fix the problem. 1st, you can declare relative path to the file:
package_data={
'foopackage': [
'../README.txt',
],
},
In this case the fill be installed in foopackage/../README.txt, that is, in site-packages/ directory. It could be what you want.
Or not. In which case the right solution is to move the file inside foopackage/, fix MANIFEST.in and rebuild sdist and wheel(s).

Unable to import dependency installed from git in setup.py

I am trying to use setup.py to install a Python package that is kept in a git repository, which we'll call my_dependency. In my_package, I have a setup.py file with:
setup(
...
install_requires=[
...
'my_dependency=VERSION'
],
dependency_links=['git+https://...my_dependency.git#egg=my_dependency-VERSION',]
)
When I run my setup file (python setup.py develop), the dependency appears to install; it shows up as my_dependency==VERSION when I run pip freeze. However, when I start a python session and call import my_dependency, I get ImportError: No module named my_dependency.
I don't know if this is possibly the source of the problem, but when running setup.py, I get a warning:
Processing dependencies for my_package==0.1
Searching for my_dependency==VERSION
Doing git clone from https://.../my_dependency.git to /var/folders/.../.../T/easy_install-_rWjyp/my_dependency.git
Best match: my_dependency VERSION
Processing my_dependency.git
Writing /var/folders/.../my_dependency.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/.../my_dependency.git/egg-dist-tmp-UMiNdL
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Copying my_dependency-VERSION-py2.7.egg to /.../my_package/venv/lib/python2.7/site-packages
Adding my_dependency VERSION to easy-install.pth file
However, I am able to use the package if I install it through pip, like this: pip install -e git+https://.../my_dependency.git#egg=my_dependency-VERSION
For reference, the dependency package structure looks like this:
my_dependency/
my_dependency/
__init__.py
setup.py
And its setup.py contains this:
from setuptools import setup
setup(
name='my_dependency',
version='VERSION',
description='...',
author='...',
url='https://...',
license='MIT',
install_requires=[
'numpy',
],
zip_safe=False,
)

The solution was (in retrospect) pretty silly. My dependency package was missing this line in its setup.py:
packages=['my_dependency'],
That meant the package was correctly building and installing, but it wasn't actually including the code in the package. This became apparent when I looked at the SOURCES.txt in the egg-info: it didn't include any of the Python source files in the package.

pip install on archive fails to install git dependency

I have a package that depends on the following github project: https://github.com/jbittencourt/python-escpos.git (not to be confused with the python-escpos package that exists on PyPi).
So in my package's requirements.txt I have git+https://github.com/jbittencourt/python-escpos.git#egg=escpos.
When I install my package using either pip install -r requirements.txt or with python setup.py install (which parses the same requirements.txt file for its install_requires parameter) - everything works great.
The problem is when I archive my package into mypackage.tar.gz. After archiving the very same code that worked a second ago, and running pip install mypackage.tar.gz I get the following error:
Collecting escpos (from shopic==1.10.0)
Could not find a version that satisfies the requirement escpos (from shopic==1.10.0) (from versions: )
No matching distribution found for escpos (from shopic==1.10.0)
From playing around with it, I noticed that w/e I type after the "#egg=" part of git+https://github.com/jbittencourt/python-escpos.git#egg=escpos - it simply tries to get it off PyPi (and fails, in this case). E.g. if I wrote (for some odd reason) git+https://github.com/jbittencourt/python-escpos.git#egg=requests it'd install the requests package.
I don't understand why it works differently when all I do is pack the files in a .tar.gz, and I have no idea how to work around it. I cannot use one of the other installation methods because I have another package that has this one as its dependency, and when I run pip install on that package, it has this tar.gz file as one of its dependencies, and so it fails.
Any ideas why this happens and how to resolve it? Thanks.
Edit: Here's my setup.py for reference:
from setuptools import setup, find_packages
import pip
links = []
requires = []
requirements = pip.req.parse_requirements('requirements.txt', session=pip.download.PipSession())
for item in requirements:
# we want to handle package names and also repo urls
if getattr(item, 'url', None): # older pip has url
links.append(str(item.url))
if getattr(item, 'link', None): # newer pip has link
links.append(str(item.link))
if item.req:
requires.append(str(item.req))
setup(name='mypackage',
version='1.1.0',
description='My Package',
author='EK',
author_email='me#example.com',
url='http://www.example.com',
license='MIT',
packages=find_packages(),
package_data={'': ['*.jar']},
zip_safe=False,
install_requires=requires,
dependency_links=links)

Python dependency resolution

I have previously created a python package and uploaded it to pypi. The package depends upon 2 other packages defined within the setup.py file:
from setuptools import setup
from dominos.version import Version
def readme():
with open('README.rst') as file:
return file.read()
setup(name='dominos',
version=Version('0.0.1').number,
author='Tomas Basham',
url='https://github.com/tomasbasham/dominos',
license='MIT',
packages=['dominos'],
install_requires=[
'ratelimit',
'requests'
],
include_package_data=True,
zip_safe=False)
As both of these were already installed within my virtualenv this package would run fine.
Now trying to consume this package within another python application (and within a separate virtualenv) I have defined the following requirements.txt file:
dominos==0.0.1
geocoder==1.13.0
For reference dominos is the package I uploaded to pypi. Now running pip install --no-cache-dir -r requirements.txt fails because dependencies of dominos are missing:
ImportError: No module named ratelimit
Surely pip should be resolving these dependencies since I have defined them in the setup.py file of dominos. Clarity on this would be great.

pip install package call setup.py install instead of setup.py bdist_wheel

I have create a private package and simply all things are well. setup.py bdist_wheel simply create the wheel for it and other commands works correct.
also pip install package_xxx.whl works fine and installs the created wheel correctly. but when I try to pip install the package from a requirement file or git repo or local path, it sucks...
As you know the pip will call setup.py file
In my case the setup.py file called twice with following arguments: (captured from sys.argv)
['-c', 'egg_info', '--egg-base', 'pip-egg-info']
['-c', 'install', '--record', '/long/path/to/install-record.txt', '--single-version-externally-managed', '--compile']
but for example if I pip install coverage the setup.py file of coverage package called twice with following:
['-c', 'egg_info', '--egg-base', 'pip-egg-info']
['-c', 'bdist_wheel', '-d', '/long/path/to/tmp82jyoapip-wheel-', '--python-tag', 'cp27']
the first call to setup.py in my package and the coverage.py package is same and pip gets the egg-info data. then it call setup.py bdist_wheel on coverage.py (the desired behavior) but call setup.py install on my package (broken behavior) that lead to create egg-like project not wheel one.
my setup.py file is in following gist: https://gist.github.com/wtayyeb/f26578fe6ff17dc6acd3
it is beside other files in the package and as I say all things are working except pip install /path/to/mypackage
Thanks.

The problem was in folder name that containes setup.py. I have found it with inspecting pip and found the critical check which was direct the process to legacy method. see below link if you intrest in it.
https://stackoverflow.com/a/35590238/875667