I want to create a package that has the following structure:
project/
plugin_dir/
templates/
sh/
I have a setup.py file like the following:
setup(name='my_plugin',
version='1.0.0',
packages=['my_plugin'],
install_requires=['slackclient>=1.0.5', 'airflow==1.8', 'snakebite>=2.11.0', 'prometheus-client>=0.0.19'],
dependency_links=[],
# scripts=findall('sh/')+findall('templates/'),
data_files=[('/sh', findall('sh/'))],
include_package_data=True,
zip_safe=False
)
I run the command python setup.py bdist_wheel and it doesn't include the files from sh or templates directory when I look in the wheel.
What am I doing wrong?
Nevermind it works fine, I didn't see it in the build/ but once you unpack the wheel running pip install my_plugin --target . --no-deps it's included.
Related
I wanted to try a github project named deformable kernels, and followed the steps described in the README.md file:
conda env create -f environment.yml
cd deformable_kernels/ops/deform_kernel;
pip install -e .;
The structure of deformable_kernel/ops/deform_kernel is showed here:
.
csrc
filter_sample_depthwise_cuda.cpp
filter_sample_depthwise_cuda.h
filter_sample_depthwise_cuda_kernel.cu
nd_linear_sample_cuda.cpp
nd_linear_sample_cuda.h
nd_linear_sample_cuda_kernel.cu
functions
filter_sample_depthwise.py
__init__.py
nd_linear_sample.py
__init__.py
modules
filter_sample_depthwise.py
__init__.py
setup.py
And the content of file setup.py is showed here:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
setup(
name='filter_sample_depthwise',
ext_modules=[
CUDAExtension(
'filter_sample_depthwise_cuda',
[
'csrc/filter_sample_depthwise_cuda.cpp',
'csrc/filter_sample_depthwise_cuda_kernel.cu',
]
),
],
cmdclass={'build_ext': BuildExtension}
)
setup(
name="nd_linear_sample",
ext_modules=[
CUDAExtension(
"nd_linear_sample_cuda",
[
"csrc/nd_linear_sample_cuda.cpp",
"csrc/nd_linear_sample_cuda_kernel.cu",
],
)
],
cmdclass={"build_ext": BuildExtension},
)
When I install this directory using command pip install -e ., it failed and the result is:
Obtaining file:///home/xxx/Downloads/deformable_kernels/deformable_kernels/ops/deform_kernel
ERROR: More than one .egg-info directory found in /tmp/pip-pip-egg-info-pta6z__q
So I tried to separate the 2 setup()s in different setup.py files. It worked but I didn't get a python file. Instead a .so file was generated.
Does anyone know how to solve a problem like this?
Check your pip version. I've had the same error (when installing other things in dev mode with pip) and downgrading to pip version 20.0.2 worked. Unsure why, but I've seen other folks on the internet solve the problem similarly.
This looks an awful lot like this question, but it isn't quite.
The answers to that question cover how to get extra data into a source distribution and binary distribution, but don't actually address how to get package data installed when included in a source distribution, or how to get them installed in a wheel file.
Here's an example of what I mean (python 3.7.4):
% ls
MANIFEST.in README.txt foopackage setup.py venv
% cat setup.py
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
setup(
name='foopackage',
version='0.0.1',
description='A Package Full of Foo',
packages=find_packages(),
package_data={
'foopackage': [
'README.txt',
],
},
include_package_data=True,
)
% cat MANIFEST.in
include README.txt
% . venv/bin/activate
(venv)
% python setup.py sdist bdist_wheel
[...]
% ls dist
foopackage-0.0.1-py3-none-any.whl foopackage-0.0.1.tar.gz
(venv)
% unzip -v dist/foopackage-0.0.1-py3-none-any.whl| grep README.txt
(venv)
% tar tvzf dist/foopackage-0.0.1.tar.gz| grep README.txt
-rw-r--r-- 0 matt staff 0 8 Nov 17:21 foopackage-0.0.1/README.txt
(venv)
% deactivate
% cd ../foo
% . py/bin/activate
(py)
% pip install ../foopackage/dist/foopackage-0.0.1.tar.gz
Processing ../foopackage/dist/foopackage-0.0.1.tar.gz
Installing collected packages: foopackage
Found existing installation: foopackage 0.0.1
Uninstalling foopackage-0.0.1:
Successfully uninstalled foopackage-0.0.1
Running setup.py install for foopackage ... done
Successfully installed foopackage-0.0.1
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(py)
% find . -name README.txt
(py)
%
The README.txt file isn't included in the wheel, and despite being included in the sdist file, it is not installed when that package is installed.
I can't seem to find the magic incantation to get setuptools to actually install this file somewhere. What am I missing?
I should probably point out that the real package I'm building has to be py27 and py34+ compatible, in case that constrains the solution in any way.
The problem is in
package_data={
'foopackage': [
'README.txt',
],
},
With this code you declare that your foopackage contains a file README.txt (file patterns in package_data are treated relative to the packages they belong to). But it is not! The file is in the root folder, not in the foopackage/ subfolder. This is also the reason why the file is not included into your wheel(s).
By listing the file in MANIFEST.in you forced setuptools to include the file into source dist but setuptools doesn't know what to do with the file — it expects the file to be inside foopackage/.
There are 2 ways to fix the problem. 1st, you can declare relative path to the file:
package_data={
'foopackage': [
'../README.txt',
],
},
In this case the fill be installed in foopackage/../README.txt, that is, in site-packages/ directory. It could be what you want.
Or not. In which case the right solution is to move the file inside foopackage/, fix MANIFEST.in and rebuild sdist and wheel(s).
I am trying to use setup.py to install a Python package that is kept in a git repository, which we'll call my_dependency. In my_package, I have a setup.py file with:
setup(
...
install_requires=[
...
'my_dependency=VERSION'
],
dependency_links=['git+https://...my_dependency.git#egg=my_dependency-VERSION',]
)
When I run my setup file (python setup.py develop), the dependency appears to install; it shows up as my_dependency==VERSION when I run pip freeze. However, when I start a python session and call import my_dependency, I get ImportError: No module named my_dependency.
I don't know if this is possibly the source of the problem, but when running setup.py, I get a warning:
Processing dependencies for my_package==0.1
Searching for my_dependency==VERSION
Doing git clone from https://.../my_dependency.git to /var/folders/.../.../T/easy_install-_rWjyp/my_dependency.git
Best match: my_dependency VERSION
Processing my_dependency.git
Writing /var/folders/.../my_dependency.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/.../my_dependency.git/egg-dist-tmp-UMiNdL
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Copying my_dependency-VERSION-py2.7.egg to /.../my_package/venv/lib/python2.7/site-packages
Adding my_dependency VERSION to easy-install.pth file
However, I am able to use the package if I install it through pip, like this: pip install -e git+https://.../my_dependency.git#egg=my_dependency-VERSION
For reference, the dependency package structure looks like this:
my_dependency/
my_dependency/
__init__.py
setup.py
And its setup.py contains this:
from setuptools import setup
setup(
name='my_dependency',
version='VERSION',
description='...',
author='...',
url='https://...',
license='MIT',
install_requires=[
'numpy',
],
zip_safe=False,
)
The solution was (in retrospect) pretty silly. My dependency package was missing this line in its setup.py:
packages=['my_dependency'],
That meant the package was correctly building and installing, but it wasn't actually including the code in the package. This became apparent when I looked at the SOURCES.txt in the egg-info: it didn't include any of the Python source files in the package.
I have a virtualenv with multiple little projects in it. Consider that they are all equal, so my folder structure looks something like this:
categorisation_ml/
categorisation.py
setup.py
__init__.py
nlp/
nlp.py
setup.py
__init__.py
etc/
__init__.py
I want to install both packages into the same virtualenv so that they are both accessible everywhere within the virtualenv.
Using this and this guide, I have created a setup.py script like this (for categorisation in this case):
from setuptools import setup, find_packages
setup(
name = "categorisation",
version = "1.0",
scripts = ['categorisation.py']
)
then, I run python setup.py install , which seems to complete successfully.
When I cd into nlp/, enter python command line and try
import categorisation, I get:
ImportError: No module named categorisation.
What am I missing?
It seems that the package structure and setup.py is off. It should be something like this:
irrelevant_package_name/
__init__.py
setup.py
categorisation_ml/
categorisation.py
__init__.py
nlp/
nlp.py
__init__.py
and then the install script looking like this:
from setuptools import setup, find_packages
setup(
name='package_name',
version='1.0.0',
description='This is a working setup.py',
url='http://somesite.com',
author='Roman',
author_email='roman#somesite.com',
packages=find_packages(),
install_requires=[
'numpy',
],
zip_safe=False
)
Then install it like this:
python setup.py install #(just installs it as is)
python setup.py develop #(Keeps track of changes for development)
If you pip freeze this should come up
package_name==1.0.0
And then in python imports should look like this:
from categorisation_ml import categorisation
from nlp import nlp
I'm using pip to install a package from a git repository:
pip install -e git+git://github.com/knipknap/SpiffWorkflow.git#master#egg=SpiffWorkflow-dev
The repo gets cloned without a problem, but installation fails with this message:
Running setup.py egg_info for package SpiffWorkflow
Installing collected packages: SpiffWorkflow
Running setup.py develop for SpiffWorkflow
error: ("Can't get a consistent path to setup script from installation
directory", '/', '/home/fcorreia/venvs/myproj/src/spiffworkflow')
I have tried taking a look to the project's setup.py, but without much success... Any idea?
For future folk, if you're using an older version of setuptools on windows 10 and it looks like it has an extra slash, you need to update the python package 'setuptools' to get around this windows 10 python bug
you can update any number of ways, but one is python -m pip install --upgrade setuptools
You need to have a pyproject.toml file in your package. I have no idea why this makes the error go away, but it works. This file is part of PEP 518 "Specifying Minimum Build System Requirements for Python Projects".
You can have your package in a src subfolder if you have a pyproject.toml in your project:
/src/yourpackage/__init__.py
/setup.py
/pyproject.toml
I have no idea why this works, but it makes the error message go away when you run pip install -e . to install the package in "editable" mode. The file doesn't even have to have anything in it, it can be a blank file and the error goes away.
(To figure this out, I found a working project that had its package stored under a src folder and kept deleting things until I got that error. This is clearly some bug in Pip. I have version 18.1 on Windows 10 for Python 3.7 on my machine.)
It is because the flag -e means "editable", and it is the same doing python setup.py develop, that creates a symbolic link from <PACKAGE_NAME_LOWERCASE> to your site-packages directory and not running an usual installation.
Looking at SpiffWorkflow's setup.py I can see where the problem relies:
srcdir = join(dirname(__file__), 'src')
setup(...,
package_dir = {'': srcdir})
It says that the package content is located at src, instead of spiffworkflow (what develop mode expects).
You can just drop the -e flag and be happy:
pip install git+git://github.com/knipknap/SpiffWorkflow.git#master#egg=SpiffWorkflow-dev
References:
https://github.com/pypa/pip/issues/126
http://packages.python.org/distribute/setuptools.html#develop
https://bitbucket.org/tarek/distribute/issue/177/setuppy-develop-
In my case problem was with package_dir = {'': './src'}: I have specified path to dir, rather than dir name, which for some reason worked fine with setup.py bdist_wheel.
I had the followng structure for my project:
+-- project/
+-- src/
| +-- project/
| +-- __init__.py
+-- doc/
+-- tests/
+-- setup.py
+-- ...
So basically, all python code in /src/project. This permits to avoid project to be directly imported from tests scripts or whatever.
setup.py content:
setuptools.setup(
...
packages=setuptools.find_packages('src'),
package_dir={'': 'src'},
...
)
Now I want to pull all this one level below, so that the overall project can have different components, as follows:
+-- project/
+-- backend/
| +-- src/
| +-- project/
| +-- __init__.py
|
+-- frontend/
| +-- ...
|
+-- doc/
+-- tests/
+-- setup.py
+-- ...
So I got the error message as in OP when trying to pip install -e ., even after trying to fix all paths.
I solved it by following update in setup.py:
setuptools.setup(
...
packages=setuptools.find_packages('backend/src'),
package_dir={'': 'backend/src'},
...
)
Hope this helps!
In my cases, https://github.com/quiver-team/torch-quiver/blob/main/setup.py
I remove the ./ chars from the beginning of my package dir configuration.
# package_dir = './srcs/python'
package_dir = 'srcs/python'
And I install it successfully.
refer to https://github.com/pypa/setuptools/discussions/3755