Develop Reference

Cannot import personal python module (installed with pip -e) from outside the module's directory

I am developing a package for internal use, and said package has a setup.py file (see below). I am a bit baffled because when I install my package (inside an environment & in editable mode so changes get reflected as I develop it), the import works if I am in the development directory, but says "package not found" from any other directory.
Moreover, pip list shows the package.
To install, I do the following:
~ > cd path/to/package
package > conda activate env
package (env) > pip install -e .
The package installs. Now, the problem is
package (env) > python -c "import mypackage" # works!
package (env) > cd
~ (env) > python -c "import mypackage" # error!
~ (env) > pip list | grep "mypackage"
mypackage 0.1.0
What's happening? Documentation says
-e, --editable <path/url>
Install a project in editable mode (i.e. setuptools “develop mode”) from a local project path or a VCS url.
Which doesn't really tell me that it should only work in said local project path...
setup.py
import os
import sys
from setuptools import find_packages, setup
def read(fname):
return open(os.path.join(os.path.dirname(__file__), fname)).read()
LICENSE = "MIT"
setup(
name="name",
version="0.1.0",
author="organization",
description="...",
long_description=read('README.md'),
packages=find_packages(),
license=LICENSE,
classifiers=[
...
],
)

You need to add where you have your module to the system path
import sys
sys.path.append(yourPathHere)
import yourModule
Edit - for clarity - the above goes into the .py that calls your module, not the setup.py that you use for compiling it.
Huh - I see your environment does see it with the grep. Prob disregard the above then!!

How to solve the failure of getting a python file from cpp extension of pytorch using setuptools?

I wanted to try a github project named deformable kernels, and followed the steps described in the README.md file:
conda env create -f environment.yml
cd deformable_kernels/ops/deform_kernel;
pip install -e .;
The structure of deformable_kernel/ops/deform_kernel is showed here:
.
csrc
filter_sample_depthwise_cuda.cpp
filter_sample_depthwise_cuda.h
filter_sample_depthwise_cuda_kernel.cu
nd_linear_sample_cuda.cpp
nd_linear_sample_cuda.h
nd_linear_sample_cuda_kernel.cu
functions
filter_sample_depthwise.py
__init__.py
nd_linear_sample.py
__init__.py
modules
filter_sample_depthwise.py
__init__.py
setup.py
And the content of file setup.py is showed here:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
setup(
name='filter_sample_depthwise',
ext_modules=[
CUDAExtension(
'filter_sample_depthwise_cuda',
[
'csrc/filter_sample_depthwise_cuda.cpp',
'csrc/filter_sample_depthwise_cuda_kernel.cu',
]
),
],
cmdclass={'build_ext': BuildExtension}
)
setup(
name="nd_linear_sample",
ext_modules=[
CUDAExtension(
"nd_linear_sample_cuda",
[
"csrc/nd_linear_sample_cuda.cpp",
"csrc/nd_linear_sample_cuda_kernel.cu",
],
)
],
cmdclass={"build_ext": BuildExtension},
)
When I install this directory using command pip install -e ., it failed and the result is:
Obtaining file:///home/xxx/Downloads/deformable_kernels/deformable_kernels/ops/deform_kernel
ERROR: More than one .egg-info directory found in /tmp/pip-pip-egg-info-pta6z__q
So I tried to separate the 2 setup()s in different setup.py files. It worked but I didn't get a python file. Instead a .so file was generated.
Does anyone know how to solve a problem like this?

Check your pip version. I've had the same error (when installing other things in dev mode with pip) and downgrading to pip version 20.0.2 worked. Unsure why, but I've seen other folks on the internet solve the problem similarly.

Unable to import dependency installed from git in setup.py

I am trying to use setup.py to install a Python package that is kept in a git repository, which we'll call my_dependency. In my_package, I have a setup.py file with:
setup(
...
install_requires=[
...
'my_dependency=VERSION'
],
dependency_links=['git+https://...my_dependency.git#egg=my_dependency-VERSION',]
)
When I run my setup file (python setup.py develop), the dependency appears to install; it shows up as my_dependency==VERSION when I run pip freeze. However, when I start a python session and call import my_dependency, I get ImportError: No module named my_dependency.
I don't know if this is possibly the source of the problem, but when running setup.py, I get a warning:
Processing dependencies for my_package==0.1
Searching for my_dependency==VERSION
Doing git clone from https://.../my_dependency.git to /var/folders/.../.../T/easy_install-_rWjyp/my_dependency.git
Best match: my_dependency VERSION
Processing my_dependency.git
Writing /var/folders/.../my_dependency.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/.../my_dependency.git/egg-dist-tmp-UMiNdL
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Copying my_dependency-VERSION-py2.7.egg to /.../my_package/venv/lib/python2.7/site-packages
Adding my_dependency VERSION to easy-install.pth file
However, I am able to use the package if I install it through pip, like this: pip install -e git+https://.../my_dependency.git#egg=my_dependency-VERSION
For reference, the dependency package structure looks like this:
my_dependency/
my_dependency/
__init__.py
setup.py
And its setup.py contains this:
from setuptools import setup
setup(
name='my_dependency',
version='VERSION',
description='...',
author='...',
url='https://...',
license='MIT',
install_requires=[
'numpy',
],
zip_safe=False,
)

The solution was (in retrospect) pretty silly. My dependency package was missing this line in its setup.py:
packages=['my_dependency'],
That meant the package was correctly building and installing, but it wasn't actually including the code in the package. This became apparent when I looked at the SOURCES.txt in the egg-info: it didn't include any of the Python source files in the package.

creating python whl with directories outside package

I want to create a package that has the following structure:
project/
plugin_dir/
templates/
sh/
I have a setup.py file like the following:
setup(name='my_plugin',
version='1.0.0',
packages=['my_plugin'],
install_requires=['slackclient>=1.0.5', 'airflow==1.8', 'snakebite>=2.11.0', 'prometheus-client>=0.0.19'],
dependency_links=[],
# scripts=findall('sh/')+findall('templates/'),
data_files=[('/sh', findall('sh/'))],
include_package_data=True,
zip_safe=False
)
I run the command python setup.py bdist_wheel and it doesn't include the files from sh or templates directory when I look in the wheel.
What am I doing wrong?

Nevermind it works fine, I didn't see it in the build/ but once you unpack the wheel running pip install my_plugin --target . --no-deps it's included.

Python dependency resolution

I have previously created a python package and uploaded it to pypi. The package depends upon 2 other packages defined within the setup.py file:
from setuptools import setup
from dominos.version import Version
def readme():
with open('README.rst') as file:
return file.read()
setup(name='dominos',
version=Version('0.0.1').number,
author='Tomas Basham',
url='https://github.com/tomasbasham/dominos',
license='MIT',
packages=['dominos'],
install_requires=[
'ratelimit',
'requests'
],
include_package_data=True,
zip_safe=False)
As both of these were already installed within my virtualenv this package would run fine.
Now trying to consume this package within another python application (and within a separate virtualenv) I have defined the following requirements.txt file:
dominos==0.0.1
geocoder==1.13.0
For reference dominos is the package I uploaded to pypi. Now running pip install --no-cache-dir -r requirements.txt fails because dependencies of dominos are missing:
ImportError: No module named ratelimit
Surely pip should be resolving these dependencies since I have defined them in the setup.py file of dominos. Clarity on this would be great.