I am trying to create a conda package to distribute a python tool. Part of the tool is cythonized, and it works perfectly using python setup.py install.
I create the tar properly but when I try to install it, the package does not contain the .py files that links the python imports and the .so files.
So when I try to import that packages I get a module not found.
The only think I have found around cython and conda is to introduce cython requirement in the build/run section in the meta.yaml, but I don't know why those .py files are not included.
This is my meta.yaml
package:
name: project
version: 1.0.0
source:
path: /home/user/project
requirements:
build:
- python >=2.7
- jinja2
- numpy
- scipy
- matplotlib
- pysam
- setuptools
- h5py
- cython
run:
- python >=2.7
- jinja2
- numpy
- scipy
- matplotlib
- pysam >=0.8
- setuptools
- h5py
- cython
build:
preserve_egg_dir: True
entry_points:
- exec_file = project.run_exec:main
about:
license: GPL3
summary: "PROJECT"
my setup.py file looks like
from setuptools import setup, find_packages
from distutils.core import Extension
from Cython.Build import cythonize
extensions = [Extension('project.src.norm', ['project/src/norm.pyx'])]
setup(
name="PROJECT",
packages=find_packages(),
version="1.0.0",
description="PROJECT",
author='Lab',
author_email='email',
url='http://',
license='LICENSE.txt',
include_package_data=True,
entry_points={'console_scripts': ['exec_file = project.run_exec:main']},
zip_safe=False,
ext_modules=cythonize(extensions),
classifiers=[
'Development Status :: 4 - Beta',
'Environment :: Console',
'Intended Audience :: Bioinformaticians',
'License :: OSI Approved :: BSD License',
'Operating System :: MacOS',
'Operating System :: Microsoft :: Windows',
'Operating System :: POSIX',
'Programming Language :: Python :: 2.7',
]
)
The directory structures is
project/
setup.py
__init__.py
MANIFEST.in
requirements.txt
README.md
info/
meta.yaml
build.sh
bld.bat
project/
src/
norm.pyx
run_exec.py
subproject/
<etc...>
EDITED:
Today I tried using python setup.py bdist_conda but the behavior is the same, or it is a conda issue or it is an specific problem on my configuration.
if that is the case I guess is is setup.py....
I did find the problem was in my environment. I had defined a PYTHONPATH variable that was making conda generate incorrectly the package. Instead of going to the build package, it was importing my source code directly. Removing the PYTHONPATH the problem is solved.
Related
This question already has answers here:
Pip install from pypi works, but from testpypi fails (cannot find requirements)
(2 answers)
Closed 2 years ago.
I am building a Python package and my package has some install requirements. This is my setup.py file code:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="simpleEDA",
version="0.0.1",
author="Muhammad Shahid Sharif",
author_email="chshahidhamdam#gmail.com",
description="A wrapper around Pandas to perform Simple EDA with less code.",
long_description=long_description,
long_description_content_type="text/markdown",
url="github link here",
packages=['simple_eda'],
install_requires = ['matplotlib',
'numpy',
'numpydoc',
'pandas',
'scikit-image',
'scikit-learn',
'scipy',
'seaborn'],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independen.t",
],
python_requires='>=3.5',
)
I have created the whl file and uploaded it on test PyPI. here is the link
pip install -i https://test.pypi.org/simple/ simpleEDA==0.0.1
If I try to install it, it gives me this error.
Could not find a version that satisfies the requirement numpydoc (from simpleEDA==0.0.1) (from versions: )
No matching distribution found for numpydoc (from simpleEDA==0.0.1)
Why is my install_requires is not working? Why it is not installing libraries?
You're attempting to install using TestPyPI as the index:
pip install -i https://test.pypi.org/simple/ simpleEDA==0.0.1
However most of your subdependencies don't exist on TestPyPI, for example https://test.pypi.org/project/numpydoc/ is 404.
Depending on what you're using TestPyPI for, you might be better off making a pre-release on PyPI instead.
I am trying to make a python package which I want to install using pip install . locally. The package name is listed in pip freeze but import <package> results in an error No module named <package>. Also the site-packages folder does only contain a dist-info folder. find_packages() is able to find packages. What am I missing?
import io
import os
import sys
from shutil import rmtree
from setuptools import find_packages, setup, Command
# Package meta-data.
NAME = '<package>'
DESCRIPTION = 'description'
URL = ''
EMAIL = 'email'
AUTHOR = 'name'
# What packages are required for this module to be executed?
REQUIRED = [
# 'requests', 'maya', 'records',
]
# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!
here = os.path.abspath(os.path.dirname(__file__))
# Where the magic happens:
setup(
name=NAME,
#version=about['__version__'],
description=DESCRIPTION,
# long_description=long_description,
author=AUTHOR,
author_email=EMAIL,
url=URL,
packages=find_packages(),
# If your package is a single module, use this instead of 'packages':
# py_modules=['mypackage'],
# entry_points={
# 'console_scripts': ['mycli=mymodule:cli'],
# },
install_requires=REQUIRED,
include_package_data=True,
license='MIT',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy'
],
)
Since the question has become quite popular, here are the diagnosis steps to go through when you're missing files after installation. Imagine having an example project with the following structure:
root
├── spam
│ ├── __init__.py
│ ├── data.txt
│ ├── eggs.py
│ └── fizz
│ ├── __init__.py
│ └── buzz.py
├── bacon.py
└── setup.py
Now I run pip install ., check that the package is installed:
$ pip list
Package Version
---------- -------
mypkg 0.1
pip 19.0.1
setuptools 40.6.3
wheel 0.32.3
but see neither spam, nor spam/eggs.py nor bacon.py nor spam/fizz/buzz.py in the list of files belonging to the installed package:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
mypkg-0.1.dist-info/DESCRIPTION.rst
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/metadata.json
mypkg-0.1.dist-info/top_level.txt
So what to do now?
Diagnose by inspecting the wheel build log
Unless told not to do so, pip will always try to build a wheel file and install your package from it. We can inspect the log for the wheel build process if reinstalling in the verbose mode. First step is to uninstall the package:
$ pip uninstall -y mypkg
...
then install it again, but now with an additional argument:
$ pip install . -vvv
...
Now if I inspect the log:
$ pip install . -vvv | grep 'adding'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
I notice that no files from the spam directory or bacon.py are mentioned anywhere. This means they were simply not included in the wheel file and hence not installed by pip. The most common error sources are:
Missing packages: check the packages argument
Verify you have passed the packages argument to the setup function. Check that you have mentioned all of the packages that should be installed. Subpackages will not be collected automatically if only the parent package is mentioned! For example, in the setup script
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam']
)
spam will be installed, but not spam.fizz because it is a package itself and must be mentioned explicitly. Fixing it:
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam', 'spam.fizz']
)
If you have lots of packages, use setuptools.find_packages to automate the process:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages() # will return a list ['spam', 'spam.fizz']
)
In case you are missing a module:
Missing modules: check the py_modules argument
In the above examples, I will be missing bacon.py after installation since it doesn't belong to any package. I have to provide its module name in the separate argument py_modules:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
py_modules=['bacon']
)
Missing data files: check the package_data argument
I have all the source code files in place now, but the data.txt file is still not installed. Data files located under package directories should be added via the package_data argument. Fixing the above setup script:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
package_data={'spam': ['data.txt']},
py_modules=['bacon']
)
Don't be tempted to use the data_files argument. Place the data files under a package and configure package_data instead.
After fixing the setup script, verify the package files are in place after installation
If I now reinstall the package, I will notice all of the files are added to the wheel:
$ pip install . -vvv | grep 'adding'
adding 'bacon.py'
adding 'spam/__init__.py'
adding 'spam/data.txt'
adding 'spam/eggs.py'
adding 'spam/fizz/__init__.py'
adding 'spam/fizz/buzz.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
They will also be visible in the list of files belonging to mypkg:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
__pycache__/bacon.cpython-36.pyc
bacon.py
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/top_level.txt
spam/__init__.py
spam/__pycache__/__init__.cpython-36.pyc
spam/__pycache__/eggs.cpython-36.pyc
spam/data.txt
spam/eggs.py
spam/fizz/__init__.py
spam/fizz/__pycache__/__init__.cpython-36.pyc
spam/fizz/__pycache__/buzz.cpython-36.pyc
spam/fizz/buzz.py
For me, I noticed something weird if you do this:
# Not in the setup.py directory
python /path/to/folder/setup.py bdist_wheel
It will only install the .dist-info folder in your site-packages folder when you install the wheel.
However, if you do this:
cd /path/to/folder \
&& python setup.py bdist_wheel
The wheel will include all your files.
I had the same problem, and updating setuptools helped:
python3 -m pip install --upgrade pip setuptools wheel
After that, reinstall the package, and it should work fine :)
Make certain that your src files are in example_package_YOUR_USERNAME_HERE (this is the example package name that is used in the docs) and not in src. Errantly putting the files in src can have the effect described in the question.
Reference: https://packaging.python.org/en/latest/tutorials/packaging-projects/
The package should be set up like this:
packaging_tutorial/
└── src/
└── example_package_YOUR_USERNAME_HERE/
├── __init__.py
└── example.py
If you are on Windows 10+, one way you could make sure that you had all the correct installations was to click start in the bottom left-hand corner and search cmd.exe and right-click on "Command Prompt" (Make sure you choose "Run as Administrator"). Type "cd path to your Python 3.X installation". You can find this path in File Explorer (go to the folder where Python is installed) and then at the top. Copy this, and put it in where I wrote above path to your Python 3.X installation. Once you do that and click enter, type "python -m pip install package" (package signifies the package you would like to install). Your Python program should now work perfectly.
I am trying to make a python package which I want to install using pip install . locally. The package name is listed in pip freeze but import <package> results in an error No module named <package>. Also the site-packages folder does only contain a dist-info folder. find_packages() is able to find packages. What am I missing?
import io
import os
import sys
from shutil import rmtree
from setuptools import find_packages, setup, Command
# Package meta-data.
NAME = '<package>'
DESCRIPTION = 'description'
URL = ''
EMAIL = 'email'
AUTHOR = 'name'
# What packages are required for this module to be executed?
REQUIRED = [
# 'requests', 'maya', 'records',
]
# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!
here = os.path.abspath(os.path.dirname(__file__))
# Where the magic happens:
setup(
name=NAME,
#version=about['__version__'],
description=DESCRIPTION,
# long_description=long_description,
author=AUTHOR,
author_email=EMAIL,
url=URL,
packages=find_packages(),
# If your package is a single module, use this instead of 'packages':
# py_modules=['mypackage'],
# entry_points={
# 'console_scripts': ['mycli=mymodule:cli'],
# },
install_requires=REQUIRED,
include_package_data=True,
license='MIT',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'License :: OSI Approved :: MIT License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy'
],
)
Since the question has become quite popular, here are the diagnosis steps to go through when you're missing files after installation. Imagine having an example project with the following structure:
root
├── spam
│ ├── __init__.py
│ ├── data.txt
│ ├── eggs.py
│ └── fizz
│ ├── __init__.py
│ └── buzz.py
├── bacon.py
└── setup.py
Now I run pip install ., check that the package is installed:
$ pip list
Package Version
---------- -------
mypkg 0.1
pip 19.0.1
setuptools 40.6.3
wheel 0.32.3
but see neither spam, nor spam/eggs.py nor bacon.py nor spam/fizz/buzz.py in the list of files belonging to the installed package:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
mypkg-0.1.dist-info/DESCRIPTION.rst
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/metadata.json
mypkg-0.1.dist-info/top_level.txt
So what to do now?
Diagnose by inspecting the wheel build log
Unless told not to do so, pip will always try to build a wheel file and install your package from it. We can inspect the log for the wheel build process if reinstalling in the verbose mode. First step is to uninstall the package:
$ pip uninstall -y mypkg
...
then install it again, but now with an additional argument:
$ pip install . -vvv
...
Now if I inspect the log:
$ pip install . -vvv | grep 'adding'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
I notice that no files from the spam directory or bacon.py are mentioned anywhere. This means they were simply not included in the wheel file and hence not installed by pip. The most common error sources are:
Missing packages: check the packages argument
Verify you have passed the packages argument to the setup function. Check that you have mentioned all of the packages that should be installed. Subpackages will not be collected automatically if only the parent package is mentioned! For example, in the setup script
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam']
)
spam will be installed, but not spam.fizz because it is a package itself and must be mentioned explicitly. Fixing it:
from setuptools import setup
setup(
name='mypkg',
version='0.1',
packages=['spam', 'spam.fizz']
)
If you have lots of packages, use setuptools.find_packages to automate the process:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages() # will return a list ['spam', 'spam.fizz']
)
In case you are missing a module:
Missing modules: check the py_modules argument
In the above examples, I will be missing bacon.py after installation since it doesn't belong to any package. I have to provide its module name in the separate argument py_modules:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
py_modules=['bacon']
)
Missing data files: check the package_data argument
I have all the source code files in place now, but the data.txt file is still not installed. Data files located under package directories should be added via the package_data argument. Fixing the above setup script:
from setuptools import find_packages, setup
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
package_data={'spam': ['data.txt']},
py_modules=['bacon']
)
Don't be tempted to use the data_files argument. Place the data files under a package and configure package_data instead.
After fixing the setup script, verify the package files are in place after installation
If I now reinstall the package, I will notice all of the files are added to the wheel:
$ pip install . -vvv | grep 'adding'
adding 'bacon.py'
adding 'spam/__init__.py'
adding 'spam/data.txt'
adding 'spam/eggs.py'
adding 'spam/fizz/__init__.py'
adding 'spam/fizz/buzz.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
They will also be visible in the list of files belonging to mypkg:
$ pip show -f mypkg
Name: mypkg
Version: 0.1
...
Files:
__pycache__/bacon.cpython-36.pyc
bacon.py
mypkg-0.1.dist-info/INSTALLER
mypkg-0.1.dist-info/METADATA
mypkg-0.1.dist-info/RECORD
mypkg-0.1.dist-info/WHEEL
mypkg-0.1.dist-info/top_level.txt
spam/__init__.py
spam/__pycache__/__init__.cpython-36.pyc
spam/__pycache__/eggs.cpython-36.pyc
spam/data.txt
spam/eggs.py
spam/fizz/__init__.py
spam/fizz/__pycache__/__init__.cpython-36.pyc
spam/fizz/__pycache__/buzz.cpython-36.pyc
spam/fizz/buzz.py
For me, I noticed something weird if you do this:
# Not in the setup.py directory
python /path/to/folder/setup.py bdist_wheel
It will only install the .dist-info folder in your site-packages folder when you install the wheel.
However, if you do this:
cd /path/to/folder \
&& python setup.py bdist_wheel
The wheel will include all your files.
I had the same problem, and updating setuptools helped:
python3 -m pip install --upgrade pip setuptools wheel
After that, reinstall the package, and it should work fine :)
Make certain that your src files are in example_package_YOUR_USERNAME_HERE (this is the example package name that is used in the docs) and not in src. Errantly putting the files in src can have the effect described in the question.
Reference: https://packaging.python.org/en/latest/tutorials/packaging-projects/
The package should be set up like this:
packaging_tutorial/
└── src/
└── example_package_YOUR_USERNAME_HERE/
├── __init__.py
└── example.py
If you are on Windows 10+, one way you could make sure that you had all the correct installations was to click start in the bottom left-hand corner and search cmd.exe and right-click on "Command Prompt" (Make sure you choose "Run as Administrator"). Type "cd path to your Python 3.X installation". You can find this path in File Explorer (go to the folder where Python is installed) and then at the top. Copy this, and put it in where I wrote above path to your Python 3.X installation. Once you do that and click enter, type "python -m pip install package" (package signifies the package you would like to install). Your Python program should now work perfectly.
I'm trying to install my package using pip from github repository. All actions i do in venv (w/ --no-site-packages).
In package i use pbr. I can't install one of my requirements that located in private repository (i use ssh link), i'm getting strange message:
Requirement already satisfied: python>=lpcclient in
/usr/lib/python2.7/lib-dynload (from ansible-lpc-
modules==0.0.1.dev168)
setup.py:
from setuptools import setup
setup(setup_requires=['pbr>=1.8'], pbr=True)
setup.cfg:
[metadata]
name = ansible-lpc-modules
author = Petr
version = 0.0.1
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
Development Status :: In Progress
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2.7
[pbr]
skip_authors = True
skip_changelog = True
[files]
packages =
ansible/modules/lpc_mdls
ansible/module_utils/lpc_utils
requirements.txt:
ansible
-e git+ssh://git#git.lpc.org/vpc/python-lpcclient.git#develop#egg=python-lpcclient
What i do wrong? I don't have 'lpcclient' installed globally.
I'm trying to install my own program via Pip and the PyPI with the usual command pip install tvrenamr however I'm getting the error below:
Downloading/unpacking tvrenamr
Running setup.py egg_info for package tvrenamr
Usage: tvr [options] <file/folder>
-c: error: no such option: --egg-base
Complete output from command python setup.py egg_info:
Usage: tvr [options] <file/folder>
-c: error: no such option: --egg-base
----------------------------------------
Command python setup.py egg_info failed with error code 2
Storing complete log in /Users/george/.pip/pip.log
It's been a while since I checked but I used to be able to pip install my code with the previous versions I put on the PyPI, however I updated to the latest Pip - not sure if would cause things to break or not!
So when pip is running python setup.py egg_info it seems that TvRenamr is being called, or at least my option parser class has been imported.
My setup.py:
from os.path import dirname, join
from setuptools import setup, find_packages
from tvrenamr import get_version
def fread(fname):
return open(join(dirname(__file__), fname)).read()
setup(
name = 'tvrenamr',
version = get_version(),
description = 'Rename tv show files using online databases',
long_description = fread('README.markdown'),
author = 'George Hickman',
author_email = 'george#ghickman.co.uk',
url = 'http://github.com/ghickman/tvrenamr',
license = 'MIT',
packages = find_packages(exclude=['tests']),
entry_points = {'console_scripts': ['tvr = tvrenamr.tvrenamr:run',],},
classifiers = [
'Environment :: Console',
'Intended Audience :: End Users/Desktop',
'License :: OSI Approved :: MIT License',
'Operating System :: MacOS',
'Operating System :: Microsoft :: Windows',
'Operating System :: POSIX :: Linux',
'Programming Language :: Python :: 2.6',
'Topic :: Multimedia',
'Topic :: Utilities',
'Natural Language :: English'],
install_requires = ('lxml', 'pyyaml',)
)
tvrenamr/__init__.py:
__version__ = (3, 0, 0)
def get_version():
return '.'.join(map(str, __version__))
My only thoughts on how it's getting tvrenamr's options now are that either find_packages or the entry_points option are in some way importing tvrenamr.py and thus options.py??
All versions of TvRenamr were uploaded to the PyPI with python setup.py sdist upload.
I really am stumped with this problem - any help much appreciated!
EDIT: I can run python setup.py egg_info with no errors.
Unfortunately this was a case of setup tools masking an error in the setup.py that was being caused by a bad version string in tvrenamr/__init__.py.
I picked up on the error after installing manually with python setup.py install into a clean virtualenv so something in my environment must have been affecting things before too.