I am trying to set a package with sub-packages in python. Here is the tree structure that I have at the moment:
myPackage
├── __init__.py
├── mySubPackage1
│ ├── foo2.py
│ ├── foo.py
│ └── __init__.py
├── mySubPackage2
│ ├── bar2.py
│ ├── bar.py
│ └── __init__.py
└── setup.py
All __init__.py are empty. Here is my setup.py:
from distutils.core import setup
if __name__ == "__main__":
setup(
name='myPackage',
package_dir = {
'mySubPackage1': 'mySubPackage1',
'mySubPackage2': 'mySubPackage2'},
packages=['mySubPackage1', 'mySubPackage2'],
)
The problem is that, when I run python setup.py install from myPackage, the sub packages are installed into dist-packages:
/usr/local/lib/python2.7/dist-packages/mySubPackage1
/usr/local/lib/python2.7/dist-packages/mySubPackage2
I guess the problem is my setup.py, but I don't know how to fix it? Should it be in the parent directory of myPackage? If so, then how does it work when I pack the package into a zip using python setup.py sdist?
Just use setuptools instead of distutils, it has find_packages exactly for that purpose:
from setuptools import setup, find_packages
setup(
name='myPackage',
packages=find_packages(),
)
TL;DR: Nest the package in another package having the same name.
I nested the super-package myPackage inside a directory (having the same name) as follow:
myPackage
├── myPackage
│ ├── __init__.py
│ ├── mySubPackage1
│ │ ├── foo1.py
│ │ ├── foo2.py
│ │ └── __init__.py
│ └── mySubPackage2
│ ├── bar1.py
│ ├── bar2.py
│ └── __init__.py
└── setup.py
Then, I updated the setup.py:
from distutils.core import setup
if __name__ == "__main__":
setup(
name='myPackage',
package_dir = {
'myPackage': 'myPackage',
'myPackage.mySubPackage1': 'myPackage/mySubPackage1',
'myPackage.mySubPackage2': 'myPackage/mySubPackage2'},
packages=['myPackage', 'myPackage.mySubPackage1',
'myPackage.mySubPackage2']
)
Now, sudo python setup.py install behaves as I expect and in dist-packages I have the following structure:
myPackage
├── __init__.py
├── __init__.pyc
├── mySubPackage1
│ ├── foo1.py
│ ├── foo1.pyc
│ ├── foo2.py
│ ├── foo2.pyc
│ ├── __init__.py
│ └── __init__.pyc
└── mySubPackage2
├── bar1.py
├── bar1.pyc
├── bar2.py
├── bar2.pyc
├── __init__.py
└── __init__.pyc
and an egg file.
This is almost good. Now it is not platform independent because of the usage of /. To fix this, I edited setup.py as follow:
from distutils.core import setup
from distutils import util
if __name__ == "__main__":
pathMySubPackage1 = util.convert_path('myPackage/mySubPackage1')
pathMySubPackage2 = util.convert_path('myPackage/mySubPackage2')
setup(
name='myPackage',
package_dir = {
'myPackage': 'myPackage',
'myPackage.mySubPackage1': pathMySubPackage1,
'myPackage.mySubPackage2': pathMySubPackage2},
packages=['myPackage', 'myPackage.mySubPackage1',
'myPackage.mySubPackage2']
)
Related
I want to run my built CLI like other cli tools, for eg, kubectl, redis, etc. Currently, I run my cli as: python3 cli.py subarg --args; instead, I want to run: invdb subarg --args where invdb is the Python package.
The structure of the project repository is:
.
├── CHALLENGE.md
├── Pipfile
├── Pipfile.lock
├── README.md
├── __pycache__
│ └── config.cpython-38.pyc
├── data_platform_challenge_darwin
├── data_platform_challenge_linux
├── data_platform_challenge_windows
├── discussion_answers_rough_work
├── dist
│ ├── invdb-0.0.1.tar.gz
│ └── invdb-tesla-kebab-mai-haddi-0.0.1.tar.gz
├── example.json
├── invdb
│ ├── __init__.py
│ ├── analysis.py
│ ├── cleanup.py
│ ├── cli.py
│ ├── config.py
│ ├── etl.py
│ ├── groups.py
│ ├── initialize_db.py
│ └── nodes.py
├── invdb.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
├── setup.py
├── test.db
└── tests
setuptools (or is it distutils? The line is so blurry) provides an entry_points.console_scripts option that can do this for you when installing your package. I will provide an example repository at the bottom of my summary.
Construct a project tree like so:
# /mypackage/mymodule.py
print("We did it!")
# /pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
# this is required for Python to recognize setuptools as the build backend
[metadata]
name = sample_module
version = 0.0.1
author = Adam Smith
description = console_script example
[bdist_wheel]
universal = true
[options]
packages = my_package
python_requires = >=2.7
entry_points =
[console_scripts]
sample_module = my_package.my_module:main
then run the following at the shell:
$ python3 -mpip install .
(ed. this will install the file locally. To build a wheel (to install elsewhere) try pep517)
If you get a warning about the installation script not being on your PATH, you should consider adding it. Otherwise, just run your new script
$ sample_module
We did it!
GitLab: nottheeconomist/console_script_example
Since you already have a setup.py, consider adding the following entry to your setuptools.setup call:
# ...
setuptools.setup(
# ...
entry_points = {
'console_scripts': ['sample_module=my_package.my_module:main']
}
)
I have a directory structure which looks like below :
.
├── bitbucket-pipelines.yml
├── MANIFEST.in
├── pylintrc
├── setup.cfg
├── setup.py
├── src
│ ├── bin
│ │ ├── __init__.py
│ │ └── project.py
│ ├── __init__.py
│ └── ml_project
│ ├── configurations
│ │ └── precommit
│ ├── core
│ │ ├── command
│ │ │ ├── abs_command.py
│ │ │ ├── __init__.py
│ │ │ ├── no_command.py
│ │ │ ├── precommit.py
│ │ │ ├── project_utils.py
│ │ │ ├── setupsrc.py
│ │ │ └── setuptox.py
│ │ ├── configurations
│ │ │ └── precommit
│ │ └── __init__.py
│ └── __init__.py
└── tox.ini
When i do the packaging for the project my requirement is to basically copy the files .gitlint and .pre-commit-config.yaml files inside the configurations/precommit folder of my ml_project package. configurations is just a normal directory and not a Python package as it does not contain .py files.
A small edit the .gitlint and .pre-commit-config.yaml are in the same level as setup.py is.
My setup.py looks like below :
"""Setup script."""
import io
import re
import os
import shutil
from setuptools import setup
PROJECT_NAME = "ml_project"
CONFIGURATIONS_DIR_NAME = "configurations"
FULL_CONFIG_DIR = os.path.join("src", PROJECT_NAME, CONFIGURATIONS_DIR_NAME)
def get_version() -> str:
"""Return the version stored in `ml_project/__init__.py:__version__`."""
# see https://github.com/pallets/flask/blob/master/setup.py
with io.open("src/ml_project/__init__.py", "rt", encoding="utf8") as init_file:
return re.search(r'__version__ = "(.*?)"', init_file.read()).group(1)
def add_config_files_for_package(source_dir: str = None) -> None:
if not source_dir:
source_dir = os.path.dirname(os.path.abspath(__file__))
config_files = {"precommit": [".gitlint", ".pre-commit-config.yaml"]}
for config in config_files:
config_dir = os.path.join(source_dir, FULL_CONFIG_DIR, config)
for file in config_files[config]:
shutil.copyfile(
os.path.join(source_dir, file), os.path.join(config_dir, file)
)
add_config_files_for_package()
setup(version=get_version())
So i am using the add_config_files_for_package function to do the copying when i run python setup.py sdist.
I have a MANIFEST.in file which looks like below :
include .gitlint
include .pre-commit-config.yaml
graft src/ml_project
And finally below is my setup.cfg :
[options]
package_dir =
=src
packages = find:
include_package_data = true
install_requires =
click
pre-commit
pyyaml
gitlint
[options.packages.find]
where = src
[options.entry_points]
console_scripts =
project = bin.project:main
[options.extras_require]
tests =
pytest
pytest-mock
pyfakefs
pyyaml
configparser
linting =
pylint
testdocs =
pydocstyle
pre-commit =
pre-commit
[semantic_release]
version_variable = ml_project/__init__.py:__version__
This runs fine but my question is : is there a better and more standard way of doing this stuff ? Like without writing the function in the first place at all?
Thanks for any pointers in advance.
As mentioned in the comments, it could be a good idea to place these files in the src/ml_project/configurations/precommit directory and create symbolic links to these files at the root of the project. Symbolic links should play well with git, but some platforms (Windows for example) don't have good support for them.
Or the copy of these files could be just another step in the build process (eventually via a custom setuptools command), that should be triggered from a Makefile (for example, or any other similar tool), and from the CI/CD toolchains (bitbucket-pipelines.yml in this case).
I wrote a small tool (package) that reuses an existing namespace, pki.server. I named my package as pki.server.healthcheck. The old namespace did not use setuptools to install the package, while my package uses it.
Contents of setup.py
from setuptools import setup
setup(
name='pkihealthcheck',
version='0.1',
packages=[
'pki.server.healthcheck.core',
'pki.server.healthcheck.meta',
],
entry_points={
# creates bin/pki-healthcheck
'console_scripts': [
'pki-healthcheck = pki.server.healthcheck.core.main:main'
]
},
classifiers=[
'Programming Language :: Python :: 3.6',
],
python_requires='!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
setup_requires=['pytest-runner',],
tests_require=['pytest',],
)
The installation tree (from scenario 1 below) looks like:
# tree /usr/lib/python3.8/site-packages/pki/
├── __init__.py <---- Has methods and classes
├── cli
│ ├── __init__.py <---- Has methods and classes
│ ├── <some files>
├── server
│ ├── cli
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <Some files>
│ ├── deployment
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <some files>
│ │ └── scriptlets
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <some files>
│ ├── healthcheck
│ │ ├── core
│ │ │ ├── __init__.py <---- EMPTY
│ │ │ └── main.py
│ │ └── pki
│ │ ├── __init__.py <---- EMPTY
│ │ ├── certs.py
│ │ └── plugin.py
│ └── instance.py <---- Has class PKIInstance
└── <snip>
# tree /usr/lib/python3.8/site-packages/pkihealthcheck-0.1-py3.8.egg-info/
├── PKG-INFO
├── SOURCES.txt
├── dependency_links.txt
├── entry_points.txt
└── top_level.txt
I read the official documentation and experimented with all 3 suggested methods. I saw the following results:
Scenario 1: Native namespace packages
At first, everything seemed smooth. But:
# This used to work before my package gets installed
>>> import pki.server
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pki.server' has no attribute 'instance'
Now, only this works
>>> import pki.server.instance
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
>>> instance
pki-tomcat
Scenario 2: pkgutil-style namespace packages
I am restricted from using this method as my other __init__.py contain classes and functions
Scenario 3: pkg_resources-style namespace packages
Though this method was not-recommended, I went ahead and experimented with it by adding namespace=pki.server.healthcheck to my setup.py. This made all pki.* modules invisible
So I am convinced that Scenario 1 seems to be the closest to what I'm trying to achieve. I was reading an old post to understand more on how import in python works.
My question is: Why does a perfectly working snippet break after I install my package?
Your __init__.py files need to import the files. You have two options--absolute and relative imports:
Relative Imports
pki/__init__.py:
from . import server
pki/server/__init__.py:
from . import instance
Absolute Imports
pki/__init__.py:
import pki.server
pki/server/__init__.py:
import pki.server.instance
Background
My code has the following structure:
.
└── my_package
├── __init__.py
├── classif
│ ├── __init__.py
│ ├── inference.py
│ └── models.py
├── datasets
│ ├── __init__.py
│ └── datasets.py
└── matching
├── __init__.py
├── elastic.py
└── search.py
After a sphinx-quickstart (with autodoc) I ran sphinx-apidoc as follows:
sphinx-apidoc -f -o source my_package -e -M
I now have the following files:
.
├── Makefile
├── build
├── my_package
│ ├── __init__.py
│ ├── __pycache__
│ ├── classif
│ ├── datasets
│ └── matching
└── source
├── _static
├── _templates
├── conf.py
├── index.rst
├── modules.rst
├── my_package.classif.inference.rst
├── my_package.classif.models.rst
├── my_package.classif.rst
├── my_package.datasets.datasets.rst
├── my_package.datasets.rst
├── my_package.matching.elastic.rst
├── my_package.matching.rst
├── my_package.matching.search.rst
└── my_package.rst
I also modified conf.py to add:
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('..'))
and added modules to the index.rst:
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
# [more lines]
Now here is the issue
After running make clean ; make html I get the warnings:
/path/to/my_package.classif.rst: WARNING: document isn't included in any toctree
/path/to/my_package.datasets.rst: WARNING: document isn't included in any toctree
/path/to/my_package.matching.rst: WARNING: document isn't included in any toctree
Which makes sense as sphinx-apidoc did not reference them in my_package.rst. How can I solve this?
The issue was actually known -> https://github.com/sphinx-doc/sphinx/issues/4446
Upgrading to sphinx 1.7.6 solved it.
May be lame question for many of you smart people out there, but I am struggling with a simple python package creation.
My dir structure for package is :
address-book/
├── __init__.py
├── dist
│ └── book-0.1.tar.gz
├── address-book
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── person.py
│ └── person.pyc
├── address_book.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── not-zip-safe
│ └── top_level.txt
├── setup.py
└── tests
├── __init__.py
├── __init__.pyc
├── person_test.py
└── person_test.pyc
The setup.py is as follows:
from setuptools import setup, find_packages
setup(name='address-book',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Address Book',
author_email='flyingcircus#example.com',
license='MIT',
packages=find_packages('.'),
test_suite="tests",
zip_safe=False)
The SOURCES.txt:
setup.py
address-book/__init__.py
address-book/person.py
address_book.egg-info/PKG-INFO
address_book.egg-info/SOURCES.txt
address_book.egg-info/dependency_links.txt
address_book.egg-info/not-zip-safe
address_book.egg-info/top_level.txt
tests/__init__.py
tests/person_test.py
In the person_test.py I am not able to import person.py, what could be the reason?
Solution
In case if someone struggles with the same problem, mine got fixed by not using hyphens as package name. Simple and worked!
It looks like you have address-book and address_book.egg-info. I think it should be address-book.egg-info.