I have a directory with following structure,
.
├── awesome
│ ├── alice
│ │ ├── conf.py
│ │ └── __init__.py
│ ├── bob
│ │ ├── conf.py
│ │ └── __init__.py
│ ├── conf.py
│ ├── __init__.py
│ └── john
│ ├── conf.py
│ └── __init__.py
├── not_awesome_1
│ ├── __init__.py
│ └── something.py
├── not_awesome_2
│ ├── __init__.py
│ └── something.py
└── setup.py
I want to make the awesome package to be shippable. So, I made the setup.py as below,
from setuptools import find_packages, setup
setup(
name="my-awesome-package",
version="0.1.0",
description="",
long_description="",
license="BSD",
packages=find_packages(where="awesome"),
include_package_data=True,
author="JPG",
author_email="foo#gmail.com",
install_requires=[],
)
I ran the command python3 setup.py bdist_wheel and it gave me the result
running bdist_wheel
running build
running build_py
error: package directory 'alice' does not exist
What was I trying to achieve?
I wanted to decouple the awesome package and wanted to reuse it in multiple projects as I'm currently using the same in not_awesome_1 or not_awesome_2 packages.
In other words, after the successful installation of my-awesome-package I should be able to use the awesome packge as
from awesome.alice.conf import Alice
alice = Alice()
What have I tried so far?
replaced packages=find_packages(where="awesome"), with packages=find_packages(),, but, during the build it also includes the not_awesome_X packages as well - which is not intended.
Intriduced package_dir as well
setup(
# other options
packages=find_packages(where="awesome"),
package_dir={"": "awesome"},
)
But, this doesn't allow me to import my packages as from awesome.alice.conf import Alice, but, from alice.conf import Alice (ie, awesome is missing)
Questions?
What was I doing wrong here?
How to properly configure packages and package_dir?
I encountered a similar error. Try manually defining both the top-level package and the sub-packages:
packages=["awesome", "awesome.alice", "awesome.bob", "awesome.john", "awesome.something.somethingelse"].
Edit:
The issue is that using the where kwarg defines the package to search in. Since you have packages in the root of the project that should not be bundled, you'll likely need to manually add the parent package's name in front of each of its sub-packages.
from setuptools import find_packages
if __name__ == "__main__":
print(find_packages(where="awesome"))
# ['bob', 'alice', 'john', 'john.child']
# the problem here is that 'awesome' is the root, not the current directory containing awesome
root_package = "awesome"
print([root_package] + [f"{root_package}.{item}" for item in find_packages(where=root_package)])
# ['awesome', 'awesome.bob', 'awesome.alice', 'awesome.john', 'awesome.john.child']
Then, in your setup.py:
...
root_package = "awesome"
...
setup(
# other options
packages=[root_package] + [f"{root_package}.{item}" for item in find_packages(where=root_package)],
# package_dir={"": "awesome"}, <- not needed
)
Related
I want to run my built CLI like other cli tools, for eg, kubectl, redis, etc. Currently, I run my cli as: python3 cli.py subarg --args; instead, I want to run: invdb subarg --args where invdb is the Python package.
The structure of the project repository is:
.
├── CHALLENGE.md
├── Pipfile
├── Pipfile.lock
├── README.md
├── __pycache__
│ └── config.cpython-38.pyc
├── data_platform_challenge_darwin
├── data_platform_challenge_linux
├── data_platform_challenge_windows
├── discussion_answers_rough_work
├── dist
│ ├── invdb-0.0.1.tar.gz
│ └── invdb-tesla-kebab-mai-haddi-0.0.1.tar.gz
├── example.json
├── invdb
│ ├── __init__.py
│ ├── analysis.py
│ ├── cleanup.py
│ ├── cli.py
│ ├── config.py
│ ├── etl.py
│ ├── groups.py
│ ├── initialize_db.py
│ └── nodes.py
├── invdb.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
├── setup.py
├── test.db
└── tests
setuptools (or is it distutils? The line is so blurry) provides an entry_points.console_scripts option that can do this for you when installing your package. I will provide an example repository at the bottom of my summary.
Construct a project tree like so:
# /mypackage/mymodule.py
print("We did it!")
# /pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
# this is required for Python to recognize setuptools as the build backend
[metadata]
name = sample_module
version = 0.0.1
author = Adam Smith
description = console_script example
[bdist_wheel]
universal = true
[options]
packages = my_package
python_requires = >=2.7
entry_points =
[console_scripts]
sample_module = my_package.my_module:main
then run the following at the shell:
$ python3 -mpip install .
(ed. this will install the file locally. To build a wheel (to install elsewhere) try pep517)
If you get a warning about the installation script not being on your PATH, you should consider adding it. Otherwise, just run your new script
$ sample_module
We did it!
GitLab: nottheeconomist/console_script_example
Since you already have a setup.py, consider adding the following entry to your setuptools.setup call:
# ...
setuptools.setup(
# ...
entry_points = {
'console_scripts': ['sample_module=my_package.my_module:main']
}
)
I'm trying to create a python package, and I've added some files which are needed for the module to function, like this: https://docs.python.org/2/distutils/setupscript.html#distutils-additional-files
Due to circumstances, this is the method I need to use, if at all possible. I also need to end up with a source distributable, so something that works when making other types of python distributables doesn't work for me.
My setup.py looks like this:
from setuptools import setup
setup(name='mypackage',
version='0.1',
py_modules=['mypackage'],
install_requires=['numpy'],
data_files=[('data', ['data/file0.npz', 'data/file1.npz'])]
)
The directory structure looks like this:
├── PKG-INFO
├── data
│ ├── data0.npz
│ └── data1.npz
├── dist
│ ├── mypackage-0.1.zip
├── mypackage.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── requires.txt
│ └── top_level.txt
├── mypackage.py
├── setup.cfg
└── setup.py
I'm trying to load it in like this(every function but init removed for simplicity):
import numpy as np
class MyClass ():
def __init__(self):
self.data0 = np.load("data/file0.npz")
self.data1 = np.load("data/file1.npz")
And get this error when trying to instantiate the class:
No such file or directory: 'data/file0.npz'
What do I need to change to get this working?
To load package resources, I usually use pkg_resources module
Here is an example to get resource file relative to current module:
from pkg_resources import resource_filename
def main():
print(resource_filename(__name__, 'data/test.txt'))
In your setup.py you can use package_data to include package data files.
setup(
name='',
# (...)
package_data={
'': [
'*.txt',
],
},
)
Note: To make it works, data has to be a python module.
I wrote a small tool (package) that reuses an existing namespace, pki.server. I named my package as pki.server.healthcheck. The old namespace did not use setuptools to install the package, while my package uses it.
Contents of setup.py
from setuptools import setup
setup(
name='pkihealthcheck',
version='0.1',
packages=[
'pki.server.healthcheck.core',
'pki.server.healthcheck.meta',
],
entry_points={
# creates bin/pki-healthcheck
'console_scripts': [
'pki-healthcheck = pki.server.healthcheck.core.main:main'
]
},
classifiers=[
'Programming Language :: Python :: 3.6',
],
python_requires='!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
setup_requires=['pytest-runner',],
tests_require=['pytest',],
)
The installation tree (from scenario 1 below) looks like:
# tree /usr/lib/python3.8/site-packages/pki/
├── __init__.py <---- Has methods and classes
├── cli
│ ├── __init__.py <---- Has methods and classes
│ ├── <some files>
├── server
│ ├── cli
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <Some files>
│ ├── deployment
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <some files>
│ │ └── scriptlets
│ │ ├── __init__.py <---- Has methods and classes
│ │ ├── <some files>
│ ├── healthcheck
│ │ ├── core
│ │ │ ├── __init__.py <---- EMPTY
│ │ │ └── main.py
│ │ └── pki
│ │ ├── __init__.py <---- EMPTY
│ │ ├── certs.py
│ │ └── plugin.py
│ └── instance.py <---- Has class PKIInstance
└── <snip>
# tree /usr/lib/python3.8/site-packages/pkihealthcheck-0.1-py3.8.egg-info/
├── PKG-INFO
├── SOURCES.txt
├── dependency_links.txt
├── entry_points.txt
└── top_level.txt
I read the official documentation and experimented with all 3 suggested methods. I saw the following results:
Scenario 1: Native namespace packages
At first, everything seemed smooth. But:
# This used to work before my package gets installed
>>> import pki.server
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pki.server' has no attribute 'instance'
Now, only this works
>>> import pki.server.instance
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
>>> instance
pki-tomcat
Scenario 2: pkgutil-style namespace packages
I am restricted from using this method as my other __init__.py contain classes and functions
Scenario 3: pkg_resources-style namespace packages
Though this method was not-recommended, I went ahead and experimented with it by adding namespace=pki.server.healthcheck to my setup.py. This made all pki.* modules invisible
So I am convinced that Scenario 1 seems to be the closest to what I'm trying to achieve. I was reading an old post to understand more on how import in python works.
My question is: Why does a perfectly working snippet break after I install my package?
Your __init__.py files need to import the files. You have two options--absolute and relative imports:
Relative Imports
pki/__init__.py:
from . import server
pki/server/__init__.py:
from . import instance
Absolute Imports
pki/__init__.py:
import pki.server
pki/server/__init__.py:
import pki.server.instance
What is the right way to package Alembic migration files in a Setuptools setup.py file? Everything is in my repo root as alembic/.
This is a Python application, not a library.
My desired installation flow is that someone can pip install the wheel that is my application. They would then be able to initialize the application database by running something like <app> alembic upgrade --sqlalchemy.url=<db_url>. Upgrades would then require a pip install -U, after which they can run the Alembic command again.
Is this unorthodox?
If not, how would I accomplish this? Certainly a console_scripts entry_points. But beyond that?
I am not sure this is the right way but I did it this way:
First, you can add sort of custom options to alembic using the -x option and you can find details explained in this great answer. This allows you to specify the db_url at runtime and make it override the value in the config.ini.
Then I managed to package alembic and my migrations by moving the alembic.ini file and the alembic directory from my project root to my top-level python package:
<project root>
├── src
│ └── <top-level package dir>
│ ├── alembic
│ │ ├── env.py
│ │ ├── README
│ │ ├── script.py.mako
│ │ └── versions
│ │ ├── 58c8dcd5fbdc_revision_1.py
│ │ └── ec385b47da23_revision_2.py
│ ├── alembic.ini
│ ├── __init__.py
│ └── <other files and dirs>
└── <other files and dirs>
This allows to use the setuptools package_data directive inside my setup.py:
setup(
name=<package_name>,
package_dir={'': 'src'},
packages=find_packages(where='src'),
package_data={
'<top-level package dir>': ['alembic.ini', 'alembic/*', 'alembic/**/*'],
},
[...]
)
A this point, the alembic config and revisions are correctly packaged but the alembic.ini settings have to be tweaked to reflect the new directory tree. It can be done using the %(here)s param which contains the absolute path of the directory containing the alembic.ini file:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = %(here)s/alembic
[...]
# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
version_locations = %(here)s/alembic/versions
[...]
Finally, you have to call alembic with the -c option which allows to provide the path of the config file:
alembic -c <path to alembic.ini> ...
One way to do this which keeps the main alembic folder along the main package folder is to treat the alembic folder as it's own package to be installed along side your main package.
To do this you must rename it (it can't be called alembic, as it will be a top level package, so needs a unique name - I've used migrations), and add a __init__.py file in the alembic folder and the versions folder.
Running the migrations on deployment requires knowing the path to the installed package - a simple way to do this is to provide a console scripts that applies the migrations.
So the project structure looks like this:
<project root>
├── setup.py
├── mypackage
│ └── <project source files...>
│
├── migrations
│ ├── __init__.py
│ ├── alembic.ini
│ ├── apply.py
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ ├── __init__.py
│ ├── 58c8dcd5fbdc_revision_1.py
│ └── ec385b47da23_revision_2.py
│
└── <other files and dirs>
And setup.py:
from setuptools import find_packages
from setuptools import setup
setup(
name='mypackage',
packages=find_packages(exclude=('tests',)),
package_data={'migrations': ['alembic.ini']},
entry_points={
'console_scripts': ['apply-migrations=migrations.apply:main'],
},
install_requires=[
"SQLAlchemy==1.3.0",
"alembic==1.0.10",
# ...
]
)
And finally migrations/apply.py:
# Python script that will apply the migrations up to head
import alembic.config
import os
here = os.path.dirname(os.path.abspath(__file__))
alembic_args = [
'-c', os.path.join(here, 'alembic.ini'),
'upgrade', 'head'
]
def main():
alembic.config.main(argv=alembic_args)
Now after installing your wheel, you will have a command apply-migrations which you can invoke directly. Note the version I've implemented here doesn't have any arguments - though if you wanted to pass eg. --sqlalchemy.url you could add it in alembic_args.
Personally I prefer to set the url in migrations/env.py. For example if you had an environment variable called SQLACLHEMYURL you could add this in migrations/env.py:
import os
config.set_main_options(os.getenv('SQLALCHEMYURL'))
Then you can invoke:
SQLALCHEMYURL=... apply-migrations
On deploment.
I am trying to set a package with sub-packages in python. Here is the tree structure that I have at the moment:
myPackage
├── __init__.py
├── mySubPackage1
│ ├── foo2.py
│ ├── foo.py
│ └── __init__.py
├── mySubPackage2
│ ├── bar2.py
│ ├── bar.py
│ └── __init__.py
└── setup.py
All __init__.py are empty. Here is my setup.py:
from distutils.core import setup
if __name__ == "__main__":
setup(
name='myPackage',
package_dir = {
'mySubPackage1': 'mySubPackage1',
'mySubPackage2': 'mySubPackage2'},
packages=['mySubPackage1', 'mySubPackage2'],
)
The problem is that, when I run python setup.py install from myPackage, the sub packages are installed into dist-packages:
/usr/local/lib/python2.7/dist-packages/mySubPackage1
/usr/local/lib/python2.7/dist-packages/mySubPackage2
I guess the problem is my setup.py, but I don't know how to fix it? Should it be in the parent directory of myPackage? If so, then how does it work when I pack the package into a zip using python setup.py sdist?
Just use setuptools instead of distutils, it has find_packages exactly for that purpose:
from setuptools import setup, find_packages
setup(
name='myPackage',
packages=find_packages(),
)
TL;DR: Nest the package in another package having the same name.
I nested the super-package myPackage inside a directory (having the same name) as follow:
myPackage
├── myPackage
│ ├── __init__.py
│ ├── mySubPackage1
│ │ ├── foo1.py
│ │ ├── foo2.py
│ │ └── __init__.py
│ └── mySubPackage2
│ ├── bar1.py
│ ├── bar2.py
│ └── __init__.py
└── setup.py
Then, I updated the setup.py:
from distutils.core import setup
if __name__ == "__main__":
setup(
name='myPackage',
package_dir = {
'myPackage': 'myPackage',
'myPackage.mySubPackage1': 'myPackage/mySubPackage1',
'myPackage.mySubPackage2': 'myPackage/mySubPackage2'},
packages=['myPackage', 'myPackage.mySubPackage1',
'myPackage.mySubPackage2']
)
Now, sudo python setup.py install behaves as I expect and in dist-packages I have the following structure:
myPackage
├── __init__.py
├── __init__.pyc
├── mySubPackage1
│ ├── foo1.py
│ ├── foo1.pyc
│ ├── foo2.py
│ ├── foo2.pyc
│ ├── __init__.py
│ └── __init__.pyc
└── mySubPackage2
├── bar1.py
├── bar1.pyc
├── bar2.py
├── bar2.pyc
├── __init__.py
└── __init__.pyc
and an egg file.
This is almost good. Now it is not platform independent because of the usage of /. To fix this, I edited setup.py as follow:
from distutils.core import setup
from distutils import util
if __name__ == "__main__":
pathMySubPackage1 = util.convert_path('myPackage/mySubPackage1')
pathMySubPackage2 = util.convert_path('myPackage/mySubPackage2')
setup(
name='myPackage',
package_dir = {
'myPackage': 'myPackage',
'myPackage.mySubPackage1': pathMySubPackage1,
'myPackage.mySubPackage2': pathMySubPackage2},
packages=['myPackage', 'myPackage.mySubPackage1',
'myPackage.mySubPackage2']
)