What is the right way to package Alembic migration files in a Setuptools setup.py file? Everything is in my repo root as alembic/.
This is a Python application, not a library.
My desired installation flow is that someone can pip install the wheel that is my application. They would then be able to initialize the application database by running something like <app> alembic upgrade --sqlalchemy.url=<db_url>. Upgrades would then require a pip install -U, after which they can run the Alembic command again.
Is this unorthodox?
If not, how would I accomplish this? Certainly a console_scripts entry_points. But beyond that?
I am not sure this is the right way but I did it this way:
First, you can add sort of custom options to alembic using the -x option and you can find details explained in this great answer. This allows you to specify the db_url at runtime and make it override the value in the config.ini.
Then I managed to package alembic and my migrations by moving the alembic.ini file and the alembic directory from my project root to my top-level python package:
<project root>
├── src
│ └── <top-level package dir>
│ ├── alembic
│ │ ├── env.py
│ │ ├── README
│ │ ├── script.py.mako
│ │ └── versions
│ │ ├── 58c8dcd5fbdc_revision_1.py
│ │ └── ec385b47da23_revision_2.py
│ ├── alembic.ini
│ ├── __init__.py
│ └── <other files and dirs>
└── <other files and dirs>
This allows to use the setuptools package_data directive inside my setup.py:
setup(
name=<package_name>,
package_dir={'': 'src'},
packages=find_packages(where='src'),
package_data={
'<top-level package dir>': ['alembic.ini', 'alembic/*', 'alembic/**/*'],
},
[...]
)
A this point, the alembic config and revisions are correctly packaged but the alembic.ini settings have to be tweaked to reflect the new directory tree. It can be done using the %(here)s param which contains the absolute path of the directory containing the alembic.ini file:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = %(here)s/alembic
[...]
# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
version_locations = %(here)s/alembic/versions
[...]
Finally, you have to call alembic with the -c option which allows to provide the path of the config file:
alembic -c <path to alembic.ini> ...
One way to do this which keeps the main alembic folder along the main package folder is to treat the alembic folder as it's own package to be installed along side your main package.
To do this you must rename it (it can't be called alembic, as it will be a top level package, so needs a unique name - I've used migrations), and add a __init__.py file in the alembic folder and the versions folder.
Running the migrations on deployment requires knowing the path to the installed package - a simple way to do this is to provide a console scripts that applies the migrations.
So the project structure looks like this:
<project root>
├── setup.py
├── mypackage
│ └── <project source files...>
│
├── migrations
│ ├── __init__.py
│ ├── alembic.ini
│ ├── apply.py
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ ├── __init__.py
│ ├── 58c8dcd5fbdc_revision_1.py
│ └── ec385b47da23_revision_2.py
│
└── <other files and dirs>
And setup.py:
from setuptools import find_packages
from setuptools import setup
setup(
name='mypackage',
packages=find_packages(exclude=('tests',)),
package_data={'migrations': ['alembic.ini']},
entry_points={
'console_scripts': ['apply-migrations=migrations.apply:main'],
},
install_requires=[
"SQLAlchemy==1.3.0",
"alembic==1.0.10",
# ...
]
)
And finally migrations/apply.py:
# Python script that will apply the migrations up to head
import alembic.config
import os
here = os.path.dirname(os.path.abspath(__file__))
alembic_args = [
'-c', os.path.join(here, 'alembic.ini'),
'upgrade', 'head'
]
def main():
alembic.config.main(argv=alembic_args)
Now after installing your wheel, you will have a command apply-migrations which you can invoke directly. Note the version I've implemented here doesn't have any arguments - though if you wanted to pass eg. --sqlalchemy.url you could add it in alembic_args.
Personally I prefer to set the url in migrations/env.py. For example if you had an environment variable called SQLACLHEMYURL you could add this in migrations/env.py:
import os
config.set_main_options(os.getenv('SQLALCHEMYURL'))
Then you can invoke:
SQLALCHEMYURL=... apply-migrations
On deploment.
Related
I have a directory with following structure,
.
├── awesome
│ ├── alice
│ │ ├── conf.py
│ │ └── __init__.py
│ ├── bob
│ │ ├── conf.py
│ │ └── __init__.py
│ ├── conf.py
│ ├── __init__.py
│ └── john
│ ├── conf.py
│ └── __init__.py
├── not_awesome_1
│ ├── __init__.py
│ └── something.py
├── not_awesome_2
│ ├── __init__.py
│ └── something.py
└── setup.py
I want to make the awesome package to be shippable. So, I made the setup.py as below,
from setuptools import find_packages, setup
setup(
name="my-awesome-package",
version="0.1.0",
description="",
long_description="",
license="BSD",
packages=find_packages(where="awesome"),
include_package_data=True,
author="JPG",
author_email="foo#gmail.com",
install_requires=[],
)
I ran the command python3 setup.py bdist_wheel and it gave me the result
running bdist_wheel
running build
running build_py
error: package directory 'alice' does not exist
What was I trying to achieve?
I wanted to decouple the awesome package and wanted to reuse it in multiple projects as I'm currently using the same in not_awesome_1 or not_awesome_2 packages.
In other words, after the successful installation of my-awesome-package I should be able to use the awesome packge as
from awesome.alice.conf import Alice
alice = Alice()
What have I tried so far?
replaced packages=find_packages(where="awesome"), with packages=find_packages(),, but, during the build it also includes the not_awesome_X packages as well - which is not intended.
Intriduced package_dir as well
setup(
# other options
packages=find_packages(where="awesome"),
package_dir={"": "awesome"},
)
But, this doesn't allow me to import my packages as from awesome.alice.conf import Alice, but, from alice.conf import Alice (ie, awesome is missing)
Questions?
What was I doing wrong here?
How to properly configure packages and package_dir?
I encountered a similar error. Try manually defining both the top-level package and the sub-packages:
packages=["awesome", "awesome.alice", "awesome.bob", "awesome.john", "awesome.something.somethingelse"].
Edit:
The issue is that using the where kwarg defines the package to search in. Since you have packages in the root of the project that should not be bundled, you'll likely need to manually add the parent package's name in front of each of its sub-packages.
from setuptools import find_packages
if __name__ == "__main__":
print(find_packages(where="awesome"))
# ['bob', 'alice', 'john', 'john.child']
# the problem here is that 'awesome' is the root, not the current directory containing awesome
root_package = "awesome"
print([root_package] + [f"{root_package}.{item}" for item in find_packages(where=root_package)])
# ['awesome', 'awesome.bob', 'awesome.alice', 'awesome.john', 'awesome.john.child']
Then, in your setup.py:
...
root_package = "awesome"
...
setup(
# other options
packages=[root_package] + [f"{root_package}.{item}" for item in find_packages(where=root_package)],
# package_dir={"": "awesome"}, <- not needed
)
I want to run my built CLI like other cli tools, for eg, kubectl, redis, etc. Currently, I run my cli as: python3 cli.py subarg --args; instead, I want to run: invdb subarg --args where invdb is the Python package.
The structure of the project repository is:
.
├── CHALLENGE.md
├── Pipfile
├── Pipfile.lock
├── README.md
├── __pycache__
│ └── config.cpython-38.pyc
├── data_platform_challenge_darwin
├── data_platform_challenge_linux
├── data_platform_challenge_windows
├── discussion_answers_rough_work
├── dist
│ ├── invdb-0.0.1.tar.gz
│ └── invdb-tesla-kebab-mai-haddi-0.0.1.tar.gz
├── example.json
├── invdb
│ ├── __init__.py
│ ├── analysis.py
│ ├── cleanup.py
│ ├── cli.py
│ ├── config.py
│ ├── etl.py
│ ├── groups.py
│ ├── initialize_db.py
│ └── nodes.py
├── invdb.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
├── setup.py
├── test.db
└── tests
setuptools (or is it distutils? The line is so blurry) provides an entry_points.console_scripts option that can do this for you when installing your package. I will provide an example repository at the bottom of my summary.
Construct a project tree like so:
# /mypackage/mymodule.py
print("We did it!")
# /pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
# this is required for Python to recognize setuptools as the build backend
[metadata]
name = sample_module
version = 0.0.1
author = Adam Smith
description = console_script example
[bdist_wheel]
universal = true
[options]
packages = my_package
python_requires = >=2.7
entry_points =
[console_scripts]
sample_module = my_package.my_module:main
then run the following at the shell:
$ python3 -mpip install .
(ed. this will install the file locally. To build a wheel (to install elsewhere) try pep517)
If you get a warning about the installation script not being on your PATH, you should consider adding it. Otherwise, just run your new script
$ sample_module
We did it!
GitLab: nottheeconomist/console_script_example
Since you already have a setup.py, consider adding the following entry to your setuptools.setup call:
# ...
setuptools.setup(
# ...
entry_points = {
'console_scripts': ['sample_module=my_package.my_module:main']
}
)
I'm trying to create a python package, and I've added some files which are needed for the module to function, like this: https://docs.python.org/2/distutils/setupscript.html#distutils-additional-files
Due to circumstances, this is the method I need to use, if at all possible. I also need to end up with a source distributable, so something that works when making other types of python distributables doesn't work for me.
My setup.py looks like this:
from setuptools import setup
setup(name='mypackage',
version='0.1',
py_modules=['mypackage'],
install_requires=['numpy'],
data_files=[('data', ['data/file0.npz', 'data/file1.npz'])]
)
The directory structure looks like this:
├── PKG-INFO
├── data
│ ├── data0.npz
│ └── data1.npz
├── dist
│ ├── mypackage-0.1.zip
├── mypackage.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── requires.txt
│ └── top_level.txt
├── mypackage.py
├── setup.cfg
└── setup.py
I'm trying to load it in like this(every function but init removed for simplicity):
import numpy as np
class MyClass ():
def __init__(self):
self.data0 = np.load("data/file0.npz")
self.data1 = np.load("data/file1.npz")
And get this error when trying to instantiate the class:
No such file or directory: 'data/file0.npz'
What do I need to change to get this working?
To load package resources, I usually use pkg_resources module
Here is an example to get resource file relative to current module:
from pkg_resources import resource_filename
def main():
print(resource_filename(__name__, 'data/test.txt'))
In your setup.py you can use package_data to include package data files.
setup(
name='',
# (...)
package_data={
'': [
'*.txt',
],
},
)
Note: To make it works, data has to be a python module.
I have the following directory structure in Ubuntu. I'm trying to import the module config from my local package my_app into my script my_app_script.py
$ tree
.
├── my_app/
│ ├── config.py
│ ├── __init__.py
│ └── test/
├── my_app-info # created by pip install -e .
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── bin/
│ └── my_app_script.py
├── LICENSE
├── README.md
└── setup.py
# setup.py
setup(
name='my_app',
version='0.1.2',
description='',
url='',
packages=['my_app'],
scripts=['bin/my_app_script.py'],
install_requires=[],
python_requires='>=3.6',
)
# my_app_script.py
from my_app import config
When I run my_app_script.py it results in "ImportError: cannot import name 'config'
What am I doing wrong?
Edit:
I am trying to follow this guide on packaging a program.
You need an __init__.py file in the parent directory as well as in bin directory.
You can use either of below approaches.
The first approach seems best to me as the script will always set the path relative to it, and will also work if you clone your repos.
Add an __init__.py in parent directory(Next to setup.py).
And add below line in my_app_script.py
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, "my_app")))
What this will do is add .../my_app to PYTHONPATH at runtime, when my_app_script.py is executed.
Add an env.sh in parent directory. Below should be the contents of env.sh.
export PYTHONPATH=$(pwd):$PYTHONPATH
Now cd int the directory where env.sh is kept and then source it
source env.sh
Now run your my_app_script.py
python bin/my_app_script.py
Set PYTHONPATH from commandline.
PYTHONPATH=$(pwd) python bin/my_app_script.py
Trying to configure pytest with django, the project already has a lot of test not written with pytest (written with unittest) but I am trying to get them run with pytest so I can write pytest tests and get it work with old tests.
I know pytest-django checks for the manage.py file in the root dir of a django project but this project the manage.py file is not in the root dir so I guess that's why the error below is thrown when I run pytest however running pytest and supplying a particular file works. How do I specify where manage.py is? As I can't find this in the documentation
pytest-django could not find a Django project (no manage.py file could be found).
You must explicitly add your Django project to the Python path to have it picked up.
you can define a python path to python commands that you want to run:
PYTHONPATH=/your/path/to/your/django/project/ pytest
or export your pythonpath before you run the pytest command:
export PYTHONPATH=/your/path/to/your/django/project/
pytest
As a standard practice, you should add a setup.cfg file to your root, with the following block -
[tool:pytest]
DJANGO_SETTINGS_MODULE=<package_name>.settings.py
You can later use the same file for linters by adding specific blocks for them.
In my case, I created a configure file named pytest.ini in the tests folder with the following content:
[pytest]
pythonpath = ../bitpin
DJANGO_SETTINGS_MODULE = bitpin.settings
python_files = tests.py test_*.py *_tests.py
Repository tree:
bitpin-repo
├── bitpin
│ ├── bitpin
│ │ ├── asgi.py
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ ├── __init__.py
│ └── manage.py
├── tests
│ ├── test_models.py
│ ├── pytest.ini
│ └── conftest.py
└── setup.py
[NOTE]
If the tests/ is next to the project, replace bitpin with .:
[pytest]
pythonpath = .