When create a console_scripts as an entry_points, how do you access a data file (package_data) within the package?
setup(
# other stuff
entry_points={
'console_scripts': [
'runme = mypackage.a_py_file:a_function_within_the_py_file',
]
}
)
Directory structure:
├── mypackage
│ ├── __init__.py
│ └── a_py_file.py
├── requirements.txt
├── setup.py
└── data
├── a.data
└── b.data
Python file to handle console_scripts:
# a_py_file.py
def a_function_within_the_py_file:
# HOW TO READ a.data FILE HERE
How about changing cwd?
import os
os.chdir(__file__)
conftest.py sounds like a good place to do this. Or the file which is attached to your test command.
So this is what I did and it worked:
import os
import pkg_resources
os.chdir(pkg_resources.get_distribution('mypackage').location)
# running as if the script is invoked from project's root
Related
I am trying to better understand importing modules. I read about how to do this from here https://stackoverflow.com/a/14132912/14179793 and I can get it to work using solution 1. There is an additional variable that I need to figure out though.
This is a dummy project I am testing with:
.
├── a_package
│ ├── __init__.py
│ └── lib_a.py
├── b_package
│ ├── __init__.py
│ └── test_script.py
├── main.py
└── src
└── src_lib
└── src_lib.py
With this setup I can do:
python -m b_package.test_script
this is lib a function
This is src_lib_function.
test_script.py:
from a_package.lib_a import lib_a_function
from src.src_lib.src_lib import src_lib_function
if __name__ == '__main__':
lib_a_function()
src_lib_function()
pass
The goal is to make b_package/test_script.py executable without using python test_script ie ./test_script
However, adding the shebang at the top #!/usr/bin/env python causes an import error:
$ ./b_package/test_script.py
Traceback (most recent call last):
File "./b_package/test_script.py", line 2, in <module>
from a_package.lib_a import lib_a_function
ModuleNotFoundError: No module named 'a_package'
I assume it is because python is not loading it as a module based off the above mentioned question but I am not sure how to resolve this.
I ended up using setuptools as suggested by kosciej16 to achieve the desired results.
New project structure:
.
├── a_package
│ ├── __init__.py
│ └── lib_a.py
├── b_package
│ ├── __init__.py
│ └── test_script.py
├── main.py
├── pyproject.toml
├── setup.cfg
└── src
├── __init__.py
└── src_lib
├── __init__.py
└── src_lib.py
setup.cfg:
[metadata]
name = cli_test
version = 0.0.1
[options]
packages = find:
[options.entry_points]
console_scripts =
test_script = b_package.test_script:main
This allows the user to clone the repo and run pip install . from the top level then they can execute the script by just typing test_script
I'm trying to create a python package, and I've added some files which are needed for the module to function, like this: https://docs.python.org/2/distutils/setupscript.html#distutils-additional-files
Due to circumstances, this is the method I need to use, if at all possible. I also need to end up with a source distributable, so something that works when making other types of python distributables doesn't work for me.
My setup.py looks like this:
from setuptools import setup
setup(name='mypackage',
version='0.1',
py_modules=['mypackage'],
install_requires=['numpy'],
data_files=[('data', ['data/file0.npz', 'data/file1.npz'])]
)
The directory structure looks like this:
├── PKG-INFO
├── data
│ ├── data0.npz
│ └── data1.npz
├── dist
│ ├── mypackage-0.1.zip
├── mypackage.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── requires.txt
│ └── top_level.txt
├── mypackage.py
├── setup.cfg
└── setup.py
I'm trying to load it in like this(every function but init removed for simplicity):
import numpy as np
class MyClass ():
def __init__(self):
self.data0 = np.load("data/file0.npz")
self.data1 = np.load("data/file1.npz")
And get this error when trying to instantiate the class:
No such file or directory: 'data/file0.npz'
What do I need to change to get this working?
To load package resources, I usually use pkg_resources module
Here is an example to get resource file relative to current module:
from pkg_resources import resource_filename
def main():
print(resource_filename(__name__, 'data/test.txt'))
In your setup.py you can use package_data to include package data files.
setup(
name='',
# (...)
package_data={
'': [
'*.txt',
],
},
)
Note: To make it works, data has to be a python module.
I have the following directory structure in Ubuntu. I'm trying to import the module config from my local package my_app into my script my_app_script.py
$ tree
.
├── my_app/
│ ├── config.py
│ ├── __init__.py
│ └── test/
├── my_app-info # created by pip install -e .
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── bin/
│ └── my_app_script.py
├── LICENSE
├── README.md
└── setup.py
# setup.py
setup(
name='my_app',
version='0.1.2',
description='',
url='',
packages=['my_app'],
scripts=['bin/my_app_script.py'],
install_requires=[],
python_requires='>=3.6',
)
# my_app_script.py
from my_app import config
When I run my_app_script.py it results in "ImportError: cannot import name 'config'
What am I doing wrong?
Edit:
I am trying to follow this guide on packaging a program.
You need an __init__.py file in the parent directory as well as in bin directory.
You can use either of below approaches.
The first approach seems best to me as the script will always set the path relative to it, and will also work if you clone your repos.
Add an __init__.py in parent directory(Next to setup.py).
And add below line in my_app_script.py
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, "my_app")))
What this will do is add .../my_app to PYTHONPATH at runtime, when my_app_script.py is executed.
Add an env.sh in parent directory. Below should be the contents of env.sh.
export PYTHONPATH=$(pwd):$PYTHONPATH
Now cd int the directory where env.sh is kept and then source it
source env.sh
Now run your my_app_script.py
python bin/my_app_script.py
Set PYTHONPATH from commandline.
PYTHONPATH=$(pwd) python bin/my_app_script.py
├── ledger
│ ├── __init__.py
│ ├── ledger_data.py
│ └── ledger_model.py
├── main.py
├── sscommon
│ ├── __init__.py
│ └── logging.py
└── tests
└── test_ledger_data.py
I need to import classes from ledger_data module when running test_ledger_data.py. I currently do sys.path.append("../") in test_ledger_data.py or I have to add symbolik links to all modules being used to tests directory. Both options seem incorrect. How to do it correctly?
If I just run the file either from project root or tests directories I get error:
from ledger.ledger_data import LedgerData
ImportError: No module named 'ledger'
You can create an __init__.py file in your folder, and import the parent dir using:
parent_dir = os.path.abspath(os.path.join(os.path.abspath(__file__), os.pardir))
sys.append(parent_dir)
This uses os.path to find out the directory based on your file location.
Update: create the above __init__.py and reside it inside tests/ folder.
Then, in your test_ledge_data.py put at the head of the file from __init__ import *; this will import everything in your init file to your module namespace.
What is the right way to package Alembic migration files in a Setuptools setup.py file? Everything is in my repo root as alembic/.
This is a Python application, not a library.
My desired installation flow is that someone can pip install the wheel that is my application. They would then be able to initialize the application database by running something like <app> alembic upgrade --sqlalchemy.url=<db_url>. Upgrades would then require a pip install -U, after which they can run the Alembic command again.
Is this unorthodox?
If not, how would I accomplish this? Certainly a console_scripts entry_points. But beyond that?
I am not sure this is the right way but I did it this way:
First, you can add sort of custom options to alembic using the -x option and you can find details explained in this great answer. This allows you to specify the db_url at runtime and make it override the value in the config.ini.
Then I managed to package alembic and my migrations by moving the alembic.ini file and the alembic directory from my project root to my top-level python package:
<project root>
├── src
│ └── <top-level package dir>
│ ├── alembic
│ │ ├── env.py
│ │ ├── README
│ │ ├── script.py.mako
│ │ └── versions
│ │ ├── 58c8dcd5fbdc_revision_1.py
│ │ └── ec385b47da23_revision_2.py
│ ├── alembic.ini
│ ├── __init__.py
│ └── <other files and dirs>
└── <other files and dirs>
This allows to use the setuptools package_data directive inside my setup.py:
setup(
name=<package_name>,
package_dir={'': 'src'},
packages=find_packages(where='src'),
package_data={
'<top-level package dir>': ['alembic.ini', 'alembic/*', 'alembic/**/*'],
},
[...]
)
A this point, the alembic config and revisions are correctly packaged but the alembic.ini settings have to be tweaked to reflect the new directory tree. It can be done using the %(here)s param which contains the absolute path of the directory containing the alembic.ini file:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = %(here)s/alembic
[...]
# version location specification; this defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
version_locations = %(here)s/alembic/versions
[...]
Finally, you have to call alembic with the -c option which allows to provide the path of the config file:
alembic -c <path to alembic.ini> ...
One way to do this which keeps the main alembic folder along the main package folder is to treat the alembic folder as it's own package to be installed along side your main package.
To do this you must rename it (it can't be called alembic, as it will be a top level package, so needs a unique name - I've used migrations), and add a __init__.py file in the alembic folder and the versions folder.
Running the migrations on deployment requires knowing the path to the installed package - a simple way to do this is to provide a console scripts that applies the migrations.
So the project structure looks like this:
<project root>
├── setup.py
├── mypackage
│ └── <project source files...>
│
├── migrations
│ ├── __init__.py
│ ├── alembic.ini
│ ├── apply.py
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ ├── __init__.py
│ ├── 58c8dcd5fbdc_revision_1.py
│ └── ec385b47da23_revision_2.py
│
└── <other files and dirs>
And setup.py:
from setuptools import find_packages
from setuptools import setup
setup(
name='mypackage',
packages=find_packages(exclude=('tests',)),
package_data={'migrations': ['alembic.ini']},
entry_points={
'console_scripts': ['apply-migrations=migrations.apply:main'],
},
install_requires=[
"SQLAlchemy==1.3.0",
"alembic==1.0.10",
# ...
]
)
And finally migrations/apply.py:
# Python script that will apply the migrations up to head
import alembic.config
import os
here = os.path.dirname(os.path.abspath(__file__))
alembic_args = [
'-c', os.path.join(here, 'alembic.ini'),
'upgrade', 'head'
]
def main():
alembic.config.main(argv=alembic_args)
Now after installing your wheel, you will have a command apply-migrations which you can invoke directly. Note the version I've implemented here doesn't have any arguments - though if you wanted to pass eg. --sqlalchemy.url you could add it in alembic_args.
Personally I prefer to set the url in migrations/env.py. For example if you had an environment variable called SQLACLHEMYURL you could add this in migrations/env.py:
import os
config.set_main_options(os.getenv('SQLALCHEMYURL'))
Then you can invoke:
SQLALCHEMYURL=... apply-migrations
On deploment.