Develop Reference

Python: Best way to get shared object library path from within a pypi package

I'm developing a Python package that includes an extension module:
# setup.py
from distutils.core import setup, Extension
setup(
name="myPythonPkg",
# ... all other args
packages=["myPythonPkg"],
ext_modules=[
Extension('myFastCfunctions', ['myFastCfunctions.c'])
]
)
When I test the installation of this package with python setup.py install --prefix=$PWD/prefix I see (roughly):
<prefix>
└── lib
└── python3.10
└── site-packages
├── myFastCfunctions.cpython-310-x86_64-linux-gnu.so
├── myPythonPkg
│   ├── __init__.py
└── myPythonPkg-1.0.2-py3.10.egg-info
Inside myPythonPkg/__init__.py I'd like to get the path of myFastCfunctions.cpython-310-x86_64-linux-gnu.so and load it via ctypes.cdll.LoadLibrary. Of course I can paste that path directly, but I was wondering if there is a smarter, more platform and version agnostic way to doing that.

Use EXTENSION_SUFFIXES[0], example:
from importlib.machinery import EXTENSION_SUFFIXES
myFastCfunctions_file = path.join(
path.join(
path.dirname(__file__), '..',
'myFastCfunctions{}'.format(
EXTENSION_SUFFIXES[0]
)
)
)
myFastCfunctions = cdll.LoadLibrary(myFastCfunctions_file)
Link to docs.

How to access text file in project root from python package under `src/` directory

I want my package's version number to live in a single place where everything that needs it can refer to it.
I found several suggestions in this Python guide to Single Sourcing the Package Version and decided to try #4, storing it in a simple text file in my project root named VERSION.
Here's a shortened version of my project's directory tree (you can see the the full project on GitHub):
.
├── MANIFEST.in
├── README.md
├── setup.py
├── VERSION
├── src/
│   └── fluidspaces/
│   ├── __init__.py
│   ├── __main__.py
│   ├── i3_commands.py
│   ├── rofi_commands.py
│   ├── workspace.py
│   └── workspaces.py
└── tests/
   ├── test_workspace.py
   └── test_workspaces.py
Since VERSION and setup.py are siblings, it's very easy to read the version file inside the setup script and do whatever I want with it.
But VERSION and src/fluidspaces/__main__.py aren't siblings and the main module doesn't know the project root's path, so I can't use this approach.
The guide had this reminder:
Warning: With this approach you must make sure that the VERSION file is included in all your source and binary distributions (e.g. add include VERSION to your MANIFEST.in).
That seemed reasonable - instead of package modules needing the project root path, the version file could be copied into the package at build time for easy access - but I added that line to the manifest and the version file still doesn't seem to be showing up in the build anywhere.
To build, I'm running pip install -U . from the project root and inside a virtualenv. Here are the folders that get created in <virtualenv>/lib/python3.6/site-packages as a result:
fluidspaces/
├── i3_commands.py
├── __init__.py
├── __main__.py
├── __pycache__/ # contents snipped
├── rofi_commands.py
├── workspace.py
└── workspaces.py
fluidspaces-0.1.0-py3.6.egg-info/
├── dependency_links.txt
├── entry_points.txt
├── installed-files.txt
├── PKG-INFO
├── SOURCES.txt
└── top_level.txt
More of my configuration files:
MANIFEST.in:
include README.md
include VERSION
graft src
prune tests
setup.py:
#!/usr/bin/env python3
from setuptools import setup, find_packages
def readme():
'''Get long description from readme file'''
with open('README.md') as f:
return f.read()
def version():
'''Get version from version file'''
with open('VERSION') as f:
return f.read().strip()
setup(
name='fluidspaces',
version=version(),
description='Navigate i3wm named containers',
long_description=readme(),
author='Peter Henry',
author_email='me#peterhenry.net',
url='https://github.com/mosbasik/fluidspaces',
license='MIT',
classifiers=[
'Development Status :: 3 - Alpha',
'Programming Language :: Python :: 3.6',
],
packages=find_packages('src'),
include_package_data=True,
package_dir={'': 'src'},
package_data={'': ['VERSION']},
setup_requires=[
'pytest-runner',
],
tests_require=[
'pytest',
],
entry_points={
'console_scripts': [
'fluidspaces = fluidspaces.__main__:main',
],
},
python_requires='~=3.6',
)
I found this SO question Any python function to get “data_files” root directory? that makes me think the pkg_resources library is the answer to my problems, but I've not been able to figure out how to use it in my situation.
I've been having trouble because most examples I've found have python packages directly in the project root instead of isolated in a src/ directory. I'm using a src/ directory because of recommendations like these:
PyTest: Good Practices: Tests Outside Application Code
Ionel Cristian Mărieș - Packaging a Python Library
Hynek Schlawack - Testing and Packaging
Other knobs I've found and tried twisting a little are the package_data, include_package_data, and data_files kwargs for setup(). Don't know how relevent they are. Seems like there's some interplay between things declared with these and things declared in the manifest, but I'm not sure about the details.

Chatted with some people in the #python IRC channel on Freenode about this issue. I learned:
pkg_resources was probably how I should to do what I was asking for, but it would require putting the version file in the package directory instead of the project root.
In setup.py I could read in such a version file from the package directory without importing the package itself (a no-no for a few reasons) but it would require hard-coding the path from the root to the package, which I wanted to avoid.
Eventually I decided to use the setuptools_scm package to get version information from my git tags instead of from a file in my repo (someone else was doing that with their package and their arguments were convincing).
As a result, I got my version number in setup.py very easily:
setup.py:
from setuptools import setup, find_packages
def readme():
'''Get long description from readme file'''
with open('README.md') as f:
return f.read()
setup(
name='fluidspaces',
use_scm_version=True, # use this instead of version
description='Navigate i3wm named containers',
long_description=readme(),
author='Peter Henry',
author_email='me#peterhenry.net',
url='https://github.com/mosbasik/fluidspaces',
license='MIT',
classifiers=[
'Development Status :: 3 - Alpha',
'Programming Language :: Python :: 3.6',
],
packages=find_packages('src'),
package_dir={'': 'src'},
setup_requires=[
'pytest-runner',
'setuptools_scm', # require package for setup
],
tests_require=[
'pytest',
],
entry_points={
'console_scripts': [
'fluidspaces = fluidspaces.__main__:main',
],
},
python_requires='~=3.6',
)
but I ended up having to have a hard-coded path indicating what the project root should be with respect to the package code, which is kind of what I had been avoiding before. I think this issue on the setuptools_scm GitHub repo might be why this is is necessary.
src/fluidspaces/__main__.py:
import argparse
from setuptools_scm import get_version # import this function
def main(args=None):
# set up command line argument parsing
parser = argparse.ArgumentParser()
parser.add_argument('-V', '--version',
action='version',
version=get_version(root='../..', relative_to=__file__)) # and call it here

For folks still looking for the answer to this, below is my attempt at following variety #4 of the guide to Single Sourcing the Package Version. It's worth noting WHY you might choose this solutions when there are other simpler ones. As the link notes, this approach is useful when you have external tools that might also want to easily check the version (e.g. CI/CD tools).
File tree
myproj
├── MANIFEST.in
├── myproj
│   ├── VERSION
│   └── __init__.py
└── setup.py
myproj/VERSION
1.4.2
MANIFEST.in
include myproj/VERSION
setup.py
with open('myproj/VERSION') as version_file:
version = version_file.read().strip()
setup(
...
version=version,
...
include_package_data=True, # needed for the VERSION file
...
)
myproj/__init__.py
import pkgutil
__name__ = 'myproj'
__version__ = pkgutil.get_data(__name__, 'VERSION').decode()
It's worth noting that setting configuration in setup.cfg is a nice, clean alternative to including everything in the setup.py setup function. Instead of reading version in setup.py, and then including in the function, you could do the following:
setup.cfg
[metadata]
name = my_package
version = attr: myproj.VERSION
In the full example I chose to leave everything in setup.py for the ease of one less file and uncertainty about whether or not potential whitespace around the version in the VERSION file would be stripped by the cfg solution.

Python packaging: exclude directory from bdist_wheel

I have the following project structure:
.
├── docs
├── examples
├── MANIFEST.in
├── README.rst
├── setup.cfg
├── setup.py
└── myproject
I want to bundle my project into a wheel. For this, I use the following setup.py:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='myproject',
version='1.0',
description='Great project'
long_description=open('README.rst').read(),
author='Myself'
packages=find_packages(exclude=['tests','test','examples'])
)
When running python setup.py bdist_wheel, the examples directory is included in the wheel. How do I prevent this?
According to
Excluding a top-level directory from a setuptools package
I would expect that examples is excluded.

I solved the issue by using a suffixed star, examples*, i.e.:
find_packages(exclude=['*tests','examples*'])
(Note that I am writing '*tests' with a leading star,because I have test packages within each code package, as in myproject.mypackage.tests. Somehow the suffixed star seems to not be necessary if there is already a prefixed one)

Including and accessing additional files in python packages

I know roughly similar questions have already been asked, but I can't seem to find the solution to my particular problem (or my error!).
I am building a small Python package for myself, so I can use several functions without caring about folders and paths. For some of these functions (e.g., for interpolation), I need additional files which should also be copied when installing the package. I can't get this to work no matter what I try. I am also puzzled about how to add these files without explicitly specifying their paths once installed.
Here is the structure of my package
my_package
├── setup.py
├── README.rst
├── MANIFEST.in
├── my_package
│ ├── __init__.py
│ └── some_stuff.py
├── tables
│ ├── my_table.txt
my_Table.txt is the additional file that I need to install, so I have set my MANIFEST.in to
include README.rst
recursive-include tables *
And my setup.py looks like this (including the include_package_data=True statement)
from setuptools import setup
setup(name='my_package',
version='0.1',
description='Something',
url='http://something.com',
author='me',
author_email='an_email',
license='MIT',
packages=['my_package'],
include_package_data=True,
zip_safe=False)
However, after running python setup.py install, I can't find my_table.txt anywhere. What am I doing wrong? Where/how are these files copied? And after installing the package, how would you get the path of my_table.txt without explicitly writing it?
Thanks a lot!

I took the time to try your code/structure.
As it is, with packages=['my_package'], it only install the content of "my_package" (the subfolder).
You could use "find_packages" in your setup.py, I made it works with your structure.
from setuptools import setup, find_packages
setup(name='my_package',
version='0.1',
description='Something',
url='http://something.com',
author='me',
author_email='an_email',
license='MIT',
packages=find_packages(),
include_package_data=True,
zip_safe=False)
You can read more on "find_packages" here:
https://pythonhosted.org/setuptools/setuptools.html#using-find-packages
Hope this help.

MANIFEST.in, package_data, and data_files clarification?

I am trying to create a Python package, and I have a directory structure like this:
mypkg/
├── __init__.py
├── module1
│   ├── x.py
│   ├── y.py
│   └── z.txt
└── module2
├── a.py
└── b.py
Then I added all the files in MANIFEST.in and when I check the created archive, it had all the files.
When I do python setup.py install in the dist-packages/mypkg/module1. I see only the Python files and not z.txt.
I have z.txt in both MANIFEST.in and setup.py:
setup (
packages = [
'mypkg',
'mypkg.module1',
'mypkg.module2',
],
package_data = {
'mypkg': ['module1/z.txt']
},
include_package_data = True,
...
)
I tried adding the file as data_files as well but that created a directory in /usr/local. I want to keep it inside the source code directory as the code uses that data.
I have read the posts listed below but I keep getting confused about what is the right way to keep z.txt in the right location after setup.py install.
MANIFEST.in ignored on "python setup.py install" - no data files installed?
Installing data files into site-packages with setup.py
http://blog.codekills.net/2011/07/15/lies,-more-lies-and-python-packaging-documentation-on--package_data-/

Try using setuptools instead of distutils.

Update: It got fixed when I started using setuptools instead of distutils.core. I think it was some problem with distutils not agreeing with manifest while setuptools worked without any changes in the code. I recommend using setuptools in the future. Using the link here : setup tools- developers guide