Develop Reference

Python: Best way to get shared object library path from within a pypi package

I'm developing a Python package that includes an extension module:
# setup.py
from distutils.core import setup, Extension
setup(
name="myPythonPkg",
# ... all other args
packages=["myPythonPkg"],
ext_modules=[
Extension('myFastCfunctions', ['myFastCfunctions.c'])
]
)
When I test the installation of this package with python setup.py install --prefix=$PWD/prefix I see (roughly):
<prefix>
└── lib
└── python3.10
└── site-packages
├── myFastCfunctions.cpython-310-x86_64-linux-gnu.so
├── myPythonPkg
│   ├── __init__.py
└── myPythonPkg-1.0.2-py3.10.egg-info
Inside myPythonPkg/__init__.py I'd like to get the path of myFastCfunctions.cpython-310-x86_64-linux-gnu.so and load it via ctypes.cdll.LoadLibrary. Of course I can paste that path directly, but I was wondering if there is a smarter, more platform and version agnostic way to doing that.

Use EXTENSION_SUFFIXES[0], example:
from importlib.machinery import EXTENSION_SUFFIXES
myFastCfunctions_file = path.join(
path.join(
path.dirname(__file__), '..',
'myFastCfunctions{}'.format(
EXTENSION_SUFFIXES[0]
)
)
)
myFastCfunctions = cdll.LoadLibrary(myFastCfunctions_file)
Link to docs.

including python file from project root in setup.py build

I am trying to include a python file in the build/lib directory created when running
python setup.py install
In particular, I would like to include a simple configuration file ('definitions.py') that defines a ROOT_DIR variable, which is then used by subpackages. The 'definitions.py' file contains:
import os
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
My goal is to have configuration files within each subpackage ('config.py') call ROOT_DIR to build their own absolute paths:
from definitions import ROOT_DIR
PACKAGE_DIR = os.path.join(ROOT_DIR, 'package1/')
The idea is drawn from this stackoverflow answer: https://stackoverflow.com/a/25389715.
However, this 'definitions.py' file never shows up in the build directory when running 'setup.py install'.
Here is the directory structure of the project:
project
|
├── setup.py
|
├── definitions.py
|
├── package1
| ├── __init__.py
| ├── config.py
| └── ...
|
├── package2
| ├── __init__.py
| └── ...
└── ...
My multiple attempts have failed (trying, e.g. the suggestions offered in https://stackoverflow.com/a/11848281). As far as I can tell, it's because definitions.py is in the top-level of my project structure (which lacks an __init__.py file).
I have tried:
1) ...using the 'package-data' variable in setuptools.setup()
package_data={'package': ['./definitions.py']}
but definitions.py does not show up in the build (I think because definitions.py is not in a 'package' that has an __init__.py?).
2) ...using a MANIFEST.in file, but this also does not work(I think because MANIFEST does not work with .py files?)
My question:
Is there a way to include definitions.py in the build directory? Or, is there a better way to provide access to absolute paths built from the top-level directory for multiple sub-packages?

If you are looking for a way to access a non-python data file in the installed module like in the question you've linked (a configuration file in the top-level package that should be accessible in subpackages), use pkg_resources machinery instead of inventing a custom path resolution. An example project structure:
project
├── setup.py
└── root
├── __init__.py
├── config.txt
├── sub1
│ └── __init__.py
└── sub2
└── __init__.py
setup.py:
from setuptools import setup
setup(
name='myproj',
...,
packages=['root', 'root.sub1', 'root.sub2'], # or setuptools.find_packages()
package_data={'root': ['config.txt']}
)
Update:
As pointed out by wim in the comments, there's now a backport for importlib.resources (which is only available in Python 3.7 and onwards) - importlib_resources, which offers a modern resource machinery that utilizes pathlib:
# access the filepath
importlib_resources.path('root', 'config.txt')
# access the contents as string
importlib_resources.read_text('root', 'config.txt')
# access the contents as file-like object
importlib_resources.open_binary('root', 'config.txt')
Original answer
Using pkg_resources, you can access the root/config.txt from any spot of your package without having to perform any path resolution at all:
import pkg_resources
# access the filepath:
filepath = pkg_resources.resource_filename('root', 'config.txt')
# access the contents as string:
contents = pkg_resources.resource_string('root', 'config.txt')
# access the contents as file-like object:
contents = pkg_resources.resource_stream('root', 'config.txt')
etc.

setup.py sdist exclude packages in subdirectory

I have the following project structure I would like to package:
├── doc
│   └── source
├── src
│   ├── core
│   │   ├── config
│   │   │   └── log.tmpl
│   │   └── job
│   ├── scripts
│   └── test
└── tools
I would like to package core under src but exclude test. Here is what I tried unsuccessfully:
setup(name='core',
version=version,
package_dir = {'': 'src'}, # Our packages live under src but src is not a package itself
packages = find_packages("src", exclude=["test"]), # I also tried exclude=["src/test"]
install_requires=['xmltodict==0.9.0',
'pymongo==2.7.2',
'ftputil==3.1',
'psutil==2.1.1',
'suds==0.4',
],
include_package_data=True,
)
I know I can exclude test using the MANIFEST.in file, but I would be happy if you could show me how to do this with setup and find_packages.
Update:
After some more playing around, I realized that building the package with python setup.py install does what I expected (that is, it excludes test). However, issuing python setup.py sdist causes everything to be included (that is, it ignores my exclude directive). I don't know whether it is a bug or a feature, but there is still the possibility of excluding files in sdist using MANIFEST.in.

find_packages("src", exclude=["test"]) works.
The trick is to remove stale files such as core.egg-info directory. In your case you need to remove src/core.egg-info.
Here's setup.py I've used:
from setuptools import setup, find_packages
setup(name='core',
version='0.1',
package_dir={'':'src'},
packages=find_packages("src", exclude=["test"]), # <- test is excluded
####packages=find_packages("src"), # <- test is included
author='J.R. Hacker',
author_email='jr#example.com',
url='http://stackoverflow.com/q/26545668/4279',
package_data={'core': ['config/*.tmpl']},
)
To create distributives, run:
$ python setup.py sdist bdist bdist_wheel
To enable the latter command, run: pip install wheel.
I've inspected created files. They do not contain test but contain core/__init__.py, core/config/log.tmpl files.

In your MANIFEST.in at project root, add
prune src/test/
then build package with python setup.py sdist

I probably just use wild cards as defined in the find_packages documentation. *test* or *tests* is something I tend to use as we save only test filenames with the word test. Simple and easy ^-^.
setup(name='core',
version=version,
package_dir = {'': 'src'}, # Our packages live under src but src is not a package itself
packages = find_packages("src", exclude=['*tests*']), # I just use wild card. Works perfect ^-^
install_requires=['xmltodict==0.9.0',
'pymongo==2.7.2',
'ftputil==3.1',
'psutil==2.1.1',
'suds==0.4',
],
include_package_data=True,
)
FYI:
I would also recommend adding following into .gitignore.
build
dist
pybueno.egg-info
And move build and pushing package to pypi or your private repository bit into CI/CD to make whole setup look clean and neat.

Assuming that your folder is called tests and not test, it should work with the following code:
setup(name='core',
version=version,
package_dir = {'': 'src'}, # Our packages live under src but src is not a package itself
packages = find_packages('src', exclude=['tests'])
install_requires=['xmltodict==0.9.0',
'pymongo==2.7.2',
'ftputil==3.1',
'psutil==2.1.1',
'suds==0.4',
],
include_package_data=True,
)

Confused about the package_dir and packages settings in setup.py

Here is my project directory structure, which includes the project folder, plus
a "framework" folder containing packages and modules shared amongst several projects
which resides at the same level in the hierarchy as the project folders:
Framework/
package1/
__init__.py
mod1.py
mod2.py
package2/
__init__.py
moda.py
modb.py
My_Project/
src/
main_package/
__init__.py
main_module.py
setup.py
README.txt
Here is a partial listing of the contents of my setup.py file:
from distutils.core import setup
setup(packages=[
'package1',
'package2.moda',
'main_package'
],
package_dir={
'package1': '../Framework/package1',
'package2.moda': '../Framework/package2',
'main_package': 'src/main_package'
})
Here are the issues:
No dist or build directories are created
Manifest file is created, but all modules in package2 are listed, not just the moda.py module
The build terminates with an error:
README.txt: Incorrect function
I don't know if I have a single issue (possibly related to my directory structure) or if I have multiple issues but I've read everything I can find on distribution of Python applications, and I'm stumped.

If I understand correctly, the paths in package_dir should stop at the parent directory of the directories which are Python packages. In other words, try this:
package_dir={'package1': '../Framework',
'package2': '../Framework',
'main_package': 'src'})

I've had a similar problem, which was solved through the specification of the root folder and of the packages inside that root.
My package has the following structure:
.
├── LICENSE
├── README.md
├── setup.py
└── src
└── common
├── __init__.py
├── persistence.py
├── schemas.py
└── utils.py
The setup.py contains the package_dir and packages line:
package_dir={"myutils": "src"},
packages=['myutils.common'],
After running the python setup.py bdist_wheel and installing the .whl file, the package can be called using:
import myutils.common

MANIFEST.in, package_data, and data_files clarification?

I am trying to create a Python package, and I have a directory structure like this:
mypkg/
├── __init__.py
├── module1
│   ├── x.py
│   ├── y.py
│   └── z.txt
└── module2
├── a.py
└── b.py
Then I added all the files in MANIFEST.in and when I check the created archive, it had all the files.
When I do python setup.py install in the dist-packages/mypkg/module1. I see only the Python files and not z.txt.
I have z.txt in both MANIFEST.in and setup.py:
setup (
packages = [
'mypkg',
'mypkg.module1',
'mypkg.module2',
],
package_data = {
'mypkg': ['module1/z.txt']
},
include_package_data = True,
...
)
I tried adding the file as data_files as well but that created a directory in /usr/local. I want to keep it inside the source code directory as the code uses that data.
I have read the posts listed below but I keep getting confused about what is the right way to keep z.txt in the right location after setup.py install.
MANIFEST.in ignored on "python setup.py install" - no data files installed?
Installing data files into site-packages with setup.py
http://blog.codekills.net/2011/07/15/lies,-more-lies-and-python-packaging-documentation-on--package_data-/

Try using setuptools instead of distutils.

Update: It got fixed when I started using setuptools instead of distutils.core. I think it was some problem with distutils not agreeing with manifest while setuptools worked without any changes in the code. I recommend using setuptools in the future. Using the link here : setup tools- developers guide