I want to install a package with the following structure:
├── nsp
│ ├── __init__.pxd
│ └── A
| ├── b.py
| ├── extension.pxd
| ├── extension.pyx
│ └── __init__.py
└── setup.py
whereby nsp is a namespace package. (Motivation: see here)
My setup script reads as follows:
from setuptools import setup
from setuptools.extension import Extension
# factory function
def my_build_ext(pars):
# import delayed:
from setuptools.command.build_ext import build_ext as _build_ext
# include_dirs adjusted:
class build_ext(_build_ext):
def finalize_options(self):
_build_ext.finalize_options(self)
# Prevent numpy from thinking it is still in its setup process:
__builtins__.__NUMPY_SETUP__ = False
import numpy
self.include_dirs.append(numpy.get_include())
#object returned:
return build_ext(pars)
extensions = [Extension(nsp.A.extension, ['nsp/A/extension.cpp'])]
setup(
cmdclass={'build_ext' : my_build_ext},
setup_requires=['numpy'],
install_requires=['numpy'],
packages=['nsp.A'],
ext_modules=extensions
package_data={
'': ['*.pxd', '*.pyx']
},
)
According to my current understanding, package_data describes all non-python files that shall be included. I desire to rebuild the exact file tree given above in the installation directory. Unfortunately, however, __init__.pxd is not copied. How could I achieve that?
I am working with Python 3.7 on Windows 10 x64.
Related
I'm developing a Python package that includes an extension module:
# setup.py
from distutils.core import setup, Extension
setup(
name="myPythonPkg",
# ... all other args
packages=["myPythonPkg"],
ext_modules=[
Extension('myFastCfunctions', ['myFastCfunctions.c'])
]
)
When I test the installation of this package with python setup.py install --prefix=$PWD/prefix I see (roughly):
<prefix>
└── lib
└── python3.10
└── site-packages
├── myFastCfunctions.cpython-310-x86_64-linux-gnu.so
├── myPythonPkg
│ ├── __init__.py
└── myPythonPkg-1.0.2-py3.10.egg-info
Inside myPythonPkg/__init__.py I'd like to get the path of myFastCfunctions.cpython-310-x86_64-linux-gnu.so and load it via ctypes.cdll.LoadLibrary. Of course I can paste that path directly, but I was wondering if there is a smarter, more platform and version agnostic way to doing that.
Use EXTENSION_SUFFIXES[0], example:
from importlib.machinery import EXTENSION_SUFFIXES
myFastCfunctions_file = path.join(
path.join(
path.dirname(__file__), '..',
'myFastCfunctions{}'.format(
EXTENSION_SUFFIXES[0]
)
)
)
myFastCfunctions = cdll.LoadLibrary(myFastCfunctions_file)
Link to docs.
In my Python project with the directory layout
.
├── justfile
├── pyproject.toml
├── README.md
├── setup.cfg
└── src
└── foobar
├── __about__.py
├── __init__.py
└── main.py
__about__.py reads
__version__ = "1.0.0"
I would like to use this version info in setup.cfg. I tried
[metadata]
name = foobar
version = attr: foobar.__about__.__version__
[options]
package_dir =
=src
packages = find:
install_requires =
rich
python_requires = >=3.6
[options.packages.find]
where=src
but this still tries to import foobar, resulting in ModuleNotFoundErrors from foobars dependencies when trying to evaluate the version string.
The __init__.py reads
from .main import solve
from .__about__ import __version__
I had been under the impression that after this PR has been merged, just the AST of the attr was evaluated. (See also this question.)
Any idea what might be wrong?
src is not part of the package import path (you wouldn't do import src.foobar with your installed package), it's just a directory.
Try
version = attr:foobar.__about__.__version__
instead (assuming you've set up the src/ layout in setup.cfg).
I found the issue. In __init__.py, the __version__ must be imported before all external dependencies. This
from .__about__ import __version__
from .main import solve
works.
I am trying to build a pip package from source code in a Git repository that has multiple packages that share a common package. (I am not allowed to change the structure of this Git repository.)
The structure is:
├── common
│ ├── __init__.py
│ ├── run_helpers
│ │ ├── __init__.py
│ │ ├── aws.py
│ │ └── s3.py
└── components
└── redshift_unload
├── redshift_unload
│ ├── __init__.py
│ └── run.py
└── setup.py
My setup.py is as follows:
from setuptools import setup, find_packages
setup(
...
packages=find_packages(),
package_dir={"": "."},
entry_points={
"console_scripts": ["redshift_unload=redshift_unload.run:main"]
}
)
Looking at other answers here, things I have tried so far include:
Specifying the actual package names in the packages= line instead of using find_packages().
Passing where="../../" to find_packages()
Using find_packages() + find_packages(where="../../") in the packages=` line.
Everything I can think of in the packages_dir line.
When I run pip install . I get, the package installs fine, but then when I run the installed python script I get:
# redshift_unload
Traceback (most recent call last):
File "/usr/local/bin/redshift_unload", line 5, in <module>
from redshift_unload.run import main
File "/usr/local/lib/python3.8/site-packages/redshift_unload/run.py", line 9, in <module>
from common._run_helpers.aws import get_boto3_session
ModuleNotFoundError: No module named 'common'
What did work:
If I moved the common directory to components/redshift_unload, then it works fine. But I can't do this. I also tried placing a symlink there in its place, but seems like that doesn't work either.
Is there a way to make this work?
I believe I have found the best solution to this.
Based on this comment here, I concluded that what I am trying to do is not intended or supported.
However, I found a workaround as follows works fine:
from pathlib import Path
from shutil import rmtree, copytree
from setuptools import setup, find_packages
src_path = Path(os.environ["PWD"], "../../common")
dst_path = Path("./common")
copytree(src_path, dst_path)
setup(
...
packages=find_packages(),
package_dir={"": "."},
entry_points={
"console_scripts": ["redshift_unload=redshift_unload.run:main"]
}
)
rmtree(dst_path)
The key insight here is that, while packaging occurs in a temporary directory, the value of os.environ["PWD"] is available to the process, such that the common directory can be copied temporarily and then cleaned up again (using shutil functions copytree and rmtree) into a location that will be found by find_packages() before the setup() function is called.
I am trying to create a setup.py for an existing project. The project has a directory structure that I cannot change. I need my setup.py to be in the same folder as my project source files.
Sample 1, directory structure.
MyModule
├── __init__.py
├── MyApp.ini
├── MyApp.py
├── setup.py
└── foo.py
This is my stetup.py
from setuptools import setup, find_packages
packages = find_packages(exclude=['ez_setup', 'tests', 'tests.*'])
console_script = list()
console_script.append('MyApp = MyApp:main')
py_modules = list()
py_modules.append('MyApp')
other_files = list()
other_files.append('MyApp.ini')
module_name = "MyModule"
mysetup = setup(name=module_name,
py_modules=py_modules,
version="1.0.0",
packages=packages,
package_dir={module_name: module_name},
package_data={module_name: other_files},
include_package_data=True,
entry_points={'console_scripts': console_script, },
zip_safe=False,
python_requires='>=2.7,<=3.0',
)
After installing MyModule via 'python setup install'. I cannot import from MyModule. 'from MyModule import MyApp' does not work. I can import directly. 'import MyApp' works. The problems is 'import foo' works as well. I have multiple projects with different foo.py.
Sample 2:
If I could change the directory structure as shown below. The install works correctly.
MyModule
├── MyModule
│ ├── foo.py
│ ├── __init__.py
│ ├── MyApp.ini
│ └── MyApp.py
└── setup.py
Is there a way to get sample 1, to install the way sample 2 does?
I was able to answer my own question. It can be done by setting package_dir up one level as shown below. I had to use data_files rather than package_data to add my support files.
Limitation: The setup script, setup.py, is installed as part of the egg. I tried to excluding it, but it gets installed anyway.
from setuptools import setup, find_packages
packages = find_packages(exclude=['ez_setup', 'tests', 'tests.*'])
console_script = list()
console_script.append('MyApp = MyModule.MyApp:main')
packages.append("MyModule")
setup(name="MyModule",
version="1.0.0",
packages=packages,
package_dir={"MyModule": "../MyModule"},
data_files=[('MyModule', ['MyApp.ini'])],
include_package_data=True,
entry_points={'console_scripts': console_script, },
zip_safe=False,
python_requires='>=2.7,<=3.0',
)
I am trying to include a python file in the build/lib directory created when running
python setup.py install
In particular, I would like to include a simple configuration file ('definitions.py') that defines a ROOT_DIR variable, which is then used by subpackages. The 'definitions.py' file contains:
import os
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
My goal is to have configuration files within each subpackage ('config.py') call ROOT_DIR to build their own absolute paths:
from definitions import ROOT_DIR
PACKAGE_DIR = os.path.join(ROOT_DIR, 'package1/')
The idea is drawn from this stackoverflow answer: https://stackoverflow.com/a/25389715.
However, this 'definitions.py' file never shows up in the build directory when running 'setup.py install'.
Here is the directory structure of the project:
project
|
├── setup.py
|
├── definitions.py
|
├── package1
| ├── __init__.py
| ├── config.py
| └── ...
|
├── package2
| ├── __init__.py
| └── ...
└── ...
My multiple attempts have failed (trying, e.g. the suggestions offered in https://stackoverflow.com/a/11848281). As far as I can tell, it's because definitions.py is in the top-level of my project structure (which lacks an __init__.py file).
I have tried:
1) ...using the 'package-data' variable in setuptools.setup()
package_data={'package': ['./definitions.py']}
but definitions.py does not show up in the build (I think because definitions.py is not in a 'package' that has an __init__.py?).
2) ...using a MANIFEST.in file, but this also does not work(I think because MANIFEST does not work with .py files?)
My question:
Is there a way to include definitions.py in the build directory? Or, is there a better way to provide access to absolute paths built from the top-level directory for multiple sub-packages?
If you are looking for a way to access a non-python data file in the installed module like in the question you've linked (a configuration file in the top-level package that should be accessible in subpackages), use pkg_resources machinery instead of inventing a custom path resolution. An example project structure:
project
├── setup.py
└── root
├── __init__.py
├── config.txt
├── sub1
│ └── __init__.py
└── sub2
└── __init__.py
setup.py:
from setuptools import setup
setup(
name='myproj',
...,
packages=['root', 'root.sub1', 'root.sub2'], # or setuptools.find_packages()
package_data={'root': ['config.txt']}
)
Update:
As pointed out by wim in the comments, there's now a backport for importlib.resources (which is only available in Python 3.7 and onwards) - importlib_resources, which offers a modern resource machinery that utilizes pathlib:
# access the filepath
importlib_resources.path('root', 'config.txt')
# access the contents as string
importlib_resources.read_text('root', 'config.txt')
# access the contents as file-like object
importlib_resources.open_binary('root', 'config.txt')
Original answer
Using pkg_resources, you can access the root/config.txt from any spot of your package without having to perform any path resolution at all:
import pkg_resources
# access the filepath:
filepath = pkg_resources.resource_filename('root', 'config.txt')
# access the contents as string:
contents = pkg_resources.resource_string('root', 'config.txt')
# access the contents as file-like object:
contents = pkg_resources.resource_stream('root', 'config.txt')
etc.