I am packaging a personal Python project and uploading it to pypi. It has the following structure:
root
├── package
│ ├── __init__.py
│ ├── foo.py
│ └── subpackage
│ ├── bar.py
│ └── __init__.py
├── setup.py
└── test.py
Now, foo.py has a function called public_function that depends on bar.py:bar_function , so I am structuring my code as:
# package/foo.py
from package.subpackage import bar_function
def public_function():
return bar_function()
# package/subpackage/__init__.py
from .bar import bar_function
My goal is to let people call public_function while hiding all its dependencies like this:
import package
package.bar_function()
So package/__init__.py looks like:
from .bar import bar_function
This is working correctly when importing from test.py, but when building using setuptools, things start getting weird.
After checking these similar questions (one, two and three) these are the two approaches I have tried, but don't work as I want:
Approach 1
As I don't want to expose subpackage, I tried:
# package/setup.py
from setuptools import setup
setup(
name='package',
packages=['package']
)
But when I do import package (after uploading to pypi and running pip install package), I get the following error:
File /venv/lib/python3.8/site-packages/package/foo.py:1
----> 1 from package.subpackage import bar_function
ImportError: cannot import name 'bar_function' from 'package.subpackage' (unknown location)
(I also tried with a relative import from .subpackage ... but didn't work)
Approach 2
As it looked like the subpackages are not making it to the build, I tried:
# package/setup.py
from setuptools import setup, find_packages
setup(
name='package',
packages=find_packages()
)
Which doesn't yield the error anymore, but now I end up exposing everything else (i.e I can do import foo and import subpackage) which I don't want.
What should be the right way to set up my package? Is it even possible to achieve what I want in Python?
Related
I'm trying to create a single module package with data. My package depends on the Polars DataFrame library and I want to include datasets in the package that will be available straight upon importing as a Polars DataFrame. I want to make the package available on pipi. So far no solution I have tried using pkg_resources or os seems to work. The directory structure is
package_name
├── src
│ ├── __init__.py
│ ├── package_name.py
│ └── data
│ └── file1.csv
│ └── file2.csv
└── setup.py
The setup.py file looks like this
from setuptools import setup, find_packages
setup(
name='package_name',
version='0.1.0',
description='My Package',
author="Sebastian",
package_dir={"": "src"},
py_modules=["package_name"],
python_requires=">=3.1",
install_requires=['polars', 'datetime'],
license='GPL-3',
include_package_data=True,
package_data={'': ['data/*.csv']},
)
Now package_name.py starts like this:
import polars as _pl
from datetime import datetime as _datetime, timedelta as _timedelta
file1 = _pl.read_csv("data/file1.csv")
file2 = _pl.read_csv("data/file2.csv")
This if course does not work, but the goal is that these files are read whenever I call import package_name, so that I can access them using e.g. package_name.file1. My interpretation of the pkg_resourcessolution in this case does not work either:
import pkg_resources
file1_path = pkg_resources.resource_filename('package_name', 'data/file1.csv')
file1 = _pl.read_csv(file1_path)
Any help here is much appreciated.
I am trying to build a pip package from source code in a Git repository that has multiple packages that share a common package. (I am not allowed to change the structure of this Git repository.)
The structure is:
├── common
│ ├── __init__.py
│ ├── run_helpers
│ │ ├── __init__.py
│ │ ├── aws.py
│ │ └── s3.py
└── components
└── redshift_unload
├── redshift_unload
│ ├── __init__.py
│ └── run.py
└── setup.py
My setup.py is as follows:
from setuptools import setup, find_packages
setup(
...
packages=find_packages(),
package_dir={"": "."},
entry_points={
"console_scripts": ["redshift_unload=redshift_unload.run:main"]
}
)
Looking at other answers here, things I have tried so far include:
Specifying the actual package names in the packages= line instead of using find_packages().
Passing where="../../" to find_packages()
Using find_packages() + find_packages(where="../../") in the packages=` line.
Everything I can think of in the packages_dir line.
When I run pip install . I get, the package installs fine, but then when I run the installed python script I get:
# redshift_unload
Traceback (most recent call last):
File "/usr/local/bin/redshift_unload", line 5, in <module>
from redshift_unload.run import main
File "/usr/local/lib/python3.8/site-packages/redshift_unload/run.py", line 9, in <module>
from common._run_helpers.aws import get_boto3_session
ModuleNotFoundError: No module named 'common'
What did work:
If I moved the common directory to components/redshift_unload, then it works fine. But I can't do this. I also tried placing a symlink there in its place, but seems like that doesn't work either.
Is there a way to make this work?
I believe I have found the best solution to this.
Based on this comment here, I concluded that what I am trying to do is not intended or supported.
However, I found a workaround as follows works fine:
from pathlib import Path
from shutil import rmtree, copytree
from setuptools import setup, find_packages
src_path = Path(os.environ["PWD"], "../../common")
dst_path = Path("./common")
copytree(src_path, dst_path)
setup(
...
packages=find_packages(),
package_dir={"": "."},
entry_points={
"console_scripts": ["redshift_unload=redshift_unload.run:main"]
}
)
rmtree(dst_path)
The key insight here is that, while packaging occurs in a temporary directory, the value of os.environ["PWD"] is available to the process, such that the common directory can be copied temporarily and then cleaned up again (using shutil functions copytree and rmtree) into a location that will be found by find_packages() before the setup() function is called.
I am trying to write a package with the following structure
/package
setup.py
/subpackage1
subpackage1.py
__init__.py
/subpackage2
subpackage2.py
__init__.py
/utils
some_other_files_and_codes
__init__.py
My setup.py currently looks like this:
from setuptools import setup, find_packages
setup(
name = 'subpackage1',
version = '1.0',
install_requires=['numpy',
'scipy'],
packages = find_packages(),
)
I then install it using pip install -e . from the /package folder.
However, I am not able to import subpackage2, only subpackage1.
I would like to be able to import them as
from package import subpackage1
from package import subpackage2
This is important because subpackage1 and subpackage2 exist as standalone packages too on my system.
Could someone help me with this?
The snippets you are showing do not make sense. Looks like there's a misunderstanding, in particular there's probably confusion between the name of the Python project and the names of the top-level importable packages.
In the setuptools.setup() function call, the parameter to the name argument should be the name of the project, not the name of an importable top level package. They can be the same names, but not necessarily.
The following might make it more explicit:
MyPythonProject
├── my_importable_package_one
│ ├── __init__.py
│ └── my_module_foo.py
├── my_importable_package_two
│ ├── __init__.py
│ └── my_module_bar.py
└── setup.py
setup.py
import setuptools
setuptools.setup(
name='MyPythonProject',
version='1.2.3',
packages=['my_importable_package_one', 'my_importable_package_two'],
# ...
)
from my_importable_package_one import my_module_foo
from my_importable_package_two import my_module_bar
Maybe this article on the terminology of Python packaging might help.
I'm writing a set of tools to be used in scripting and debugging, which I'm packaging. While I can import the package itself, I get a ModuleNotFoundError whenever I try to import the modules in the package.
package_dir
│ README.md
│ setup.py
│
└───package
│ │ __init__.py
│ │
│ └───utils
│ │ __init__.py
│ │ image_utils.py
│ │ ...
│
└───...
Installation:
using python setup.py install seems to work fine, and I can successfully import package
__init__.py:
My understanding is that I don't need the __init__.py files since I'm running python3.6, but I've tried every combination with and without __init__.py under package/ and utils/, with no change in circumstances. Regardless of the configuration, import package works and I cannot import any modules.
How I'm importing:
I'm in a Jupyter notebook outside the package structure, and I've tried the following:
import package (works)
from package import utils (doesn't work: ImportError)
from package.utils import image_utils (doesn't work: ModuleNotFoundError)
from package.utils.image_utils import func (doesn't work: ModuleNotFoundError)
I've also tried importing in the package's __init__.py using relative imports, with no success. I definitely don't want to use sys.path.insert() every time I want to use these tools.
I've scoured tutorials, documentation, and stack overflow. Anyone see what I'm missing?
I found that a previous version (without proper __init__.py files in all module directories or import statements in the package's __init__.py) was preventing my latest install from being recognized.
To fix this, I first tried to remove all files created during install:
python setup.py install --record installfiles.txt
But this didn't actually show me everything that I needed to remove.
I also removed the build and cached files within the package: build/, package/, dist/, and .egg-info/ (as well as all caches)
Then I removed the site package: /anaconda3/envs/[env_name]/lib/python3.6/site-packages/[package_name]
And removed the package from the easy-install .pth: /anaconda3/envs/[env_name]/lib/python3.6/site-packages/easy_install.pth
And the .egg-info: /anaconda3/envs/[env_name]/lib/python3.6/site-packages/[package-name].egg-info
This allowed me to install properly. Thanks to everyone for the help!
I have a project which I want to structure like this:
myproject
├── api
│ ├── __init__.py
│ └── api.py
├── backend
│ ├── __init__.py
│ └── backend.py
├── models
│ ├── __init__.py
│ └── some_model.py
└── __init__.py
Now, I want to import the module some_model.py in both api.py and backend.py. How do I properly do this?
I tried:
from models import some_model
but that fails with ModuleNotFoundError: No module named 'models'.
I also tried:
from ..models import some_model
which gave me ValueError: attempted relative import beyond top-level package.
What am I doing wrong here? How can I import a file from a different directory, which is not a subdirectory?
Firstly, this import statement:
from models import some_model
should be namespaced:
# in myproject/backend/backend.py or myproject/api/api.py
from myproject.models import some_model
Then you will need to get the directory which contains myproject, let's call this /path/to/parent, into the sys.path list. You can do this temporarily by setting an environment variable:
export PYTHONPATH=/path/to/parent
Or, preferably, you can do it by writing a setup.py file and installing your package. Follow the PyPA packaging guide. After you have written your setup.py file, from within the same directory, execute this to setup the correct entries in sys.path:
pip install --editable .
Unfortunately, Python will only find your file if your file is in the systems path. But fear not! There is a way around this!
Using python's sys module, we can add a directory to the path just while Python is running, and once Python stops running, it will remove it from the path.
You can do this by:
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import [file]
It is important to import sys and set the directory path before you import the file however.
Good luck!
Jordan.
I would lay out two approaches:
Simply import some_model via absolute importing:
from myproject.models import some_model
Note that the myproject should be treated as an module (i.e. having __init__.py)
Or
You can add the previous path to the sys.path which I use in such parallel level modules:
import sys
sys.path.append('../')
from models import some_model