I'm new to Python for the last 3 months but have quite a bit of development
experience. I'm trying to figure out a good way to manage a collection of
functions and classes that are shared across several projects. I'm working on
a Windows 10 machine that I don't have admin access to, so the PATH variable is not an option. I'm using VS Code with Python 3.10
I have several active projects and my current working directory structure is:
python
Project A
Project B
Project C
Common
__init__.py (empty)
ClassA.py
ClassB.py
functions.py
I've added a .pth file in AppData/Local/Programs/Python/Python310/Lib/site-packages
which contains the path to the python root folder.
Right now I'm able to use this configuration by importing each file as a separate module:
from Common.ClassA import ClassA
from Common.ClassB import ClassB
from Common import functions as fn
I would like to do something like:
from Common import ClassA, ClassB, functions as fn
Just looking for some experienced advice on how to manage this situation. Thanks to any and
all who have time to respond.
(disclaimer, I am an admin on my mac, but none of what I am doing here required sudo permissions).
One way to do that is to put your common code in a "package", say common and use pip to do an editable local install. via pip install -e common.
After installation, your Python path is modified so that it includes the directory where common lives and your project-side code can then use it like:
from common.classa import ClassA
Now, writing an installable package is not that trivial, but TRIVIAL nowadays and this is likely the more robust approach, over modifying pythonpath with .pth files - been there, done that myself.
Now, as far as what your imports can look like in your project A, B, C code you will find that many packages do an import of constituent files in their __init__.py.
common/__init__.py:
from .classa import ClassA
from .classb import ClassB
import .functions
which means ProjectB can use:
import common
a = common.ClassA()
res = common.functions.foobar(42)
You can look at sqlalchemy's init.py for that type of approach:
from .engine import AdaptedConnection as AdaptedConnection
from .engine import BaseRow as BaseRow
from .engine import BindTyping as BindTyping
from .engine import ChunkedIteratorResult as ChunkedIteratorResult
from .engine import Compiled as Compiled
from .engine import Connection as Connection
which my own code can then use as:
import sqlalchemy
...
if not isinstance(engine, sqlalchemy.engine.base.Engine):
...
Note: none of this explanation should be taken as detracting from the comments and answers reminding you that Python can put any number of functions and classes into the same .py file. Python is not Java. But in practice, a Python file with over 400-500 lines of code is probably looking for a bit of refactoring. Not least because that facilitates git-based merging if those become relevant. And also because it facilitates code discovery: "Ah, a formatting question. Let's look in formatting.py"
OK, so how much work IS setting up a locally installed package?
TLDR: very little nowadays.
Let's take the package structure and Python files first
(this is under a directory called testpip)
├── common
│ ├── __init__.py
│ └── classa.py
common/__init__.py:
from .classa import A
from pathlib import Path
class B:
def __repr__(self) -> str:
return f"{self.__class__.__name__}"
def info(self):
pa = Path(__file__)
print(f"{self} in {pa.relative_to(pa.parent)}")
common/classa.py:
class A:
def __repr__(self) -> str:
return f"{self.__class__.__name__}"
def whoami(self):
print(f"{self}")
Let's start with just that and try a pip install.
testpip % pip install -e .
Obtaining file:.../testpip
ERROR: file:.../testpip does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
OK, I know setup.py used to be complicated, but what about that pyproject.toml?
There's a write up about a minimal pyproject.toml here.
But that still seemed like a lot of stuff, so I ended up with.
echo "" > pyproject.toml. i.e. an empty pyproject.toml
(yes, a touch would do, but the OP is on Windows)
testpip % pip install -e .
Obtaining file:///.../testpip
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: common
Building editable for common (pyproject.toml) ... done
Created wheel for common: filename=common-0.0.0-0.editable-py3-none-any.whl size=2266 sha256=fe01aa92de3160527136d13a233bfd9ff92da973040981631a4bb8f372adbb0b
Stored in directory: /private/var/folders/bk/_1cwm6dj3h1c0ptrhvr2v7dc0000gs/T/pip-ephem-wheel-cache-l3v8jbfr/wheels/1a/64/41/6ec6e2e75e362f2818c47a49356f82be33b0a6dba83b41354c
Successfully built common
Installing collected packages: common
Successfully installed common-0.0.0
And now, let's go to another directory and try it out.
src/testimporter.py:
from common import A, B
a = A()
a.whoami()
b = B()
b.info()
python testimporter.py:
A
B in __init__.py
The full project structure ended up as:
.
├── common 👈 your Python code
│ ├── __init__.py
│ └── classa.py
├── common.egg-info 👈 generated by pip install -e .
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
├── pyproject.toml 👈 an EMPTY file to make pip install -e work
The easiest way would be to package everything in Common into a single .py file in the same folder as your projects.
The reasoning is that when you do
from Common.ClassA import ClassA
It looks in the Common folder, finds the ClassA file, and imports the ClassA class.
By organizing your directory structure like this:
Project A
Project B
Project C
Common.py
Then you can just run:
from Common import ClassA, ClassB, functions as fn
I've tried reading through questions about sibling imports and even the
package documentation, but I've yet to find an answer.
With the following structure:
├── LICENSE.md
├── README.md
├── api
│ ├── __init__.py
│ ├── api.py
│ └── api_key.py
├── examples
│ ├── __init__.py
│ ├── example_one.py
│ └── example_two.py
└── tests
│ ├── __init__.py
│ └── test_one.py
How can the scripts in the examples and tests directories import from the
api module and be run from the commandline?
Also, I'd like to avoid the ugly sys.path.insert hack for every file. Surely
this can be done in Python, right?
Tired of sys.path hacks?
There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.
Summary
Wrap the code into one folder (e.g. packaged_stuff)
Create setup.py script where you use setuptools.setup(). (see minimal setup.py below)
Pip install the package in editable state with pip install -e <myproject_folder>
Import using from packaged_stuff.modulename import function_name
Setup
The starting point is the file structure you have provided, wrapped in a folder called myproject.
.
└── myproject
├── api
│ ├── api_key.py
│ ├── api.py
│ └── __init__.py
├── examples
│ ├── example_one.py
│ ├── example_two.py
│ └── __init__.py
├── LICENCE.md
├── README.md
└── tests
├── __init__.py
└── test_one.py
I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.
api.py
As a test case, let's use the following ./api/api.py
def function_from_api():
return 'I am the return value from api.api!'
test_one.py
from api.api import function_from_api
def test_function():
print(function_from_api())
if __name__ == '__main__':
test_function()
Try to run test_one:
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
File ".\myproject\tests\test_one.py", line 1, in <module>
from api.api import function_from_api
ModuleNotFoundError: No module named 'api'
Also trying relative imports wont work:
Using from ..api.api import function_from_api would result into
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
File ".\tests\test_one.py", line 1, in <module>
from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package
Steps
Make a setup.py file to the root level directory
The contents for the setup.py would be*
from setuptools import setup, find_packages
setup(name='myproject', version='1.0', packages=find_packages())
Use a virtual environment
If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)
Create virtual env
python -m venv venv
Activate virtual env
source ./venv/bin/activate (Linux, macOS) or ./venv/Scripts/activate (Win)
To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.
Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis
PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>
and your folder tree should look like this**
.
├── myproject
│ ├── api
│ │ ├── api_key.py
│ │ ├── api.py
│ │ └── __init__.py
│ ├── examples
│ │ ├── example_one.py
│ │ ├── example_two.py
│ │ └── __init__.py
│ ├── LICENCE.md
│ ├── README.md
│ └── tests
│ ├── __init__.py
│ └── test_one.py
├── setup.py
└── venv
├── Include
├── Lib
├── pyvenv.cfg
└── Scripts [87 entries exceeds filelimit, not opening dir]
pip install your project in editable state
Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.
In the root directory, run
pip install -e . (note the dot, it stands for "current directory")
You can also see that it is installed by using pip freeze
(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0
Add myproject. into your imports
Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the setup.py & pip install will work still work fine. See an example below.
Test the solution
Now, let's test the solution using api.py defined above, and test_one.py defined below.
test_one.py
from myproject.api.api import function_from_api
def test_function():
print(function_from_api())
if __name__ == '__main__':
test_function()
running the test
(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!
* See the setuptools docs for more verbose setup.py examples.
** In reality, you could put your virtual environment anywhere on your hard disk.
Seven years after
Since I wrote the answer below, modifying sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements
Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using setup.cfg to store the metadata)
Using the -m flag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
For the tests, specifically, pytest is able to find the api package in this situation and takes care of the sys.path hacks for you
So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through pip -e is probably your best bet, even if it is not perfect yet.
Old answer
As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a __main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of
The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern.
(here)
Though, I use this pattern on a regular basis with
# Ugly hack to allow absolute import from the root folder
# whatever its name is. Please forgive the heresy.
if __name__ == "__main__" and __package__ is None:
from sys import path
from os.path import dirname as dir
path.append(dir(path[0]))
__package__ = "examples"
import api
Here path[0] is your running script's parent folder and dir(path[0]) your top level folder.
I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example api's parent folder).
Here is another alternative that I insert at top of the Python files in tests folder:
# Path hack.
import sys, os
sys.path.insert(0, os.path.abspath('..'))
You don't need and shouldn't hack sys.path unless it is necessary and in this case it is not. Use:
import api.api_key # in tests, examples
Run from the project directory: python -m tests.test_one.
You should probably move tests (if they are api's unittests) inside api and run python -m api.test to run all tests (assuming there is __main__.py) or python -m api.test.test_one to run test_one instead.
You could also remove __init__.py from examples (it is not a Python package) and run the examples in a virtualenv where api is installed e.g., pip install -e . in a virtualenv would install inplace api package if you have proper setup.py.
I don't yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For examples or tests to import stuff from ..\api, it would look like:
import sys.path
import os.path
# Import from sibling directory ..\api
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..")
import api.api
import api.api_key
For siblings package imports, you can use either the insert or the append method of the [sys.path][2] module:
if __name__ == '__main__' and if __package__ is None:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
import api
This will work if you are launching your scripts as follows:
python examples/example_one.py
python tests/test_one.py
On the other hand, you can also use the relative import:
if __name__ == '__main__' and if __package__ is not None:
import ..api.api
In this case you will have to launch your script with the '-m' argument (note that, in this case, you must not give the '.py' extension):
python -m packageName.examples.example_one
python -m packageName.tests.test_one
Of course, you can mix the two approaches, so that your script will work no matter how it is called:
if __name__ == '__main__':
if __package__ is None:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
import api
else:
import ..api.api
For readers in 2021: If you're not confident with pip install -e :
Consider this hierarchy, as recommended by an answer from Relative imports in Python 3:
MyProject
├── src
│ ├── bot
│ │ ├── __init__.py
│ │ ├── main.py
│ │ └── sib1.py
│ └── mod
│ ├── __init__.py
│ └── module1.py
└── main.py
The content of main.py, which is the starting point and we use absolute import (no leading dots) here:
from src.bot import main
if __name__ == '__main__':
main.magic_tricks()
The content of bot/main.py, which takes advantage of explicit relative imports:
from .sib1 import my_drink # Both are explicit-relative-imports.
from ..mod.module1 import relative_magic
def magic_tricks():
# Using sub-magic
relative_magic(in=["newbie", "pain"], advice="cheer_up")
my_drink()
# Do your work
...
Now here comes the reasoning:
When executing python MyProject/main.py, the path/to/MyProject is added into the sys.path.
The absolute import import src.bot will read it.
The from ..mod part means it will go up one level to MyProject/src.
Can we see it? YES, since path/to/MyProject is added into the sys.path.
So the point is:
We should put the main script next to MyProject/src, since that when doing relative-referencing, we won't go out of the src, and the absolute import import src. provides the just-fit scope for us: the src/ scope.
See also: ModuleNotFoundError: No module named 'sib1'
TLDR
This method does not require setuptools, path hacks, additional command line arguments, or specifying the top level of the package in every single file of your project.
Just make a script in the parent directory of whatever your are calling to be your __main__ and run everything from there. For further explanation continue reading.
Explanation
This can be accomplished without hacking a new path together, extra command line args, or adding code to each of your programs to recognize its siblings.
The reason this fails as I believe was mentioned before is the programs being called have their __name__ set as __main__. When this occurs the script being called accepts itself to be on the top level of the package and refuses to recognize scripts in sibling directories.
However, everything under the top level of the directory will still recognize ANYTHING ELSE under the top level. This means the ONLY thing you have to do to get files in sibling directories to recognize/utilize each other is to call them from a script in their parent directory.
Proof of Concept
In a dir with the following structure:
.
|__Main.py
|
|__Siblings
|
|___sib1
| |
| |__call.py
|
|___sib2
|
|__callsib.py
Main.py contains the following code:
import sib1.call as call
def main():
call.Call()
if __name__ == '__main__':
main()
sib1/call.py contains:
import sib2.callsib as callsib
def Call():
callsib.CallSib()
if __name__ == '__main__':
Call()
and sib2/callsib.py contains:
def CallSib():
print("Got Called")
if __name__ == '__main__':
CallSib()
If you reproduce this example you will notice that calling Main.py will result in "Got Called" being printed as is defined in sib2/callsib.py even though sib2/callsib.py got called through sib1/call.py. However if one were to directly call sib1/call.py (after making appropriate changes to the imports) it throws an exception. Even though it worked when called by the script in its parent directory, it will not work if it believes itself to be on the top level of the package.
You need to look to see how the import statements are written in the related code. If examples/example_one.py uses the following import statement:
import api.api
...then it expects the root directory of the project to be in the system path.
The easiest way to support this without any hacks (as you put it) would be to run the examples from the top level directory, like this:
PYTHONPATH=$PYTHONPATH:. python examples/example_one.py
Just in case someone using Pydev on Eclipse end up here: you can add the sibling's parent path (and thus the calling module's parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g. from sibling import some_class.
I wanted to comment on the solution provided by np8 but I don't have enough reputation so I'll just mention that you can create a setup.py file exactly as they suggested, and then do pipenv install --dev -e . from the project root directory to turn it into an editable dependency. Then your absolute imports will work e.g. from api.api import foo and you don't have to mess around with system-wide installations.
Documentation
If you're using pytest then the pytest docs describe a method of how to reference source packages from a separate test package.
The suggested project directory structure is:
setup.py
src/
mypkg/
__init__.py
app.py
view.py
tests/
__init__.py
foo/
__init__.py
test_view.py
bar/
__init__.py
test_view.py
Contents of the setup.py file:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
Install the packages in editable mode:
pip install -e .
The pytest article references this blog post by Ionel Cristian Mărieș.
I made a sample project to demonstrate how I handled this, which is indeed another sys.path hack as indicated above. Python Sibling Import Example, which relies on:
if __name__ == '__main__': import os import sys sys.path.append(os.getcwd())
This seems to be pretty effective so long as your working directory remains at the root of the Python project.
in your main file add this:
import sys
import os
sys.path.append(os.path.abspath(os.path.join(__file__,mainScriptDepth)))
mainScriptDepth = the depth of the main file from the root of the project.
Here is your case mainScriptDepth = "../../". Then you can import by specifying the path (from api.api import * ) from the root of your project.
The problem:
You simply can not get import mypackage to work in test.py. You will need either an editable install, change to path, or changes to __name__ and path
demo
├── dev
│ └── test.py
└── src
└── mypackage
├── __init__.py
└── module_of_mypackage.py
--------------------------------------------------------------
ValueError: attempted relative import beyond top-level package
The solution:
import sys; sys.path += [sys.path[0][:-3]+"src"]
Put the above before attempting imports in test.py. Thats it. You can now import mypackage.
This will work both on Windows and Linux. It will also not care from which path you run your script. It is short enough to slap it anywhere you might need it.
Why it works:
The sys.path contains the places, in order, where to look for packages when attempting imports if they are not found in installed site packages. When you run test.py the first item in sys.path will be something like /mnt/c/Users/username/Desktop/demo/dev i.e.: where you ran your file. The oneliner will simply add the sibling folder to path and everything works. You will not have to worry about Windows vs Linux file paths since we are only editing the last folder name and nothing else. If you project structure is already set in stone for your repository we can also reasonably just use the magic number 3 to slice away dev and substitute src
for the main question:
call sibling folder as module:
from .. import siblingfolder
call a_file.py from sibling folder as module:
from ..siblingfolder import a_file
call a_function inside a file in sibling folder as module:
from..siblingmodule.a_file import func_name_exists_in_a_file
The easiest way.
go to lib/site-packages folder.
if exists 'easy_install.pth' file, just edit it and add your directory that you have script that you want make it as module.
if not exists, just make it one...and put your folder that you want there
after you add it..., python will be automatically perceive that folder as similar like site-packages and you can call every script from that folder or subfolder as a module.
i wrote this by my phone, and hard to set it to make everyone comfortable to read.
First, you should avoid having files with the same name as the module itself. It may break other imports.
When you import a file, first the interpreter checks the current directory and then searchs global directories.
Inside examples or tests you can call:
from ..api import api
Project
1.1 User
1.1.1 about.py
1.1.2 init.py
1.2 Tech
1.2.1 info.py
1.1.2 init.py
Now, if you want to access about.py module in the User package, from the info.py module in Tech package then you have to bring the cmd (in windows) path to Project i.e.
**C:\Users\Personal\Desktop\Project>**as per the above Package example. And from this path you have to enter, python -m Package_name.module_name
For example for the above Package we have to do,
C:\Users\Personal\Desktop\Project>python -m Tech.info
Imp Points
Don't use .py extension after info module i.e. python -m Tech.info.py
Enter this, where the siblings packages are in the same level.
-m is the flag, to check about it you can type from the cmd python --help
I've been searching the net for quite some time now but I can't seem to wrap my head around on how can I distribute my python scripts for my end user.
I've been using my scripts on my command line using this command python samplemodule.py "args1"
And this is also the way I want my user to also use it on their end with their command line. But my worry is that this certain modules have dependencies on other library or modules.
My scripts are working when they are all in the Project's root directory, but everything crumbles when I try to package them and put them in sub directories.
An example of this is I can't now run my scripts since its having an error when I'm importing a module from the data subdirectory.
This is my project structure.
MyProject
\formatter
__init__.py
__main__.py
formatter.py
addfilename.py
addscrapertype.py
...\data
__init__.py
helper.py
csv_formatter.py
setup.py
The csv_formatter.py file is just a wrapper to call the formatter.main.
Update: I was now able to generate a tar.gz package but the package wasn't callable when installed on my machine.
This is the setup.py:
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="formatter",
version="1.0.1",
author="My Name",
author_email="sample#email.com",
description="A package for cleaning and reformatting csv data",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/RhaEL012/Python-Scripts",
packages=["formatter"],
include_package_data=True,
package_data={
# If any package contains *.txt or *.rst files, include them:
"": ["*.csv", "*.rst", "*.txt"],
},
entry_points={
"console_scripts": [
"formatter=formatter.formatter:main"
]
},
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.6',
install_requires=[
"pandas"
]
)
Now, after installing the package on the machine I wasn't able to call the module and it results in an error:
Z:\>addfilename "C:\Users\Username\Desktop\Python Scripts\"
Update: I try to install the setup.py in a virtual environment just to see where the error is coming from.
I install it then I get the following error: FileNotFoundError: [Errno 2] no such file or directory: 'README.md'
I try to include the README.md in the MANIFEST.in but still no luck.
So I try to make it a string just to see if the install will proceed.
The install proceed but then again, I encounter an error that says that the package directory 'formatter' does not exist
As I am not able to look into your specific files I will just explain how I usually tackle this issue.
This is the manner how I usually setup the command line interface (cli) tools. The project folder looks like:
Projectname
├── modulename
│ ├── __init__.py # this one is empty in this example
│ ├── cli
│ │ ├── __init__.py # this is the __init__.py that I refer to hereafter
│ ├── other_subfolder_with_scripts
├── setup.py
Where all functionality is within the modulename folder and subfolders.
In my __init__.py I have:
def main():
# perform the things that need to be done
# also all imports are within the function call
print('doing the stuff I should be doing')
but I think you can also import what you want into the __init__.py and still reference to it in the manner I do in setup.py.
In setup.py we have:
import setuptools
setuptools.setup(
name='modulename',
version='0.0.0',
author='author_name',
packages=setuptools.find_packages(),
entry_points={
'console_scripts': ['do_main_thing=modulename.cli:main'] # so this directly refers to a function available in __init__.py
},
)
Now install the package with pip install "path to where setup.py is" Then if it is installed you can call:
do_main_thing
>>> doing the stuff I should be doing
For the documentation I use: https://setuptools.readthedocs.io/en/latest/.
My recommendation is to start with this and slowly add the functionality that you want. Then step by step solve your problems, like adding a README.md etc.
I disagree with the other answer. You shouldn't run scripts in __init__.py but in __main__.py instead.
Projectfolder
├── formatter
│ ├── __init__.py
│ ├── cli
│ │ ├── __init__.py # Import your class module here
│ │ ├── __main__.py # Call your class module here, using __name__ == "__main__"
│ │ ├── your_class_module.py
├── setup.py
If you don't want to supply a readme, just remove that code and enter a description manually.
I use https://setuptools.readthedocs.io/en/latest/setuptools.html#find-namespace-packages instead of manually setting the packages.
You can now install your package by just running pip install ./ like you have been doing before.
After you've done that run: python -m formatter.cli arguments. It runs the __main__.py file you've created in the CLI folder (or whatever you've called it).
An important note about packaging modules is that you need to use relative imports. You'd use from .your_class_module import YourClassModule in that __init__.py for example. If you want to import something from an adjacent folder you need two dots, from ..helpers import HelperClass.
I'm not sure if this is helpful, but usually I package my python scripts using the wheel package:
pip install wheel
python setup.py sdist bdist_wheel
After those two commands a whl package is created in a 'dist' folder which you can then either upload to PyPi and download/install from there, or you can install it offline with the "pip install ${PackageName}.py"
Here's A useful user guide just in case there is something else that I didn't explain:
https://packaging.python.org/tutorials/packaging-projects/
I have several repos that I want to name space. All of the repos follow the standard Python folder structures where
repo1 - repo1 - __init__.py
Outermost repo1 folder is the root folder and the inner repo1 folder is the root of the module. All of these repos will be installed using
pip install -e .
Currently, import statements like the following is used to import these modules.
import repo1
import repo2
import repo3
Is there a way to name space these modules so that I can have
import mymodule.repo1
import mymodule.repo2
import mymodule.repo3
I have to achieve the name spacing while keeping the repos separate. Merging the repos is not an option at this moment.
Implementation details depends on your needs for version support and distribution, but take a look at setuptools namespace_packages, this will do the work.
As pointed above, packaging site has an useful page on namespaced packaging.
Example for native namespaces (python >=3.3). Project layout for isolated repos:
project_root1
├── finance_namespace # no __init__ file here, this is important
│ └── repo1
│ ├── __init__.py
│ └── module1.py
└── setup.py
===============================
# setup.py
import setuptools
setuptools.setup(
name='repo1',
version='1',
description='',
long_description='',
author='Big bank',
author_email='john#bank.com',
license='MIT',
packages=['finance_namespace.repo1'],
zip_safe=False,
)
Now, by making cd project_root1 && pip install -e . you should be able to do
>>> from finance_namespace.repo1 import module1
>>> module1.func()
After installing my python project with setup.py and executing it in terminal I get the following error:
...
from ui.mainwindow import MainWindow
File "/usr/local/lib/python2.7/dist-packages/EpiPy-0.1-py2.7.egg/epipy/ui/mainwindow.py", line 9, in <module>
from model.sir import SIR
ImportError: No module named model.sir
...
We assume we have the following structure of our project cookies:
.
├── setup.py
└── src
├── a
│ ├── aa.py
│ └── __init__.py
├── b
│ ├── bb.py
│ └── __init__.py
├── __init__.py
└── main.py
File: cookies/src/main.py
from a import aa
def main():
print aa.get_aa()
File cookies/src/a/aa.py
from b import bb
def get_aa():
return bb.get_bb()
File: cookies/src/b/bb.py
def get_bb():
return 'bb'
File: cookies/setup.py
#!/usr/bin/env python
import os
import sys
try:
from setuptools import setup, find_packages
except ImportError:
raise ImportError("Install setup tools")
setup(
name = "cookies",
version = "0.1",
author = "sam",
description = ("test"),
license = "MIT",
keywords = "test",
url = "asd#ads.asd",
packages=find_packages(),
classifiers=[
"""\
Development Status :: 3 - Alpha
Operating System :: Unix
"""
],
entry_points = {'console_scripts': ['cookies = src.main:main',],},
)
If I install cookies as root with $ python setup.py install and execute cookies I get the following error: ImportError: No module named b. How can I solve the problem.
What I would do is to use absolute imports everywhere (from epipy import ...). That's what is recommanded in PEP 328.
Your imports won't work anymore if the project is not installed. You can add the project directory to your PYTHONPATH, install the package, or, what I do when I'm in the middle of developing packages, install with the 'editable' option : pip install -e
In editable mode, instead of installing the package code in your python distribution, a pointer to your project is created. That way it is importable, but the package uses the live code in development.
Example:
I am developing a package in /home/jbchouinard/mypackage. Inside my code, I use absolute imports, e.g. from mypackage import subpackage.
If I install with pip install, the package will be installed in my distribution, let's say in /usr/lib/python2.7/dist-packages. If I make further changes to the package, I have to upgrade or uninstall/reinstall the package. This can get tedious quickly.
If I install with pip install -e, a pointer (a .pth file) is created in /usr/lib/python2.7/dist-packages towards /home/jbchouinard/mypackage. I can import mypackage as if it was installed normally, but the code used is the code at /home/jbchouinard/mypackage; any change is reflected immediately.
I had a similar issue with one of my projects.
I've been able to solve my issue by adding this line at the start of my module (before all imports besides sys & os, which are required for this insert), so that it would include the parent folder and by that it will be able to see the parent folder (turns out it doesn't do that by default):
import sys
import os
sys.path.insert(1, os.path.join(sys.path[0], '..'))
# all other imports go here...
This way, your main.py will include the parent folder (epipy).
Give that a try, hope this helps :-)