I would like to have a nice hierarchy of modules for a large
project.. (Python seems to get in the way of this) I am confused about
the distinction of modules and packages and how they relate to the C++
concept of a namespace. For concreteness my project is a compiler and
the code generation phases want to query properties from some set of
abstract representations which are maintained in a different directory
(actually far away in the hierarchy)
The problem can be stated as:
Ass: Let a.py and b.py be two source files somewhere in the project hierarchy
Then: I want to refer to the functions defined in b.py from
a.py -- ideally with a relative path from the well-defined root
directory of the project (which is /src). We want a general-purpose
solution for this, something which will always work..
Dirty hack: It sounds absurd but my putting all sub-directories that contain .py on this
project into PYTHONPATH we will be able to reference them with their name, but with this
the reader of the code loses any sense of hierarchy & relation about the different project
classes etc..
Note: The tutorial on Python.org only mentions the special case of referring from a file c.py to a file d.py placed in its parent directory. Where is the generality that makes
Python scale to really large projects here?
I am not sure if it is the question, but let us see.
Suppose I have the following package scheme (__init__.py files excluded for readability):
foo/baz/quux/b.py
foo/baz/quux/quuuux/c.py
foo/bar/a.py
My foo/baz/quux/b.py file contains this:
def func_b():
print 'func b'
and my foo/baz/quux/quuuux/c.py is:
def func_c():
print 'func c'
If the root directory which contains foo (in your case, src*) is in the Python path, aur foo/bar/a.py file can import any other module starting from foo:
import foo.baz.quux.b as b
import foo.baz.quux.quuuux.c as c
def func_a():
b.func_b()
c.func_c()
And one can use the foo/bar/a.py this way:
import foo.bar.a as a
a.func_a()
Have you tried it? Did you got some error?
* When you deploy your project, I do not believe the root will be src but let us maintain it simple by omitting it :)
Related
The Challenge
I am working on a Python project that will act as a translation layer for the SCPI command line interface on scientific instruments.
It is similar to the concept described in Multi layer package in python however my situation is slightly more complex and I can't seem to figure out how to make it work.
The following is a representation of my project structure (I am happy to change it if it is required):
Some things to keep in mind
The names of the files in translator_lib.instruments.supplierX.moduleX are only class1.py, class2.py and class3.py, the rest of the filename is for reference to describe where they are derived from.
Many of the modules and classes inside of supplier1 and supplier2 have the same names (as in the example).
Every directory contains a __init__.py file
The __init__.py files (based on the example layout) look as follows (Note I only have one module for testing)
translator_lib\__init__.py
from .instruments import supplier1
__all__ = ['supplier1']
translator_lib\instruments\__init__.py
from .supplier1 import module1
__all__ = ['module1']
What I'm trying to do
Compile three libraries called my_translator_lib, supplier1_translator_lib and supplier2_translator_lib.
Reason
The development team would import my_translator_lib to do what they need to, but if we want to send sample code to supplier1, we want to send them the supplier1_translator_lib and they should only be able to import supplier1_translator_lib
Example 1 : Developer
from translator_lib.instruments import supplier1
from translator_lib.instruments import supplier2
class DoStuff:
__init__(self):
self.sup1_class1 = supplier1.module1.class1.class1()
self.sup2_class1 = supplier2.module1.class1.class1()
Example 2 : Supplier 1
from supplier1_translator_lib import module1
from supplier1_translator_lib import module2
class DoStuff:
__init__(self):
self.class1 = module1.class1.class1()
self.class2 = module1.class2.class2()
I've tried multiple combinations and sections from How to create a Python library and Deep dive: Create and publish your first Python library. I manage to create a library and install it, but ultimately I can only import my_translator_lib and nothing else is visible or available.
Any help in this regard would be truly appreciated.
have you created the __init__.py files?
you need to have then in every subfolder to your module.
Files named __init__.py are used to mark directories on disk as Python package directories. Try to follow this:
mydir/spam/__init__.py
mydir/spam/module.py
If this not solve your problem, as a last resource you can try to sys.path.append('path_to_other_modules')
I understand this is not a 'coding' question but I need a coding solution.
I have multiple packages I wrote, all supposed to be encapsulated, and independent on external parameters other than a few input arguments.
On the other side, I have a general file constants.py with 50+ constants which those packages better use in order to provide an output dictionary without hardcoded names :
A PACKAGE OUTPUT:
{
'sub_name':xyz,
'sub_type':yzg
}
Here sub_name should be given to the package as input so the general program will know what to do with sub_name output.
How should I share constants.py with the packages ?
The obvious way is to just import constants.py, which makes the package dependent on an external file somewhere else in the program.
The other way is to keep constants in some class Keys and send it as argument.
Could/should I send constants.py as an argument ?
I find it hard to understand how packages should be written and organized when inside a larger project, in a way they can be reused by other devs independently.
You can store constants in __init__.py and import them in submodules.
Example:
main_module/__init__.py
# inside this file
CONSTANT_A = 42
CONSTANT_B = 69
then:
main_module/submodule.py
# inside this file
from main_module import CONSTANT_A
print(CONSTANT_A)
>>42
I have been tasked with researching improvements to a Python tool. However, the current tool is a mess. Just a collection of several hundred functions and imports spread across 10 files all in the same level of directory, all cross referencing each other, with each file importing some subset of the others, sometimes recursively.
There is an underlying hierarchy there, and so I intend to convert it all into a module, with sub-modules, and then to further refactor it into a class rather than a set of nested functions.
I am having some issue with imports though. I am self taught with python, and I am trying to remain PEP8 compliant, with little knowledge of how to. An additional complication is the system I am using is secure, and is unable to have an IDE installed on it, so I am battling with a glorified text editor with a terminal built in.
I thought I could import the sub-modules together in the highest level of the module, and they would then be available to all sub-modules, but this seems to produce name errors. My file structure and code is as follows:
Directory
dir
|
+-- run_module.py
|
+--top_module
|
+--__init__.py
|
+--module.py
|
+--sub_module
|
+--__init__.py
|
+--sub_module_one.py
|
+--sub_module_two.py
Content of run_module.py
from top_module import module
module.run_the_thing()
Content of init.py
Currently, both __init__ files are empty, though this issue persists if top_mod.__init__ has the imports instead of module, as well as importing module and being imported by run_module
Content of module.py
from .sub_module import sub_module_one, sub_module_two
def run_the_thing():
sub_module_one.hello()
sub_module_two.greeting()
Content of sub_module_one.py
def hello():
print('I am sub-module 1.')
Content of sub_module_two.py
def hello():
print('I am sub-module 2.')
def callout():
print('And this is my friend:')
sub_module_one.hello()
def greeting():
hello()
callout()
When run, it prints the output from sub_module_one as expected, and prints everything in sub_module_two up until the call to sub_module_one.
I cant say I am surprised by this behaviour, but is there a solution that will allow me to import all the sub-modules in the top level script, leaving them available for all the others, or will I have to have imports in each and every sub-module?
Clearly this isnt too messy for my MWE, but the real code will be horrible spaghetti.
ADVICE REQUEST: Best Practices for Python Imports
I need advice re: how major projects do Python imports and the standard / Pythonic way to set this up.
I have a project Foobar that's a subproject of another, much larger and well-used project Dammit. My project directory looks like this:
/opt/foobar
/opt/foobar/bin/startFile.py
/opt/foobar/bin/tests/test_startFile.py
/opt/foobar/foobarLib/someModule.py
/opt/foobar/foobarLib/tests/test_someModule.py
So, for the Dammit project, I add to PYTHONPATH:
export PYTHONPATH=$PYTHONPATH:/opt/foobar
This lets me add a very clear, easy-to-track-down import of:
from foobarLib.someModule import SomeClass
to all 3 of:
* Dammit - using foobarLib in the import makes everyone know it's not in Dammit project, it's in Foobar.
* My startFile.py is an adjunct process that has the same pythonpath import.
* test_someModule.py has an explicit import, too.
PROBLEM: This only works for foobarLib, not bin.
I have tests I want to run on in /opt/foobar/bin/tests/test_startFile.py. But, how to set up the path and do the import? Should I do a relative import like this:
PROJ_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__),".."))
sys.path.insert(0, PROJ_ROOT)
Or, should I rename the bin dir to be foobarBin so I do the import as:
from foobarBin.startFile import StartFile
I'm tempted to do the following:
/opt/foobarProject/foobar
/opt/foobarProject/foobar/bin/startFile.py
/opt/foobarProject/foobar/bin/tests/test_startFile.py
/opt/foobarProject/foobar/foobarLib/someModule.py
/opt/foobarProject/foobar/foobarLib/tests/test_someModule.py
then, I can do all my imports with:
import foobar.bin.startFile
import foobar.lib.someModule
Basically, for large Python projects (AND THEIR ADJUNCTS), it seems to me we have goals of:
minimize number of dirs added to pythonpath;
make it obvious where imports are coming from. That is, people use 'import lib.thing' a lot, and if there's more than one directory in the pythonpath named 'lib',
it's troublesome / non-obvious where that is, short of invoking
python and searching sys.modules, etc.
minimize number of times we add paths to sys.path since that's runtime and somewhat non-obvious.
I've used Python a long time, and been part of projects that do it various ways, some of them more stupid and some less so. I guess I'm wondering if there is a best-practices here, and the reason for it? Is my convention of adding foobarProject layer okay, good, bad, or just one more way among many?
I'm currently coding an app which is basically structured that way :
main.py
+ Package1
+--- Class1.py
+--- Apps
+ Package2
+--- Class1.py
+--- Apps
So I have two questions :
First, inside both packages, there are modules needed by all Apps, eg : re. Is there a way I can import a module for the whole package at once, instead of importing it in every file that needs it ?
And, as you can see, Class1 is used in both packages. Is there a good way to share it between both packages to avoid code duplication ?
I would strongly recommend against doing this: by separating the imports from the module that uses the functionality, you make it more difficult to track dependencies between modules.
If you really want to do it though, one option would be to create a new module called common_imports (for example) and have it do the imports you are after.
Then in your other modules, add the following:
from common_imports import *
This should give you all the public names from that module (including all the imports).
To answer your second question, if your two modules named Class1.py are in fact the same, then you should not copy it to both packages. Place it in a package which will contain only code which is common to both, and then import it. It is absolutely not necessary to copy the file and try to maintain each change in both copies.
Q1:
You Must find a way to import your package, thus you have two choices:
(Please correct me if I'm wrong or not thorough)
1. Look at James' solution, which you need to define a class, put all the modules inside and finally import them to your Sub-classes
2. Basically import nothing to your Main class, but instead, import only once to your Sub-classes
For example:(inside A_1 subclass)
import re
def functionThatUseRe(input):
pass
Then inside your main class, just do
try:
from YourPackage import A_1 #windows
except:
import A_1 #MAC OSX
A_1.functionThatUseRe("")
And you completely avoided importing modules multiple times
Q2: put your class1.py in the same directory with your main class, or move it to another folder, in Package1(&2).Apps
import Class1
Start using the code from there