This works:
from story.apps.document import core
print core.submit()
This doesn't work:
from story import apps
print apps.document.core.submit()
"story" is a directory. Inside it, there is "apps" directory. Inside it, there is "document" directory. "core.py" is a file.
There is a __init__.py in every directory.
The __init__.py file tells python to interpret the directory as a package, but it does not necessarily tell python to import sub-packages or other files from the directory (although it may, if you add the appropriate import statements).
With large package hierarchies, it is often preferable to require sub-packages to be imported explicitly.
When you do story.apps.document import core, you're telling the Python interpreter to match a module of the description story.apps.document, import it, then load the variable core from its namespace into your current one.
Because core is a file module it has in its namespace variables defined within that file e.g., submit.
When you do from story import apps, you're telling the Python interpreter to match a module of the description story, import it, then load the variable apps from its namespace into your current one.
Because apps is a directory module it has within its namespace variables defined in its __init__.py and other modules in that directory. So apps knows about the document but it doesn't know anything about document's submodule core.
FYI: The reason this sometimes confuses people is because of stuff like this...
Works just fine:
# File1
import story.apps.document
story.apps.document.core()
Doesn't work:
# File2
import story
story.apps.document.core() # <-- Looks like the same function call, but is an Error
For file1, The import works because the the import operation tries to intelligently find things on the filesystem. The function call works because the module document was imported, and it is merely named story.apps.document.
For file2, the function call doesn't work because there's nothing intelligent about the dot operator, it merely attempts to access attributes on the Python object--it doesn't know anything about filesystems or modules.
when you do from story import apps, you don't include all the subpackages inside apps, for that, you do something like,
from story.apps.document import *
This only works if story/__init__.py imports apps.
IN the second example you are only importing the package. Python does not automatically import subpackages or modules. You have to explicitly do it, as in the first example. Do a dir(apps) and you'll see it is just an empty package.
Related
Consider this folder structure:
main.py
module_a/
aa.py
bb.py
__init__.py
In main.py, I import aa as:
from module_a import aa
aa.yyy()
Then in aa.py, I import bb and include its functions as:
import bb
bb.xxx()
However, when I run main.py, python says "No module named 'bb'".
May I know why this happens. What is the correct way to import bb.
Thanks!!!
I have tried to write aa.py as:
import .bb
bb.xxx()
But it still does not work.
why this happens
Because the aa folder is not a place that Python is searching for modules.
Python's imports are absolute by default. They only look in specific places determined by sys.path. In main.py, import module_a.aa works because the root folder of the project happens to be on the sys.path; that folder contains a module_a folder; and that folder contains aa.py.
What is the correct way to import bb.
Please use relative imports between files in your package. In this case, the necessary import in aa.py looks like:
from . import bb
Absolute imports are error-prone; a project that uses two packages whose contents have overlapping names will run into namespace collisions. (Sadly, the standard library uses absolute imports in most places, such that projects need to ban certain module names for safety.) Relative imports will also require much less maintenance, should you later rename a sub-package.
The only thing relative imports require is that the package gets loaded, which typically will happen automatically with the first (yes, absolute) import of any of the package contents. When the package is loaded, it automatically sets a __package__ attribute on the modules in that package, which Python can use to resolve the relative imports. It's important to note that relative imports are relative to the package hierarchy, not the directory structure, which is why this is necessary; imports like from .. import example do not work by figuring out the current file location and then going up a level in the directory hierarchy. Instead, they check the __package__ to figure out what the containing package is, then check the file/folder location for that, and work from there.
If the "driver" script is within the package, run it as a module, using the -m switch for Python. For example, from the root folder, if module_a/aa.py is the driver, use python -m module_a.aa. This instructs Python that module_a is the containing package for aa.py, and ensures it gets loaded even though no import in the code has loaded it.
Contrary to what many people will wrongly tell you, it is almost never required to manipulate sys.path; there are popular Python projects on GitHub, running hundreds of thousands of lines of code, which either do not use it at all or use it only once in an ancillary role (perhaps because of a special requirement for a documentation tool). Just don't do it.
Also contrary to what many people will wrongly tell you, __init__.py files are not required to create packages in Python. They are simply a place where additional code can be placed for package initialization - for example, to create aliases for sub-package contents, or to limit what will be imported with a *-import (by setting __all__).
import .bb
This is just invalid. Relative imports only use the from syntax. See above for the correct syntax.
Suppose the same file structure as stated in OP, and:
main.py:
import module_a.aa
module_a.aa.thisFile()
module_a.aa.module_a.bb.thisFile()
aa.py:
import module_a.bb
def thisFile():
print("aa")
bb.py:
def thisFile():
print("bb")
Then, this will print
aa
bb
like you would expect. The main difference here, is that bb.py is imported in aa.py via module_a.bb. Running main.py gives no problem, however, running aa.py does not work in this way. That is why you might want to add folders to your path, such that you can call a function from a different file without this trouble. This is done via:
import os, inspect, sys
current_folder = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_folder = os.path.dirname(current_folder)
sys.path.insert(0,parent_folder)
Then you can import your files such as import file. If you consider this option, I would suggest to do some reading about how this works. Tip: make sure you avoid cyclic import problems.
I have a rather simple setup:
[FOLDER]
|-> [Lib]
__init__.py (__all__=["modA","modB"])
modA.py (contains class named classA)
modB.py (contains class named classB + from modA import classA)
test1.py (from Lib.modA import classA
from Lib.modB import classB)
|-> [example]
test2.py (import sys
sys.path.append("../")
from Lib.modA import classA
from Lib.modB import classB)
Running test1.py from the Lib folder works perfectly without errors. Running test2.py from the example folder on the other hand requires the sys-patch to find Lib at all; however, it then crashes with No module named modA tracing back to the from modA import classA in modB.py via from Lib.modB import classB in test2.py.
How is one supposed to define an import in a module such that it will also work irrespective of the possible location of any future script that may use/import said module?
Python programs should be thought of as packages and modules, not as directories and files. While there is some overlap, packages and modules are more restrictive but also better encapsulated as a result.
Mixing both – say by manually modifying sys.path – should only be done as a last resort.
TLDR:
Use qualified imports based on the package:
from Lib.modA import classA/from .modA import classA instead of from modA import classA .
Use the environment for discovery:
Add search paths via PYTHONPATH instead of sys.path.
Start by deciding which ones are the top-level packages.
This is the point where we go from "directories/files" to "package". Notably, we cannot go "above" the top-level later on, so it should contain everything we need. However, we cannot remove anything "below" either, so it should be a tight enough selection.
In the example we could go as low as treating modA and siblings as their own module-package, and as high as FOLDER encompassing the entire project.
[FOLDER]
|-> [Lib]
| |-> modA.py
: :
It is reasonable to pick Lib since it represents a self-contained part. Going higher to FOLDER would be excessive, going lower to modA and siblings would break their relation as belonging together.
Everything under the top-level package folder now belongs to the package.
Prominently, FOLDER/Lib/modA.py is now the module Lib.modA. It is not FOLDER.Lib.modA, nor just modA.
Import package content only with absolute or relative qualified names.
Now that the top-level is established, all import statements must be made with regard to it. Refer to modules by their qualified name only, even when two modules share a more common parent:
# Lib.modB
# okay, fully qualified import
from Lib.modA import classA
# broken, unqualified import
from modA import classA
In order to avoid typing out the entire fully qualified name, one can use a relative name instead. This reuses the current module name to derive the fully qualified name of the module to import.
# Lib.modB
# okay, relative import
from .modA import classA
Note: Relative imports are a package operation, not a filesystem operation. One cannot go beyond the top-level, but descend into package namespaces.
Everything inside the module is now encapsulated and self-contained.
All qualified imports will work irrespective of the location of the package itself. As long as the top-level can be imported, everything below it can be imported as well.
Notably, this works irrespective of the location of any future script that may use/import the package.
Run packaged code with absolute qualified names
When executing code inside a package, Python has to know that the code is part of the package. Thus, it must be run with its absolute qualified named. Use the -m switch to do so:
# import and execute Lib.test1
python3 -m Lib.test1
Enable packages via the environment, not the program.
The point of defining a package is getting a single, self-contained entity that represents a library. One could zip up a package or similar, and it would still be a package.
In return, this means that the code itself should not break this abstraction by directly adding/changing directories to search packages.
Instead of having scripts assume the location of the package, the environment - of the script, the user, or the entire machine – should expose the package. There are basically two ways to do this:
Announce the package location as a search location. This is suitable for development as it is flexible but hard to maintain. The PYTHONPATH is appropriate for this.
Move the package to a search location used by Python. This is suitable for production and distribution as it requires effort but is easy to maintain. Packaging and using a package manager is appropriate for this.
To what I understand, when you are trying to run manually there is no path set as the environment, and when you just do
python file.py
python doesn't know in which environment we are running, which is precisely why, we do sys.path.append, to add the sibling directory path to env.
In the case if you are using an IDE like spyder or pycharm. you have an option to set the directory in which your program is located, or in pycharm it creates the whole program as a package where all the paths are handled by IDE.
I have three scripts:
C:\code\voiceTerm\master.py:
from voice_terminal_module.voice_terminal import VoiceTerminal
vterm = VoiceTerminal()
C:\code\voiceTerm\voice_terminal_module\voice_terminal.py:
from chatbot_module.chatbot_module import Chatbot
class VoiceTerminal:
print("INITIALIZING VOICE TERMINAL")
cb = Chatbot()
C:\code\voiceTerm\voice_terminal_module\chatbot_module\chatbot_module.py:
class Chatbot:
print("CHATBOT INITIALIZED")
Here is the wierd thing: When I run chatbot_module.py it works, and if I run voice_terminal.py it works. For some reason however, master.py errors out with the following message:
Traceback (most recent call last):
File "c:\code\voiceTerm\master.py", line 1, in <module>
from voice_terminal_module.voice_terminal import VoiceTerminal
File "c:\code\voiceTerm\voice_terminal_module\voice_terminal.py", line 1, in <module>
from chatbot_module.chatbot_module import Chatbot
ModuleNotFoundError: No module named 'chatbot_module'
Why does it work sometimes, but sometimes not?
You'll need to restructure your project in a way that you have either: (1) two separate and installed packages containing each module or; (2) one package containing all modules. The simplest way forward is (2) and it looks like:
master.py
src/ # Name it anything you wish
__init__.py
voice_terminal.py
chatbot_module.py
Then use relative imports in voice_terminal.py
from .chatbot_module import Chatbot
Rule: Packages does not know others exists unless the other package is installed, added to sys.path, a subpackage, or part of a parent package.
Python general importing guide (for your own code)
I'm sure there should be a duplicate for this by now, but I can't find it, so I'm writing a full essay.
To solve the problem properly, you must take care of two things:
Make sure that sys.path - the path that Python searches for modules - includes the path to the project root.
Make sure that the import statements in the code are written to work with that path.
The search path
For the first part, you must understand how sys.path works. It is a list of folders where Python will look, that is set up automatically at startup (even if you don't import sys - just like sys.argv is).
Normally, it will be a list which contains, in order:
A path to the entry point for execution (the details of this will depend on how Python is started; typically this is '' when running a .py file directly from the same directory, a path the another directory if you do e.g. python somewhere/else/file.py, and an explicit path when using the -m switch)
Paths to system libraries
Paths to virtual environment libraries, if a virtual environment is active
Paths to things that were explicitly installed by the user
You can modify this list, with the expected impact on future import statements; but you ordinarily should not.
To ensure your project root is on the path, normally you should just install the project - ideally into a virtual environment. Please see the Python packaging guide for details. Failing that, make sure to start the project from just outside the folder containing your "top-level" code. In OP's case, that means C:\code. This will ensure that C:\code (or something equivalent) is on sys.path, which we can then rely upon for the second step.
The imports
We can fundamentally do this in two ways: By absolute imports specifying the path from the root, or by relative imports specifying the path from the current source file. Read more on Stack Overflow about relative imports here and here.
Absolute imports
In either case, we want to treat the root folder for our project (here, C:\code\voiceTerm) as a package. For absolute imports, this means we will always mention the root folder name in our import path.
Thus:
In C:\code\voiceTerm\master.py: from voiceTerm.voice_terminal_module.voice_terminal import VoiceTerminal
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from voiceTerm.voice_terminal_module.chatbot_module.chatbot_module import Chatbot
(You don't really want to have _module in your folder or file names. It doesn't really add information, makes this considerably harder to type, and is actually a bit misleading.)
We can also import entire modules: import voiceTerm.master, import voiceTerm.voice_terminal_module.voice_terminal; import voiceTerm.voice_terminal_module.chatbot_module.chatbot_module. Additionally, by naming a file __init__.py, we become able to import the folder as a package: import voiceTerm; import voiceTerm.voice_terminal_module; import voiceTerm.voice_terminal_module.chatbot_module. We can also import a module from one of those packages: from voiceTerm import master; from voiceTerm.voice_terminal_module import voice_terminal; from voiceTerm.voice_terminal_module.chatbot_module import chatbot_module.
Relative imports
With relative imports, we must use the from syntax, and we can only import things from within our own package hierarchy. However, we have the advantage that we don't have to specify full paths, and we can rename packages without having to edit the code. My personal recommendation is to use relative imports where possible; this also makes a strong visual distinction with imports from system libraries or other third-party packages.
For a relative import, first we specify the package or sub-package that we're importing from, using a relative import path. By starting with a single ., we start from the current directory, looking for another module or package within the same folder.
Thus:
In C:\code\voiceTerm\master.py: from .voice_terminal_module.voice_terminal import VoiceTerminal to import the class; from .voice_terminal_module import voice_terminal to import the module.
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from .chatbot_module.chatbot_module import Chatbot to import the class; from .chatbot_module import chatbot_module to import the module.
Additional .s navigate up the package hierarchy (but we cannot go beyond the root this way). For example, from ... import master to import the top-level master.py from the lower-down chatbot_module.py. (Of course, this wouldn't actually work in this case, because it would be a circular import.) To import another source file (as a module) from the same directory, simply from . import other_file. You get the idea.
I solved it. You can appearantly use importlib for this. I replaced the import code in my voice_terminal.py script with this:
chatbot_module = SourceFileLoader("chatbot_module", "C:/code/voiceTerm/voice_terminal_module/chatbot_module/chatbot_module.py").load_module()
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
I have some code in a project which I'd like to reuse in another project. What do I need to do (in both folders) so that I can do this?
The directory structure is something like:
Foo
Project1
file1.py
file2.py
Bar
Project2
fileX.py
fileY.py
I want to use functions from file1.py and file2.py in fileX.py and fileY.py.
Ideally both projects will be an installable python package, replete with __init__.py and setup.py. They could then be installed with python setup.py install or similar.
If that is not possible, don't use execfile()! Manipulate the PYTHONPATH to add Foo so that import Project1.file1 works.
For example, from Project2/fileX.py:
from os import path
import sys
sys.path.append(path.abspath('../Foo'))
from Project1.file1 import something
However, the real answer is to make each a discrete installable package.
There's a lot going on here. you should read about python packages and module management http://docs.python.org/2/tutorial/modules.html#packages but the basic idea is that fileX needs to know where file1 and file2 are in order to use them.
To turn a folder into a package, it just needs to contain an __init__.py file. What I would suggest you do is (in a terminal)
$ touch Foo/__init__.py
$ touch Foo/Project1/__init__.py
(assuming you're using unix/linux).
Then somehow, fileX needs to know where the Foo package is. You can call sys.path.append(PATH) where PATH is the location of Foo.
finally inside fileX.py you'd have
import sys
sys.path.append(PATH) #replace PATH with the path to Foo
from Foo.Project1 import file1
#use its functions
file1.function_name(argument)
if you really want to just say function_name without the preceeding file1. you can import all of its functions by saying from Foo.Project1.file1 import * however please note that from module import * is highly frowned upon as it mixes names and make code less readable and understandable
You may want to make a module out of it and then import whatever you need from it.
You take the code you want to use in both projects, and you put it into a module, which you extract into a third separate project. That project you make into a package, which you can work on separately. You then release version of it, and reuse them in your other projects.
It is important that you have versions that you "release" so that you can keep track of which version of the module each project uses. Otherwise you will end up in a situation where an old project stops working because you have made incompatible changes to the common module.
If it's generically usable not only for you but for others, consider uploading it to PyPI.
I think you can add Foo to the current Python path using os.path.join() or os.path.append() and do from import Project1.file1 import function_name.