I currently have a module I created that has a number of functions.
It's getting quite large so I figured I should make it into a package and split the functions up to make it more manageable.
I'm just testing out how this all works before I do this for real so apologies if it seems a bit tenuous.
I've created a folder called pack_test and in it I have:
__init__.py
foo.py
bar.py
__init__.py contains:
__all__ = ['foo', 'bar']
from . import *
import subprocess
from os import environ
In the console I can write import pack_test as pt and this is fine, no errors.
pt. and two tabs shows me that I can see pt.bar, pt.environ, pt.foo and pt.subprocess in there.
All good so far.
If I want to reference subprocess or environ in foo.py or bar.py how do I do it in there?
If in bar.py I have a function which just does return subprocess.call('ls') it errors saying NameError: name 'subprocess' is not defined. There must be something I'm missing which enables me to reference subprocess from the level above? Presumably, once I can get the syntax from that I can also just call environ in a similar way?
The alternative as I could see it would be to have import subprocess in both foo.py and bar.py but then this seems a bit odd to me to have it appear across multiple files when I could have it the once at a higher level, particularly if I went on to have a large number of files rather than just 2 in this example.
TL;DR:
__init__.py :
import foo
import bar
__all__ = ["foo", "bar"]
foo.py:
import subprocess
from os import environ
# your code here
bar.py
import subprocess
from os import environ
# your code here
There must be something I'm missing which enables me to reference subprocess from the level above?
Nope, this is the expected behaviour.
import loads a module (if it isn't already), caches it in sys.modules (idem), and bind the imported names in the current namespace. Each Python module has (or "is") it's own namespace (there's no real "global" namespace). IOW, you have to import what you need in each module, ie if foo.py needs subprocess, it must explicitely import it.
This can seem a bit tedious at first but in the long run it really helps wrt/ maintainability - you just have to read the imports at the top of your module (pep 08: always put all imports at the beginning of the module) to know where a name comes from.
Also you should not use star imports (aka wild card imports aka from xxx import *) anywhere else than in your python shell (and even then...) - it's a maintainance time bomb. Not only because you don't know where each name comes from, but also because it's a sure way to rebind an already import name. Imagine that your foo module defines function "func". Somewhere you have "from foo import *; from bar import *", then later in the code a call to func. Now someone edits bar.py and adds a (distinct) "func" function, and suddenly you call fails, because you're not calling the expected "func". Now enjoy debugging this... And real-life examples are usually a bit more complex than this.
So if you fancy your mental sanity, don't be lazy, don't try to be smart either, just do the simple obvious thing: explicitely import the names you're interested in at the top of your modules.
(been here, done that etc)
You could create modules.py containing
import subprocess
import os
Then in foo.py or any of your files just have.
from modules import *
Your import statements in your files are then static and just update modules.py when you want to add an additional module accessible to them all.
Related
I've read this post about circular imports in Python. It describes the following scenario and argues that this raises an error when run:
# module1
import module2
def function1():
module2.function2()
def function3():
print('Goodbye, World!')
# module2
import module1
def function2():
print('Hello, World!')
module1.function3()
# __init__.py
import module1
module1.function1()
But when I run this (Python 3.95), it runs perfectly fine. The post is pretty old and it doesn't specify the Python version it uses. Maybe there was some change in latter Pythons that support this?
Here's a simplified sequence of events that happen in the code in Python 3:
__init__.py starts running
An empty __main__ module is added to sys.modules
import module1 starts loading module1.py
An empty module1 module is added to sys.modules
import module2 starts loading module2.py
An empty module2 module is added to sys.modules
module2.function2 is created and added to module2.__dict__
The fact that function2 references names in module1 does not affect the creation of the function object in any way
module2 is fully loaded and execution returns to module1
module1.function1 and module1.function3 are created and added to module1.__dict__
Again, it does not matter what names the functions reference because they are not being called. AttributeError and NameError can be raised at runtime if necessary.
module1 is fully loaded and execution returns to __main__
module1.function runs successfully, since all the names it references are resolvable.
As you can see, there are no circular import issues in this particular sequence of imports because module1 and module2 do not attempt to call each other's functions. The current import system allows both modules to load before the functions are called.
The post you mention is from 2017, and must be using a version of python from before 3.0. A hint is found in the link in the following quote, which links to the python-2.x docs:
This approach doesn't contradict Python syntax, as the Python documentation says: "It is customary but not required to place all import statements at the beginning of a module (or script, for that matter)".
The paragraph after that is a bit misleading by the way:
The Python documentation also says that it is advisable to use import X, instead of other statements, such as from module import *, or from module import a,b,c.
While star imports are certainly discouraged, specific-name imports of the form from module import a,b,c are generally very much encouraged with few exceptions.
Well, I have pretty hard task and I'm completely stucked, like in any direction.
What program should do:
Import all modules (names are random) from folder
MainScript.py
modules/
mod1.py
mod2.py
mod3.py
...
Execute specific (known name, and everywhere it's same) function.
mod1.main()
mod2.main()
mod3.main()
...
As I understand it, I should list all files in folder , then make list with them and for each [x] in list import module and execute script. I've found that modules[0].main() works only if modules[0] no string, so, it should be modules[0]=main not modules[0]='main'. So and there I need somehow deal with it... but for import I don't know...
I've already googled about it, only found https://stackoverflow.com/a/1057534/10289135
And I guess it will not work for me (I also don't understand how it works and script didn't work for me)
Any ideas?
You can use the following syntax:
from filename(remove the .py) import *
This is a wild card import it imports every thing from a module literally everything .By doing this you dont need to do the work like 'filename.blabla' ,but simply you can do 'blabla'.
import os
import sys
import importlib
modules = []
for i in os.listdir("C:\\Windows\\path\\to\\your\\modules\\"):
mod = i
modules.append(mod)
sys.path.append("C:\\Windows\\path\\to\\your\\modules\\")
for i in modules:
i = i[:i.find(".")]
module = importlib.import_module(f"{i}")
module.main()
I have a Python package called Util. It includes a bunch of files. Here is the include statements on top of one of the files in there:
from config_util import ConfigUtil #ConfigUtil is a class inside the config_util module
import error_helper as eh
This works fine when I run my unit tests.
When I install Util in a virtual environment in another package everything breaks. I will need to change the import statements to
from Util.config_util import ConfigUtil
from Util import error_helper as eh
and then everything works as before. So is there any point in using the first form or is it safe to say that it is better practice to always use the second form?
If there is no point in using the first form, then why is it allowed?
Just wrong:
from config_util import ConfigUtil
import error_helper as eh
It will only work if you happen to be in the directory Util, so that the imports resolve in the current working directory. Or you have messed with sys.path using some bad hack.
Right (using absolute imports):
from Util.config_util import ConfigUtil
import Util.error_helper as eh
Also right (using relative imports):
from .config_util import ConfigUtil
import .error_helper as eh
There is no particular advantage to using relative imports, only a couple of minor things I can think of:
Saves a few bytes in the source file (so what / who cares?)
Enables you to rename the top level without editing import statements in source code (...but how often do you do that?)
For your practical problems, maybe this answer can help you.
Regarding your direct question: there's not a lot to it, but they let you move files and rename containing directories more easily. You may also prefer relative imports for stylistic reasons; I sure do.
The semantics are the same if the paths are correct. If your module is foo.bar, then from foo.bar.baz import Baz and from .baz import Baz are the same. If they don't do the same, then you're likely calling your Python file as a script (python foo/bar.py), in which case it will be module __main__ instead of foo.bar.
I might be completely wrong here, but I can't find a proper google source for the dilemma that I have:
Let's say we are using python, and we have files
foo.py and bar.py, which have the following pseudocode:
Code in foo.py:
# Code in foo.py
import sys
def foo():
# Some blah code for foo function
And code in bar.py is:
# Code in bar.py
import sys
import foo
def bar():
# Some blah code for bar function
Now, what I am wondering is : Will this not cause code bloat?, since we have imported sys twice in bar.py. Once via import sys and another time because we are doing import foo?
Additionally, what will be the correct thing to do when you have to include libraries in multiple files, which in turn will be included in other files?
Importing a module twice in python does not introduce "bloat". A second import is a mere name-lookup in a cached modules-dictionary (sys.modules, to be precise. Which in case of sys makes this even less relevant, as there is actually nothing that doesn't implicitly trigger an import of sys - although it's obviously not exposed in the namespace).
And what happens if you import some parts of a module x in foo.py, and need them and possibly others in bar.py? Having to carefully groom your imports, and then use foo.something_from_x or x.something_else_from_x in bar.py would be extremely cumbersome to write and maintain.
TLDR: don't worry. Really. Don't.
This would not cause any kind of code bloat . When you import the same library multiple times , python only actually imports it one time (the first time) , and then caches it in sys.modules , and then later on everytime you do import sys , it returns the module object from sys.modules.
A very simple example to show this -
Lets say I have an a.py -
print("In A")
This would print In A everytime the module is imported. Now lets try to import this in multiple times -
>>> import a
In A
>>> import a
>>> import a
>>> import a
As you can see the code was imported only once actually, the rest of the times the cached object was returned. To check sys.modules -
>>> import sys
>>> sys.modules['a']
<module 'a' from '\\path\to\\a.py'>
When you import a module, what happens is that python imports the code , and creates a module object and then creates a name in the local namespace with either the name of the module (if no as keyword was provided , otherwise the name provided after as keyword) , and assigns the module object to it.
The same thing happens when doing it when importing in other modules. Another example -
b.py -
import a
print("In B")
c.py -
import b
import a
print("In C")
Result of running c.py -
In A
In B
In C
As you can see , a.py was only imported once.
I have a directory structure that looks like this:
project/
__init__.py
foo/
__init.py__
first.py
second.py
third.py
plum.py
In project/foo/__init__.py I import classes from first.py, second.py and third.py and put them in __all__.
There's a class in first.py named WonderfulThing which I'd like to use in second.py, and want to import by importing * from foo. (It's outside of the scope of this question why I'd like to do so, assume I have a good reason.)
In second.py I've tried from .foo import *, from foo import * and from . import * and in none of these cases is WonderfulThing imported. I also tried from ..foo import *, which raises an error "Attempted relative import beyond toplevel package".
I've read the docs and the PEP, and I can't work out how to make this work. Any assistance would be appreciated.
Clarification/Edit: It seems like I may have been misunderstanding the way __all__ works in packages. I was using it the same as in modules,
from .first import WonderfulThing
__all__ = [ "WonderfulThing" ]
but looking at the docs again it seems to suggest that __all__ may only be used in packages to specify the names of modules to be imported by default; there doesn't seem to be any way to include anything that's not a module.
Is this correct?
A non-wildcard import failed (cannot import name WonderfulThing). Trying from . import foo failed, but import foo works. Unfortunately, dir(foo) shows nothing.
Edit: I did misunderstand the question: No __all__ is not restricted to just modules.
One question is why you want to do a relative import. There is nothing wrong with doing from project.foo import *, here. Secondly, the __all__ restriction on foo won't prevent you from doing from project.foo.first import WonderfulThing, or just from .first import WonderfulThing, which still will be the best way.
And if you really want to import a a lot of things, it's probably best to do from project import foo, and then use the things with foo.WonderfulThing instead for doing an import * and then using WonderfulThing directly.
However to answer your direct question, to import from the __init__ file in second.py you do this:
from . import WonderfulThing
or
from . import *