I was just wondering why __import__() calls a __init__ module twice when loading a package.
test.py
testpkg/
__init__.py
test.py:
pkg = __import__("testpkg", fromlist=[''])
__init__.py:
print "Called."
After calling python test.py, Called. will be printed out twice. Why does python execute the __init__ "module" twice?
This is a Python bug. Passing the null string as an element of fromlist is illegal, and should raise an exception.
There's no need to include "" in fromlist; that's implicit--the module itself is always loaded. What's actually happening is the module.submodule string is using the null string, resulting in the module name testpkg., with a trailing period. That gets imported literally, and since it has a different name than testpkg, it's imported as a separate module.
Try this:
pkg = __import__("testpkg", fromlist=[''])
import sys
print sys["testpkg"]
print sys["testpkg."]
... and you'll see the duplicate module.
Someone should probably file a ticket on this if there isn't already one; too tired to do it myself right now.
Using the fromlist=[''] hack to import a specific module is explicitly frowned upon by python-dev. While it has been filed as an issue, the chances of it being fixed are low specifically because this is viewed as a mis-use of fromlist to begin with instead of necessarily a bug and a better solution is available.
What you should be doing is using importlib.import_module (available in the standard library for Python 2.7 and Python 3.1, or from PyPI with compatibility back to Python 2.3 along with being included in Django since 1.1 as django.utils.importlib). It will prevent this problem from happening, provides a better programmatic interface for importing modules, and even lets you use relative imports when you specify the package you are importing from.
If you really cannot use importlib (e.g., PyPI dependencies are not allowed even though the code you can freely copy thanks to the PSF license and it being rather short), then you should be doing __import__("some.module"); mod = sys.modules["some.module"]. That is the official, python-dev sanctioned solution to the problem (but only after you cannot use importlib).
Related
I would like to import a module from inside a functions. For example from this:
from directory.folder.module import module
def import():
app.register_blueprint(module)
To this:
def import():
from directory.folder.module import module
But, without hardcoding it. For example:
def import():
m = "module"
from directory.folder.m import m
Is it possible? Thanks in advance
You want the importlib module.
Here's the most simplistic way to use this module. There are lots of different ways of weaving the results of calls to the module into the environment:
import importlib
math = importlib.import_module("math")
print(math.cos(math.pi))
Result:
-1.0
I've used this library a lot. I built a whole plug-in deployment system with it. Scripts for all the various deploys were dropped in directories and only imported when they were mentioned in a config file rather than everything having to be imported right away.
Something I find very cool about this module is what's stated at the very top of its documentation:
The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the import() function) in Python source code.
The intro in the 2.7 docs is interesting as well:
New in version 2.7.
This module is a minor subset of what is available in the more full-featured package of the same name from Python 3.1 that provides a complete implementation of import. What is here has been provided to help ease in transitioning from 2.7 to 3.1.
No, python import does not work this way.
Such as an example you try to import a module named mod, so you run import mod. Now interpreter will search for mod.py in a list of directories gathered from the following sources:
The directory from where the input script was run or the current directory if the interpreter is being run interactively.
The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATH environment variable.)
An installation-dependent list of directories configured at the time Python is installed.
So if you have a variable named m='mod' and run import m it will search for m.py not mod.py.
But just a silly dangerous workaround is to use exec() (WARNING FOR MALICIOUS INPUT)
m = "module"
exec(f'from directory.folder.m import {m}')
If you don't mind external modules try importlib.
You can use the importlib module to programmatically import modules.
import importlib
full_name = "package." + "module"
m = importlib.import_module(full_name)
I am working with a project that has a user-written module called types.py buried in a second-level package (its path from the project root is package/subpackage/types.py).
This is causing problems because the Python library also has a types module. When enum.py, another Python library module, attempts to import types, the user-written version is imported instead, wreaking havoc.
What's puzzling me is that the import inside enum.py does not qualify types with any package names:
# line 10 of enum.py:
from types import MappingProxyType, DynamicClassAttribute
so why is Python selecting the user-written types which is in a two-level subpackage? It seems to me the user-written types would only be imported if one uses
# what I expect an 'import' would have to be like to access the user-written types.py
from package.subpackage.types import ...
Another possible explanation would be that sys.path contained the package/subpackage directory, but this is not the case when I print its content right before the enum.py import:
enum.py: Path:
/home/me/PycharmProjects/myproject
/home/me/anaconda3/envs/myproject/lib/python37.zip
/home/me/anaconda3/envs/myproject/lib/python3.7
/home/me/anaconda3/envs/myproject/lib/python3.7/lib-dynload
/home/me/anaconda3/envs/myproject/lib/python3.7/site-packages
So, how can the importing of the user-written types.py module be explained?
UPDATE: the first comment suggests this happens because my project's path is the first item in sys.path. However, I set up a really simple project in which a module called mymodule is in package.subpackage:
Importing from mymodule without using the package and subpackage names does not work:
# main.py
# Works:
from package.subpackage.mymodule import my_module_field
# Does not work:
# from mymodule import my_module_field
So I still do not understand why the from types import in enum.py can work find the user-written types.py without the packages names.
UPDATE 2: printing out more information, I see that when I print sys.path as soon as enum.py starts (I modified the standard library file to print it), I see that the package/subpackage directory is in sys.path, even though it was not at the beginning of execution. So this explains why the user-written typos.py is being used.
The issue now is why sys.path is appended with the package/subpackage directory. I searched all occurrences of sys.path in my code and even though the current directory is appended to it at some points, it is never the package/subpackage directory. Where can this be happening?
Not sure this counts as a real answer because it would not be possible to answer it based on the question information by itself (and adding all the details to the question is impractical). In any case, here's the solution.
Basically, upon greater inspection I found out that a script invokes another script as an external process, and this latter script is in the package/subpackage directory, which is added to sys.path in the new process. About this last point, I'm not sure why; I am assuming that a script's current directory is always added to sys.path.
I'm using a from . import module statement to do exactly that: import a local module to my script. The script and module reside in the same folder.
# module.py
def foo():
print('Foo!')
# script.py
from . import module
module.foo()
> ImportError: cannot import name 'module'
This should be pretty easy, and doing just import module does work, but as this answer suggests one should, I modified the statements to the former form.
The end goal is to have a package, from which I can use things, but also to have executable scripts inside the package that import other parts of that package. Apparently, after a few days worth of searching and a few questions I still don't quite understand the import and packaging machinery.
These might be the cause:
Import statements are different in 2.7 and 3.x, I'm using 3.6, the question was on 2.7
Relative imports are different inside packages (folder with __init__.py)
The working directory is different or the folders are not in sys.path
Having an __init__ file does not make a difference at least in a fresh project in PyCharm. Also, the working directory is set to the folder of the sources and it is in path.
Have I missed something? Or rather, what's the correct way of achieving the functionality described in the end goal? Any help is greatly appreciated!
Since writing this answer I have realised it is more convenient and better style in my humble opinion to install the package with pip install -e . and use absolute imports. So even within a package writing from package.sub.module import thing. This makes refactoring a lot easier and there's no need to ever manipulate module variables or sys.path.
When running a script directly, Python consideres the name (a special variable, __name__) of that script to be "__main__". In case of an import, the name is set to the name of the module. In the latter case relative imports are fine. But import actually looks at the combination of __name__ and another special variable, __package__, which is None for an executed script, but the path to a module for an imported module, e.g. parent.sub.
The searched variable is... drumroll...
__package__ + '.' + __name__
The secret ingredient is manipulating __package__:
# script.py
__package__ = 'package_name' # or parent.sub.package
from . import module
This lets Python know you are inside a package even though the script is executed directly. However, the top level folder needs to be in sys.path and the package name has to reflect that directory structure.
See this very comprehensive answer on the topic of relative imports.
If I understand correctly, the python syntax from ... import ... can be used in two ways
from package-name import module-name
from module-name import function-name
I would like to know a bit of how Python internally treats the two different forms. Imagine, for example, that the interpreter gets "from A import B", does the interpreter actually try to determine whether A is a package-name/ module-name, or does it internally treat packages and modules as the same class of objects (something like Linux treats files and directories very similarly)?
First of all, a module is a python file that contains classes and functions. when you say From A Import B python searches for A(a module) in the standard python library and then imports B(the function or class) which is the module if it finds A. If it doesn't it goes out and starts searching in the directory were packages are stored and searches for the package name( A ) and then if it finds it, it imports the Module name(B). If it fails in the past 2 processes it returns an error.
Hope this helps.
I am using Python3.3 and I have installed a package called python-bidi, which can be found here.
I have installed the package by running: python setup.py install, which was instructed in the package documentation.
Now I am running a code that uses that package but I get this error:
File "C:\Python33\lib\site-packages\python_bidi-0.3.4-py3.3.egg\bidi\algorithm.py", line 24, in <module>
from mirror import MIRRORED
ImportError: No module named 'mirror'
However, looking at this folder: C:\Python33\lib\site-packages\python_bidi-0.3.4-py3.3.egg\bidi I can see that there is a file called mirror.py.
What is the reason for the error that I get?
Thanks for any help on this.
In Python 2.7, you can do import bidi.algorithm, but it will fail in Python 3.3. Actually, it comes from changes in the way modules are imported. Notice the python-bidi library was last updated in 2010.
See PEP 328, which also dates back 2010. This PEP addresses the following issue: "Imports can be ambiguous in the face of packages; within a package, it's not clear whether import foo refers to a module within the package or some module outside the package. (More precisely, a local module or package can shadow another hanging directly off sys.path.)"
With the new notation, that is used in Python 3.x, you would write, in "bidi/algorithm.py", from .mirror import MIRRORED instead of from mirror import MIRRORED. Notice the dot!
The import would work, but the module bidi.algorithm would still fail to load, because of other incompatibilities. Namely, there is at some point X2_X5_MAPPINGS.keys() + ['BN', 'PDF', 'B'], which fails because dict.keys() does not return a list any longer, but a "dict_keys" object.
Therefore, you won't be able to use this package in Python 3.x, unless you are willing to do some work to resolve incompatibilities.
As a side note, the new notation from .mirror import MIRRORED is also understood by Python 2.7, so it should be possible to make python-bidi to be compatible with both Python 2.x and 3.x lines.
The only thing left is to change line 41 and 42 by adding list around X2_X5_MAPPINGS.keys() and X2_X5_MAPPINGS.keys() to become:
list(X2_X5_MAPPINGS.keys())
and
list(X2_X5_MAPPINGS.keys())
and also change line 600 from:
if isinstance(unicode_or_str, unicode)
to
if isinstance(unicode_or_str, str):