I'm trying to dynamically build out __all__ in a __init__.py that will bring classes from many files into one namespace AND be given as suggestions in my editor. I've got the first part working. What I can't get working, though, is the ability to have my editor auto-complete discover everything that's in __all__ (I also tried updating __dir__ and defining a dir()/__dir__() method to no success).
For example, my directory tree is this:
things/
__init__.py
one.py
two.py
In __init__.py, I have code that automatically discovers classes One and Two (that are in their so-named files). However, in VS Code, when I type from things import , none of the suggestions are One or Two in the same way as when I type from things.one import , and have One suggested to me. If I manually type it all out, everything works fine, but I would really like to have the auto-complete working.
If I define __all__ with static names, VS Code auto-complete works as expected.
I've been scouring the Internet on this question to no avail and wonder if anyone has any tips or thoughts on how to accomplish it.
Here is what I have in __init__.py:
"""Import all submodules so they are available from the top level"""
import pkgutil
import sys
from importlib import import_module
from pathlib import Path
__all__ = []
def _class_name(module_name):
"""Assume camelCase class name form snake-case filename."""
return "".join([x.title() for x in module_name.split("_")])
# Loop through all modules in this directory and add to namespace
for (_, mod_name, _) in pkgutil.iter_modules([Path(__file__).parent]):
# Ensure that module isn't already loaded
if mod_name not in sys.modules:
loaded_mod = import_module("." + mod_name, package=__name__)
# Load class from imported module
name = _class_name(mod_name)
loaded_class = getattr(loaded_mod, name, None)
if not loaded_class:
continue
# Add this class to the top-level namespace
setattr(sys.modules[__name__], name, loaded_class)
__all__.append(getattr(loaded_mod, name))
Here is my directory:
This is my init.py:
When I use shortcut ctrl+space, I can get the following tips(hi is belong to a.py while hello is belong to b.py):
Related
I've got one source code which is compiled into two QGis plugins, on this model:
SourceCode -> build.py -> * MyPlugin.zip
* MyPluginExtended.zip
Since the cleanest way to import modules in QGis is to import them from the plugins directory (i.e. the parent directory), all the import statements in the MyPlugin code look like:
import MyPlugin.foo
import MyPlugin.foo.bar
from MyPlugin import foo`
But the problem is, the same import statements will be duplicated from the source code in the MyPluginExtended code, causing ModuleNotFound errors.
The QGis system prevents me to put my final code in a subdirectory that I would have then added to the sys.path, like this:
[Plugins directory] / MyPluginExtended / MyPlugin / [my files]
So the only solutions I could think of were:
1. Override the import system to dynamically replacing MyPlugin in every search path by MyPluginExtended
I tried to setup a custom importer, like proposed in the importlib documentation, doing something like this in my __init__ file, but none of my attempts worked:
try:
import MyPluginExtended
# this is the extended plugin, imports need to be overriden
class MyPathFinder():
# when imports contains 'MyPlugin', change the search path from `path/to/MyPlugin` to `path/to/MyPluginextended`
sys.meta_path.append(MyPathFinder)
except ImportError:
# this is the basic plugin, nothing to do
pass
2. Manually create aliases in sys.modules, like this:
import MyPlugin
sys.modules['MyPluginExtended'] = MyPlugin
But I would then have to do it for every submodule, and I don't knwo how to discover them programatically, something like:
for module_fullname, module in available_submodules():
module_fullname = module_fullname.replace('MyPlugin/', 'MyPluginExtended/')
sys.modules[module_fullname] = module
I am developping a program which runs on two different plattforms. Depending on which plattform I want to run it, the import directories and the names of the libraries change. For this reason i set a variable called RUN_ON_PC to True or False.
I want to implement a helper which sets the paths correctly and imports the libraries with the correct name depending of the platform and gives an interface with the same name of the libraries to the main program. The module myimporthelper is either in the "/mylib" or in the "/sd/mylib" directory. The other module names in these directories differ.
I try to do the following which is not working, since the imported modules from myimporthelper.py are not visible to main.py:
main.py:
RUN_ON_PC = True
import sys
if RUN_ON_PC:
sys.path.append("/mylib1")
else:
sys.path.append("/sd/mylib1")
import myimporthelper
myimporthelper.importall(RUN_ON_PC)
a = moduleA.ClassA() -> produces NameError: name not defined
myimporthelper.py:
import sys
def importall(run_on_pc):
if (run_on_pc == True):
sys.path.append("C:\\Users\\.....\\mylib")
import module1 as moduleA
else:
sys.path.append("/sd/mylib")
import module_a as moduleA
I want to keep the main.py short and want to outsource the platform dependent importing stuff to other module. I was not able to find a solution for this and would aprecciate any help.
Thanks a lot in advance.
You just have to qualify the name with the helper module name
a = myimporthelper.moduleA.ClassA()
But the moduleA name has to be accessible. If you import it inside a function in the helper it won't be, because of scope, unless you assign it to a name you previously declared as global in the helper module function.
I am trying to import modules dynamically in Python. Right now, I have a directory called 'modules' with two files inside; they are mod1.py and mod2.py. They are simple test functions to return time (ie. mod1.what_time('now') returns the current time).
From my main application, I can import as follows :
sys.path.append('/Users/dxg/import_test/modules')
import mod1
Then execute :
mod1.what_time('now')
and it works.
I am not always going to know what modules are available in the dirctory. I wanted to import as follows :
tree = []
tree = os.listdir('modules')
sys.path.append('/Users/dxg/import_test/modules')
for i in tree:
import i
However I get the error :
ImportError: No module named i
What am I missing?
The import instruction does not work with variable contents (as strings) (see extended explanation here), but with file names. If you want to import dynamically, you can use the importlib.import_module method:
import importlib
tree = os.listdir('modules')
...
for i in tree:
importlib.import_module(i)
Note:
You can not import from a directory where the modules are not included under Lib or the current directory like that (adding the directory to the path won't help, see previous link for why). The simplest solution would be to make this directory (modules) a package (just drop an empty __init__.py file there), and call importlib.import_module('..' + i, 'modules.subpkg') or use the __import__ method.
You might also review this question. It discusses a similar situation.
You can achieve something like what you are proposing, but it will involve some un-pythonic code. I do not recommend doing this:
dynamic_imports = dict()
for filename in tree:
name = filename.replace('.py', '')
dynamic_imports[name] = __import__(name)
I would like to be able to import a python module which is actually located in a subdirectory of another module.
I am developing a framework with plug-ins.
Since I'm expecting to have a few thousands (there's currently >250 already) and I don't want one big directory containing >1000 files I have them ordered in directories like this, where they are grouped by the first letter of their name:
framework\
__init__.py
framework.py
tools.py
plugins\
__init__.py
a\
__init__.py
atlas.py
...
b\
__init__.py
binary.py
...
c\
__init__.py
cmake.py
...
Since I would not like to impose a burden on developers of other plugins, or people not needing as many as I have, I would like to put each plugin in the 'framework.plugins' namespace.
This way someone adding a bunch of private plugins can just do so by adding them in the folder framework.plugins and there provide a __init__.py file containing:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
however, currently this setup is forcing them to also use the a-z subdirectories.
Sometimes a plugin is extending another plugin, so now I have a
from framework.plugins.a import atlas
and I would like to have
from framework.pugins import atlas
Is there any way to declare a namespace where the full name space name actually doesn't map to a folder structure?
I am aware of the pkg_resources package, but this is only available via setuptools, and I'd rather not have an extra dependency.
import pkg_resources
pkg_resources.declare_namespace(__name__)
The solution should work in python 2.4-2.7.3
update:
Combining the provided answers I tried to get a list of all plugins imported in the __init__.py from plugins. However, this fails due to dependencies. Since a plugin in the 'c' folder tries to import a plugin starting with 't', and this one has not been added yet.
plugins = [ x[0].find_module(x[1]).load_module(x[1]) for x in pkgutil.walk_packages([ os.path.join(framework.plugins.__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ],'framework.plugins.' ) ]
I'm not sure If I'm on the right track here, or just overcomplicating things and better write my own PEP302 importer. However, I can't seem to find any decent examples of how these should work.
Update:
I tried to follow the suggesting of wrapping the __getattr__ function in my __init__.py, but this seems to no avail.
import pkgutil
import os
import sys
plugins = [x[1] for x in pkgutil.walk_packages([ os.path.join(__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ] )]
import types
class MyWrapper(types.ModuleType):
def __init__(self, wrapped):
self.wrapped = wrapped
def __getattr__(self, name):
if name in plugins:
askedattr = name[0] + '.' + name
else:
askedattr = name
attr = getattr(self.wrapped, askedattr)
return attr
sys.modules[__name__] = MyWrapper(sys.modules[__name__])
A simple solution would be to import your a, b... modules in plugins.__init__, like:
from a import atlas
from b import binary
...
I'd suggest lazy-loading the modules by subclassing types.ModuleType and converting modname to modname[0] + '.' + modname on demand; this should be less work than implementing a module loader.
You can look at apipkg for an example of how to do this.
Don't use the pkgutil.extend_path function here, it tries to do the opposite of what you're trying to accomplish:
This will add to the package’s __path__ all subdirectories of
directories on sys.path named after the package.
This is useful if one wants to distribute different parts of a
single logical package as multiple directories.
Just extending __path__ with the subdirectories in your framework.plugins.__init__.py works just fine.
So the solution to this problem is: put this in your __init__.py:
__path__.extend([os.path.join(__path__[0],chr(y)) for y in range(ord('a'),ord('z')+1)])
This isn't a particularly fast solution (startup overheads), but what about having plugins.__init__ scrape the filesystem and import each found file into the local namespace?
import glob
import sys
thismodule = sys.modules[__name__]
for plugin in glob.glob("?/*"):
_temp = __import__(plugin.split("/")[0],
globals(),
locals(),
[plugin.split("/")[1]],
-1)
setattr(thismodule, plugin.split("/")[1], getattr(_temp, plugin.split("/")[1]))
In a big application I am working, several people import same modules differently e.g.
import x
or
from y import x
the side effects of that is x is imported twice and may introduce very subtle bugs, if someone is relying on global attributes
e.g. suppose I have a package mypakcage with three file mymodule.py, main.py and init.py
mymodule.py contents
l = []
class A(object): pass
main.py contents
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
it prints
updated list [1]
lets check []
updated list [1, 1]
lets check again []
because now there are two lists in two different modules, similarly class A is different
To me it looks serious enough because classes itself will be treated differently
e.g. below code prints False
def create():
from mypackage import mymodule
return mymodule.A()
def check(a):
import mymodule
return isinstance(a, mymodule.A)
print check(create())
Question:
Is there any way to avoid this? except enforcing that module should be imported one way onyl. Can't this be handled by python import mechanism, I have seen several bugs related to this in django code and elsewhere too.
Each module namespace is imported only once. Issue is, you're importing them differently. On the first you're importing from the global package, and on the second you're doing a local, non-packaged import. Python sees modules as different. The first import is internally cached as mypackage.mymodule and the second one as mymodule only.
A way to solve this is to always use absolute imports. That is, always give your module absolute import paths from the top-level package onwards:
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
from mypackage import mymodule
return mymodule.l
Remember that your entry point (the file you run, main.py) also should be outside the package. When you want the entry point code to be inside the package, usually you use a run a small script instead. Example:
runme.py, outside the package:
from mypackage.main import main
main()
And in main.py you add:
def main():
# your code
I find this document by Jp Calderone to be a great tip on how to (not) structure your python project. Following it you won't have issues. Pay attention to the bin folder - it is outside the package. I'll reproduce the entire text here:
Filesystem structure of a Python project
Do:
name the directory something
related to your project. For example,
if your project is named "Twisted",
name the top-level directory for its
source files Twisted. When you do
releases, you should include a version
number suffix: Twisted-2.5.
create a directory Twisted/bin and
put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project
is expressable as a single Python
source file, then put it into the
directory and name it something
related to your project. For example,
Twisted/twisted.py. If you need
multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and
place your source files in it. For
example,
Twisted/twisted/internet.py.
put
your unit tests in a sub-package of
your package (note - this means that
the single Python source file option
above was a trick - you always need at
least one other file for your unit
tests). For example,
Twisted/twisted/test/. Of course,
make it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory
called src or lib. This makes it
hard to run without installing.
put
your tests outside of your Python
package. This makes it hard to run the
tests against an installed version.
create a package that only has a
__init__.py and then put all your
code into __init__.py. Just make a
module instead of a package, it's
simpler.
try to come up with
magical hacks to make Python able to
import your module or package without
having the user add the directory
containing it to their import path
(either via PYTHONPATH or some other
mechanism). You will not correctly
handle all cases and users will get
angry at you when your software
doesn't work in their environment.
I can only replicate this if main.py is the file you are actually running. In that case you will get the current directory of main.py on the sys path. But you apparently also have a system path set so that mypackage can be imported.
Python will in that situation not realize that mymodule and mypackage.mymodule is the same module, and you get this effect. This change illustrates this:
def add(x):
from mypackage import mymodule
print "mypackage.mymodule path", mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
print "mymodule path", mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
$ export PYTHONPATH=.
$ python mypackage/main.py
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mymodule' from '/tmp/mypackage/mymodule.pyc'>
But add another mainfile, in the currect directory:
realmain.py:
from mypackage import main
and the result is different:
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
So I suspect that you have your main python file within the package. And in that case the solution is to not do that. :-)