I currently have a package structure like that:
common/
a.py
b.py
c.py
__init__.py
.svn
util/
util.py
__init__.py
I want to write a function in util.py to retrieve all of the functions in a.py, b.py, c.py
from os import listdir
files = listdir(directory_to_common)
for f in files:
if '.py' in f and '__' not in f:
module_name = f.split(.)[0]
module = __import__(module_name)
I can successfully import the module.
Than I used
inspect.getmembers(module)
but it returns a lot of un-needed information.
From here, how do I retrieve the functions inside a.py, b.py, c.py?
#a.py, b.py, c.py has similar structure as follows:
class A:
def __init__(self):
.....
def method1_in_a(self):
.....
def method2_in_a(self):
.....
To return only needed information, use the optional argument predicate. Example to find all classes in a module and all functions in a class:
classes = inspect.getmembers(module, predicate=inspect.isclass)
for cls_name, cls in classes:
functions = inspect.getmembers(cls, predicate=inspect.isfunction)
for fn_name, fn in functions:
print(cls_name, fn_name)
For a list of all available predicates, see inspect module docs (all the functions named inspect.isXYZ are predicates).
You can try this:
import inspect
from os import listdir
files = listdir(directory_to_common)
functions = [inspect.getmembers(__import__("{}.{}".format(directory_to_common, i[:-3])), inspect.isfunction) for i in files if i.endswith('.py') and not i.startswith('__')]
Related
The objective:
I have a package with submodules that I would like to be accessible in the most straightforward way possible. The submodules contain classes to take advantage of the class structure, but don't need to be initialized (as they contain static and class methods). So, ideally, I would like to access them as follows:
from myPackage.subModule import someMethod
print (someMethod)
from myPackage import subModule
print (subModule.someMethod)
import myPackage
print(myPackage.subModule.someMethod)
Here is the package structure:
myPackage ─┐
__init__.py
subModule
subModule2
etc.
Example of a typical submodule:
# submodule.py
class SomeClass():
someAttr = list(range(10))
#classmethod
def someMethod(cls):
pass
#staticmethod
def someMethod2():
pass
Here is the code I have in my '__init __.py': In order to achieve the above; it attempts to set attributes for each class at the package level, and the same for it's methods at the sub-module level.
# __init__.py
def import_submodules(package, filetypes=('py', 'pyc', 'pyd'), ignoreStartingWith='_'):
'''Import submodules to the given package, expose any classes at the package level
and their respective class methods at submodule level.
:Parameters:
package (str)(obj) = A python package.
filetypes (str)(tuple) = Filetype extension(s) to include.
ignoreStartingWith (str)(tuple) = Ignore submodules starting with given chars.
'''
if isinstance(package, str):
package = sys.modules[package]
if not package:
return
pkg_dir = os.path.dirname(os.path.abspath(package.__file__))
sys.path.append(pkg_dir) #append this dir to the system path.
for mod_name in os.listdir(pkg_dir):
if mod_name.startswith(ignoreStartingWith):
continue
elif os.path.isfile(os.path.join(pkg_dir, mod_name)):
mod_name, *mod_ext = mod_name.rsplit('.', 1)
if filetypes:
if not mod_ext or mod_ext[0] not in filetypes:
continue
mod = importlib.import_module(mod_name)
vars(package)[mod_name] = mod
classes = inspect.getmembers(mod, inspect.isclass)
for cls_name, clss in classes:
vars(package)[cls_name] = clss
methods = inspect.getmembers(clss, inspect.isfunction)
for method_name, method in methods:
vars(mod)[method_name] = method
del mod_name
import_submodules(__name__)
At issue is this line:
vars(mod)[method_name] = method
Which ultimately results in: (indicating that the attribute was not set)
from myPackage.subModule import someMethod
ImportError: cannot import name 'someMethod' from 'myPackage.subModule'
I am able to set the methods as attributes to the module within that module, but setting them from outside (ie. in the package __init __), isn't working as written. I understand this isn't ideal to begin with, but my current logic is; that the ease of use, outweighs any perceived issues with namespace pollution. I am, of course, always open to counter-arguments.
I just checked it on my machine.
Created a package myPackage with a module subModule that has a function someMethod.
I run a python shell with working directory in the same directory that the myPackage is in, and to get these 3 import statements to work:
from myPackage.subModule import someMethod
from myPackage import subModule
import myPackage
All I had to do was to create an __init__.py with this line in it:
from . import subModule
Found a nice "hacky" solution -
subModule.py:
class myClass:
#staticmethod
def someMethod():
print("I have a bad feeling about this")
myInstance = myClass()
someMethod = myInstance.someMethod
init.py is empty
Still scratching my head of why I am unable to do this from the package __init __, but this solution works with the caveat it has to be called at the end of each submodule. Perhaps someone, in the future, someone can chime in as to why this wasn't working when completely contained in the __init __.
def addMembers(module, ignoreStartingWith='_'):
'''Expose class members at module level.
:Parameters:
module (str)(obj) = A python module.
ignoreStartingWith (str)(tuple) = Ignore class members starting with given chars.
ex. call: addMembers(__name__)
'''
if isinstance(module, str):
module = sys.modules[module]
if not module:
return
classes = inspect.getmembers(module, inspect.isclass)
for cls_name, clss in classes:
cls_members = [(o, getattr(clss, o)) for o in dir(clss) if not o.startswith(ignoreStartingWith)]
for name, mem in cls_members:
vars(module)[name] = mem
This is the solution I ended up going with. It needs to be put at the end of each submodule of your package. But, it is simple and in addition to all the standard ways of importing, allows you to import a method directly:
def __getattr__(attr):
'''Attempt to get a class attribute.
:Parameters:
attr (str): A name of a class attribute.
:Return:
(obj) The attribute.
'''
try:
return getattr(Someclass, attr)
except AttributeError as error:
raise AttributeError(f'{__file__} in __getattr__\n\t{error} ({type(attr).__name__})')
from somePackage.someModule import someMethod
I am using this approach to get the current directory of a file:
import pathlib
pathlib.Path(__file__).parent.resolve()
Is it possible to extract this string into a function, that will use the path of the caller? My attempt:
# a.py
import bb.b
print(bb.b.get_current_path(__file__)) # OK: path to a.py directory (.)
print(bb.b.get_current_path()) # WRONG: path to b.py directory (bb)
# bb/b.py
import pathlib
def get_current_path(file=__file__):
return pathlib.Path(file).parent.resolve()
Is it possible to avoid using __file__ in a.py?
You can use inspect module to inspect the stack. Though it satisfies the given condition
# bb/b.py
import inspect
import pathlib
stack = inspect.stack()
file_importer = stack[6][1]
def get_current_path(file=file_importer):
return pathlib.Path(file).parent.resolve()
I'm not sure if import will be always at the frame with the index 6. Search through code_context can be used instead:
file_importer = next(frameinfo[1] for frameinfo in stack
if frameinfo[4] and frameinfo[4][0].startswith('import'))
But this approach breaks the possibility to run b.py, so that exception handling of StopIteration is required with desired behaviour.
Both approaches return the direct importer of b.py, e.g., having a file c.py and calling it from a.py returns the path of c.py
# c.py
import bb.b
get_current_path = bb.b.get_current_path
# a.py
import bb.c
print(bb.c.get_current_path(__file__)) # OK: path to a.py directory (.)
print(bb.c.get_current_path()) # WRONG: path to c.py directory (bb)
Hence, depending on further conditions desired behaviour could be reached by processing inspect.stack().
I have dir structure like this:
Proj/
run.py
Util/
__init__.py
x.py
y.py
In x.py, I define a function:
def p():
return 1
In y.py, I define:
def q():
return 2
Currently in run.py, I'll use
from Util import *
But I have to call them using
x.p()
y.q()
But I want to call them using
p()
q()
Is there a way that I can do that? Like (as I imagine)
from Util.* import *
Bring the names up into the package namespace, by using star imports in the __init__.py:
# in util/__init__.py
from util.x import *
from util.y import *
In each submodule, define the names which you want to export by using the __all__ name:
# in x.py
__all__ = ['p']
And:
# in y.py
__all__ = ['q']
This is a pretty standard usage of the __init__.py module within a package, documented here.
I have my .py module which is in C:\Python_Projects\MyModules\ with the name button_generator.py.
My code goes something like this:
module_path='C:\\Python_Projects\\MyModules'
module_name='button_generator.py'
sys.path.append(module_path)
try:
limp=importlib.import_module(module_name.split('.')[0])
except:
print 'module import error'
I have tried other versions aswell:
importlib.import_module(module_name) without the split
importlib.import_module('C:\Python_Projects\MyModules\button_generator.py')
importlib.import_module('C:\Python_Projects\MyModules\button_generator')
The folder C:\Python_Projects\MyModules is in my sys.path as I checked during debug.
Why wouldn't the module import?
I suggest you to reorder your project directories and avoid calling other modules which are not in your current directory project. You'll avoid those kind of errors.
For example, let's organize our project directories and folders to look something like this:
MyProjectFolder/
├── main.py
└── modules
├── __init__.py
└── MyLib.py
NB: Don't forget to add an empty file called __init__.py
MyLib.py :
#!/usr/bin/python3
class MyLib:
def __init__(self):
self.say_hello = "Hello i'm in modules/MyLib"
def print_say_hello(self):
print(self.say_hello)
main.py:
#!/usr/bin/python3
# from folder.file import class
from modules.MyLib import MyLib
class MainClass:
def __init__(self):
my_lib = MyLib() # load MyLib class
my_lib.print_say_hello() # access to MyLib methods
### Test
if __name__ == '__main__':
app = MainClass()
In terminal when i run:
$ python3 main.py
output:
Hello i'm in modules/MyLib
So here we have successfully imported the class in modules/MyLib.py into our main.py file.
I found the error:
After treating the ImportError exception by printing it's args, I noticed that button_generator.py had an Import that was not resolving. Basically, button_generator.py could not be imported because it had a wrong import.
I defined a package that include a dynamically growing set of modules:
- mypackage
- __init__.py
- module1.py
- module2.py
- module3.py
... many more .py files will be added
I could expose every name in every module in __init__.py like this:
from module1 import *
from module2 import *
from module3 import *
Now when I import mypackage in client code, I get all the names defined in the sub-modules:
# suppose funcA is defined in module1, class B is defined in module2
import mypackage
mypackage.funcA() # call module1.funcA
b = mypackage.B() # module2.B
The problem is, I could define many new modules in mypackage, and I don't want to add an extra line from modulex import * to __init__.py, every time I add a new module to the package.
What is the best way to dynamically export names in all submodules?
In Python 3 the solution is straightforward using importlib and pkgutil.
Placing the following code in __init__.py is the same typing from submodule import * for all submodules (even nested ones).
import importlib
import pkgutil
for mod_info in pkgutil.walk_packages(__path__, __name__ + '.'):
mod = importlib.import_module(mod_info.name)
# Emulate `from mod import *`
try:
names = mod.__dict__['__all__']
except KeyError:
names = [k for k in mod.__dict__ if not k.startswith('_')]
globals().update({k: getattr(mod, k) for k in names})
If you only want to include immediate submodules (e.g. pkg.mod but not pkg.mod.submod), replace walk_packages with iter_modules.
I emulated from mod import * based on this answer: How to do from module import * using importlib?
I'm not sure if this is what you mean:
but if i've understood correctly - do this in your __init__.py file.
import os
__all__ = []
for module in os.listdir(os.path.dirname(__file__)):
if module != '__init__.py' and module[-3:] == '.py':
__all__.append(module[:-3])
You're adding all files in the same package into the __all__
Simply adding module names to __all__ will not always serve the purpose. I came across such an issue and also required them to be imported in addition to adding them to __all__. This is the code I came up with to make it work in my case. I didn't have any sub packages, so this code works only at the top level.
modules = glob.glob(os.path.dirname(__file__) + '/*.py')
__all__ = []
for mod in modules:
if not str(mod).endswith('__init__.py'):
package_prefix = __name__ + '.'
module_name = str(mod)[str(mod).rfind('\\') + 1:-3]
__all__.append(module_name)
__import__(package_prefix + module_name, globals(), locals(), [''])