Develop Reference

Write Module with decorated functions and imports

i have this code in a python file:
from dec import my_decorator
import asyncio
#my_decorator
async def simple_method(bar): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
#my_decorator
async def other_simple_meth(bar, value):
print("Henlo from other_simple_meth:\t Val:{}".format(value))
return
async def main():
print("Start Module-Export")
open('module_functions.py', 'a').close()
# Write all decorated functions to modue_functions.py
print("Functions in module_functions.py exported")
while True:
asyncio.sleep(2)
print("z...z...Z...")
My goal is to write all decorated functions (inc. the import dependencies) into a second module file (here "module_functions.py"). My 'module_functions.py' file should look like this:
from dec import my_decorator
import asyncio
#my_decorator
async def simple_method(bar): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
#my_decorator
async def other_simple_meth(bar, value):
print("Henlo from other_simple_meth:\t Val:{}".format(value))
return
I know how to get references and names of a function, but not how to "copy/paste" the functioncode (incl. decorator and all dependencies) into a seperated file. Is this even possible?
EDIT: I know that pickle and dill exist, but this may not fullfill the goal. The problem is, that someone else may not know the order of the dumped file and loading them back may/will cause problem. As well it seems to be not possible to edit such loaded functions again.

I found a (not ideal, but ok) solution for my problems.
I) Find and write functions, coroutines etc. into a file (works):
Like #MisterMiyagi suspected, is the inspect module a good way to go. For the common stuff, it is possible with inspect.getsource() to get the code and write them into a file:
# List of wanted stuff
func_list = [simple_method, meth_with_input, meth_with_input_and_output, func_myself]
with open('module_functions.py', 'a') as module_file:
for func in func_list:
try:
module_file.write(inspect.getsource(func))
module_file.write("\n")
except:
print("Error :( ")
II) But what about decorated stuff(seems to work)?
I) will not work for decorated stuff, it is just ignored without throwing an exception. What seems to be used is from functools import wraps.
In many examples the #wraps decorator is added into the decorator class. This was not possible for me, but there is a good workaround:
#wraps(lambda: simple_method) #<---add wraps-decorator here
#my_decorator
async def simple_method(parent): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
Wraps can be placed above the original decorated method/class/function and it seems to behave like I want. Now we can add simple_methodinto the func_listof I).
III) What about the imports?
Well it seems to be quite tricky/impossible to actually read the dependencies of a function. My workaround is to drop all wanted imports into a class (sigh). This class can be throw into the func_listof I) and is written into the file.
EDIT:
There is a cleaner way, which may works, after some modification, with I) and II) as well. The magic module is ast.
I have overwritten following:
class ImportVisitor(ast.NodeVisitor):
def __init__(self, target):
super().__init__()
self.file_target = target
"pick these special nodes via overwriting: visit_classname." \
"classnames are listed in https://docs.python.org/3.6/library/ast.html#abstract-grammar"
def visit_Import(self, node):
"Overwrite func!"
"Write all statements just with import like - import ast into file_target"
str = 'import '+', '.join(alias.name for alias in node.names)
self.file_target.write(str+"\n")
def visit_ImportFrom(self, node):
"Overwrite func!"
"Write all statements with from ... import (like - from os.path import basename) into file_tagrget"
str = 'from '+ node.module+ ' import '+', '.join(alias.name for alias in node.names)
self.file_target.write(str+"\n")
Now I can parse my own script name and fill the module_file with the imports and from...imports it will find while visiting all nodes in this tree:
with open('module_functions.py', 'a') as module_file:
with open(basename(__file__), "rb") as f:
tree = ast.parse(f.read(), basename(__file__))
visitor = ImportVisitor(module_file)
visitor.visit(tree)
module_file.write("\n\n")

Aliases for commands with Python cmd module

How can I create an alias for a command in a line-oriented command interpreter implemented using the cmd module?
To create a command, I must implement the do_cmd method. But I have commands with long names (like constraint) and I want to provide aliases (in fact, shortcuts) for these commands (like co). How can I do that?
One possibility that came to my mind is to implement the do_alias (like do_co) method and just calling do_cmd (do_constraint) in this method. But this brings me new commands in the help of the CLI.
Is there any other way to achieve this? Or may be is there a way to hide commands from the help output?

The following solution makes aliased commands share a single help message. It lets you keep all of your aliases in a single easy to edit place, while making documentation much easier. It checks user input against an alias dictionary with function values and overrides both the default() (See sloth & brian) and do_help() methods.
Here I've made aliases 'c' and 'con' execute do_constraint(), 'q' invoke do_quit(), and 'h' invoke do_help(). The bonus of this solution is that 'h q' and 'help quit' print the same message. Documentation for several aliased commands can maintained in a single docstring.
import cmd
class prompt(cmd.Cmd):
def __init__(self):
cmd.Cmd.__init__(self)
self.aliases = { 'c' : self.do_constraint ,
'con' : self.do_constraint ,
'q' : self.do_quit ,
'h' : self.do_help }
def do_constraint(self, arg):
'''Constrain things.'''
print('Constraint complete.')
def do_quit(self, arg):
'''Exit the program.'''
return True
def do_help(self, arg):
'''List available commands.'''
if arg in self.aliases:
arg = self.aliases[arg].__name__[3:]
cmd.Cmd.do_help(self, arg)
def default(self, line):
cmd, arg, line = self.parseline(line)
if cmd in self.aliases:
self.aliases[cmd](arg)
else:
print("*** Unknown syntax: %s" % line)

You can overwrite the default method and search for a suitable command handler (as already suggested by Brian):
import cmd
class WithAliasCmd(cmd.Cmd):
def default(self, line):
cmd, arg, line = self.parseline(line)
func = [getattr(self, n) for n in self.get_names() if n.startswith('do_' + cmd)]
if func: # maybe check if exactly one or more elements, and tell the user
func[0](arg)

The docs mention a default method, which you can override to handle any unknown command. Code it to prefix scan a list of commands and invoke them as you suggest for do_alias.

Grep-ing classes / functions from projects

Is there a way in grep (or vim) to print out a named function/class?
i.e. From:
class InternalTimer(Sim.Process):
def __init__(self, fsm):
Sim.Process.__init__(self, name="Timer")
random.seed()
self.fsm = fsm
def Lifecycle(self, Request):
while True:
yield Sim.waitevent, self, Request
yield Sim.hold, self, Request.signalparam[0]
if(self.interrupted()):
self.interruptReset()
else:
self.fsm.process(Request.signalparam[1])
Calling $my-func-grep '__init__(self,fsm)' filename.py would produce
def __init__(self, fsm):
Sim.Process.__init__(self, name="Timer")
random.seed()
self.fsm = fsm

You could create a vim extension which effectively performs the following:
import inspect
print inspect.getsource(name_of_function)
This prints the function signature and the body of the function. If Vim has been compiled with Python support, you can write extensions in Python itself.

Python cmd module - parsing values from line

I am working on a quick python script using the cmd module that will allow the user to enter text commands followed by parameters in basic url query string format. The prompts will be answered with something like
commandname foo=bar&baz=brack
Using cmd, I can't seem to find which method to override to affect the way the argument line is handed off to all the do_* methods. I want to run urlparse.parse_qs on these values, and calling this upon line in every do_* method seems clumsy.
The precmd method gets the whole line, before the commandname is split off and interpreted, so this will not work for my purposes. I'm also not terribly familiar with how to place a decorator inside a class like this and haven't been able to pull it off without breaking the scope.
Basically, the python docs for cmd say the following
Repeatedly issue a prompt, accept input, parse an initial prefix off
the received input, and dispatch to action methods, passing them the
remainder of the line as argument.
I want to make a method that will do additional processing to that "remainder of the line" and hand that generated dictionary off to the member functions as the line argument, rather than interpreting them in every function.
Thanks!

You could potentially override the onecmd() method, as the following quick example shows. The onecmd() method there is basically a copy of the one from the original cmd.py, but adds a call to urlparse.parse_qs() before passing the arguments to a function.
import cmd
import urlparse
class myCmd(cmd.Cmd):
def onecmd(self, line):
"""Mostly ripped from Python's cmd.py"""
cmd, arg, line = self.parseline(line)
arg = urlparse.parse_qs(arg) # <- added line
if not line:
return self.emptyline()
if cmd is None:
return self.default(line)
self.lastcmd = line
if cmd == '':
return self.default(line)
else:
try:
func = getattr(self, 'do_' + cmd)
except AttributeError:
return self.default(line)
return func(arg)
def do_foo(self, arg)
print arg
my_cmd = myCmd()
my_cmd.cmdloop()
Sample output:
(Cmd) foo
{}
(Cmd) foo a b c
{}
(Cmd) foo a=b&c=d
{'a': ['b'], 'c': ['d']}
Is this what you are trying to achieve?
Here's another potential solution that uses a class decorator to modify a
cmd.Cmd subclass and basically apply a decorator function to all do_*
methods of that class:
import cmd
import urlparse
import types
# function decorator to add parse_qs to individual functions
def parse_qs_f(f):
def f2(self, arg):
return f(self, urlparse.parse_qs(arg))
return f2
# class decorator to iterate over all attributes of a class and apply
# the parse_qs_f decorator to all do_* methods
def parse_qs(cls):
for attr_name in dir(cls):
attr = getattr(cls, attr_name)
if attr_name.startswith('do_') and type(attr) == types.MethodType:
setattr(cls, attr_name, parse_qs_f(attr))
return cls
#parse_qs
class myCmd(cmd.Cmd):
def do_foo(self, args):
print args
my_cmd = myCmd()
my_cmd.cmdloop()
I quickly cobbled this together and it appears to work as intended, however, I'm
open to suggestions on any pitfalls or how this solution could be improved.

How can I get a list of all classes within current module in Python?

I've seen plenty of examples of people extracting all of the classes from a module, usually something like:
# foo.py
class Foo:
pass
# test.py
import inspect
import foo
for name, obj in inspect.getmembers(foo):
if inspect.isclass(obj):
print obj
Awesome.
But I can't find out how to get all of the classes from the current module.
# foo.py
import inspect
class Foo:
pass
def print_classes():
for name, obj in inspect.getmembers(???): # what do I do here?
if inspect.isclass(obj):
print obj
# test.py
import foo
foo.print_classes()
This is probably something really obvious, but I haven't been able to find anything. Can anyone help me out?

Try this:
import sys
current_module = sys.modules[__name__]
In your context:
import sys, inspect
def print_classes():
for name, obj in inspect.getmembers(sys.modules[__name__]):
if inspect.isclass(obj):
print(obj)
And even better:
clsmembers = inspect.getmembers(sys.modules[__name__], inspect.isclass)
Because inspect.getmembers() takes a predicate.

I don't know if there's a 'proper' way to do it, but your snippet is on the right track: just add import foo to foo.py, do inspect.getmembers(foo), and it should work fine.

What about
g = globals().copy()
for name, obj in g.iteritems():
?

I was able to get all I needed from the dir built in plus getattr.
# Works on pretty much everything, but be mindful that
# you get lists of strings back
print dir(myproject)
print dir(myproject.mymodule)
print dir(myproject.mymodule.myfile)
print dir(myproject.mymodule.myfile.myclass)
# But, the string names can be resolved with getattr, (as seen below)
Though, it does come out looking like a hairball:
def list_supported_platforms():
"""
List supported platforms (to match sys.platform)
#Retirms:
list str: platform names
"""
return list(itertools.chain(
*list(
# Get the class's constant
getattr(
# Get the module's first class, which we wrote
getattr(
# Get the module
getattr(platforms, item),
dir(
getattr(platforms, item)
)[0]
),
'SYS_PLATFORMS'
)
# For each include in platforms/__init__.py
for item in dir(platforms)
# Ignore magic, ourselves (index.py) and a base class.
if not item.startswith('__') and item not in ['index', 'base']
)
))

import pyclbr
print(pyclbr.readmodule(__name__).keys())
Note that the stdlib's Python class browser module uses static source analysis, so it only works for modules that are backed by a real .py file.

If you want to have all the classes, that belong to the current module, you could use this :
import sys, inspect
def print_classes():
is_class_member = lambda member: inspect.isclass(member) and member.__module__ == __name__
clsmembers = inspect.getmembers(sys.modules[__name__], is_class_member)
If you use Nadia's answer and you were importing other classes on your module, that classes will be being imported too.
So that's why member.__module__ == __name__ is being added to the predicate used on is_class_member. This statement checks that the class really belongs to the module.
A predicate is a function (callable), that returns a boolean value.

This is the line that I use to get all of the classes that have been defined in the current module (ie not imported). It's a little long according to PEP-8 but you can change it as you see fit.
import sys
import inspect
classes = [name for name, obj in inspect.getmembers(sys.modules[__name__], inspect.isclass)
if obj.__module__ is __name__]
This gives you a list of the class names. If you want the class objects themselves just keep obj instead.
classes = [obj for name, obj in inspect.getmembers(sys.modules[__name__], inspect.isclass)
if obj.__module__ is __name__]
This is has been more useful in my experience.

Another solution which works in Python 2 and 3:
#foo.py
import sys
class Foo(object):
pass
def print_classes():
current_module = sys.modules[__name__]
for key in dir(current_module):
if isinstance( getattr(current_module, key), type ):
print(key)
# test.py
import foo
foo.print_classes()

I think that you can do something like this.
class custom(object):
__custom__ = True
class Alpha(custom):
something = 3
def GetClasses():
return [x for x in globals() if hasattr(globals()[str(x)], '__custom__')]
print(GetClasses())`
if you need own classes

I frequently find myself writing command line utilities wherein the first argument is meant to refer to one of many different classes. For example ./something.py feature command —-arguments, where Feature is a class and command is a method on that class. Here's a base class that makes this easy.
The assumption is that this base class resides in a directory alongside all of its subclasses. You can then call ArgBaseClass(foo = bar).load_subclasses() which will return a dictionary. For example, if the directory looks like this:
arg_base_class.py
feature.py
Assuming feature.py implements class Feature(ArgBaseClass), then the above invocation of load_subclasses will return { 'feature' : <Feature object> }. The same kwargs (foo = bar) will be passed into the Feature class.
#!/usr/bin/env python3
import os, pkgutil, importlib, inspect
class ArgBaseClass():
# Assign all keyword arguments as properties on self, and keep the kwargs for later.
def __init__(self, **kwargs):
self._kwargs = kwargs
for (k, v) in kwargs.items():
setattr(self, k, v)
ms = inspect.getmembers(self, predicate=inspect.ismethod)
self.methods = dict([(n, m) for (n, m) in ms if not n.startswith('_')])
# Add the names of the methods to a parser object.
def _parse_arguments(self, parser):
parser.add_argument('method', choices=list(self.methods))
return parser
# Instantiate one of each of the subclasses of this class.
def load_subclasses(self):
module_dir = os.path.dirname(__file__)
module_name = os.path.basename(os.path.normpath(module_dir))
parent_class = self.__class__
modules = {}
# Load all the modules it the package:
for (module_loader, name, ispkg) in pkgutil.iter_modules([module_dir]):
modules[name] = importlib.import_module('.' + name, module_name)
# Instantiate one of each class, passing the keyword arguments.
ret = {}
for cls in parent_class.__subclasses__():
path = cls.__module__.split('.')
ret[path[-1]] = cls(**self._kwargs)
return ret

import Foo
dir(Foo)
import collections
dir(collections)

The following can be placed at the top of the file:
def get_classes():
import inspect, sys
return dict(inspect.getmembers(
sys.modules[__name__],
lambda member: inspect.isclass(member) and member.__module__ == __name__
))
Note, this can be placed at the top of the module because we've wrapped the logic in a function definition. If you want the dictionary to exist as a top-level object you will need to place the definition at the bottom of the file to ensure all classes are included.

Go to Python Interpreter. type help ('module_name') , then press Enter.
e.g. help('os') .
Here, I've pasted one part of the output below:
class statvfs_result(__builtin__.object)
| statvfs_result: Result from statvfs or fstatvfs.
|
| This object may be accessed either as a tuple of
| (bsize, frsize, blocks, bfree, bavail, files, ffree, favail, flag, namemax),
| or via the attributes f_bsize, f_frsize, f_blocks, f_bfree, and so on.
|
| See os.statvfs for more information.
|
| Methods defined here:
|
| __add__(...)
| x.__add__(y) <==> x+y
|
| __contains__(...)
| x.__contains__(y) <==> y in x