If I have this function, what should I do to replace the inner function with my own custom version?
def foo():
def bar():
# I want to change this
pass
# here starts a long list of functions I want to keep unchanged
def baz():
pass
Using classes this would be easily done overriding the method. Though, I can't figure out how to do that with nested functions. Changing foo to be a class (or anything else) is not an option because it comes from a given imported module I can't modify.
Here's one way of doing it, creating a new foo that "does the right thing" by hacking the function internals. ( As mentioned by #DSM ). Unfortunately we cant just jump into the foo function and mess with its internals, as they're mostly marked read only, so what we have to do is modify a copy we construct by hand.
# Here's the original function
def foo():
def bar():
print(" In bar orig")
def baz():
print(" Calling bar from baz")
bar()
print("Foo calling bar:")
bar()
print("Foo calling baz:")
baz()
# Here's using it
foo()
# Now lets override the bar function
import types
# This is our replacement function
def my_bar():
print(" Woo hoo I'm the bar override")
# This creates a new code object used by our new foo function
# based on the old foo functions code object.
foocode = types.CodeType(
foo.func_code.co_argcount,
foo.func_code.co_nlocals,
foo.func_code.co_stacksize,
foo.func_code.co_flags,
foo.func_code.co_code,
# This tuple is a new version of foo.func_code.co_consts
# NOTE: Don't get this wrong or you will crash python.
(
foo.func_code.co_consts[0],
my_bar.func_code,
foo.func_code.co_consts[2],
foo.func_code.co_consts[3],
foo.func_code.co_consts[4]
),
foo.func_code.co_names,
foo.func_code.co_varnames,
foo.func_code.co_filename,
foo.func_code.co_name,
foo.func_code.co_firstlineno,
foo.func_code.co_lnotab,
foo.func_code.co_freevars,
foo.func_code.co_cellvars )
# This is the new function we're replacing foo with
# using our new code.
foo = types.FunctionType( foocode , {})
# Now use it
foo()
I'm pretty sure its not going to catch all cases. But it works for the example (for me on an old python 2.5.1 )
Ugly bits that could do with some tidy up are:
The huge argument list being passed to CodeType
The ugly tuple constructed from co_consts overriding only one member. All the info is in co_consts to determine which to replace - so a smarter function could do this. I dug into the internals by hand using print( foo.func_code.co_consts ).
You can find some information about the CodeType and FunctionType by using the interpreter
command help( types.CodeType ).
UPDATE:
I thought this was too ugly so I built a helper function to make it prettier. With the helper you can write:
# Use our function to get a new version of foo with "bar" replaced by mybar
foo = monkey_patch_fn( foo, "bar", my_bar )
# Check it works
foo()
Here's the implementation of monkey_patch_fn:
# Returns a copy of original_fn with its internal function
# called name replaced with new_fn.
def monkey_patch_fn( original_fn, name, new_fn ):
#Little helper function to pick out the correct constant
def fix_consts(x):
if x==None: return None
try:
if x.co_name == name:
return new_fn.func_code
except AttributeError, e:
pass
return x
original_code = original_fn.func_code
new_consts = tuple( map( fix_consts, original_code.co_consts ) )
code_type_args = [
"co_argcount", "co_nlocals", "co_stacksize", "co_flags", "co_code",
"co_consts", "co_names", "co_varnames", "co_filename", "co_name",
"co_firstlineno", "co_lnotab", "co_freevars", "co_cellvars" ]
new_code = types.CodeType(
*[ ( getattr(original_code,x) if x!="co_consts" else new_consts )
for x in code_type_args ] )
return types.FunctionType( new_code, {} )
You can pass it in as an optional parameter
def foo(bar=None):
def _bar():
# I want to change this
pass
if bar is None:
bar = _bar
Related
I am just wondering what the actual explanation is as to why this is valid python.
foo.py
class Foo:
def __init__(self):
if type(self).__name__ in MAP.keys():
print('WOO HOOO')
self.Bar = True
MAP = {'Foo': Foo}
test
>>> from foo import *
>>> x = Foo()
WOO HOOO
I would have thought that the MAP dict when referenced in the constructor would be invalid.
Is this due to the fact that the constructor is only checked at run-time?
You're right, MAP is only looked up when __init__ is run.
If it helps you understand, let's strip away all the non-essential info:
It's irrelevant that __init__ is a method. You can get the same behaviour with a function.
The value of MAP is not important; we just want to check whether it exists
Let's also see what happens when MAP doesn't exist, using a try-except-else.
Put everything in one script
self.Bar and x are unused
def foo():
try:
MAP
except NameError:
print('No!')
else:
print('Yes!')
foo() # -> No!
MAP = 0
foo() # -> Yes!
I'm developing a bot on Python (2.7, 3.4). I defined a about 30+ dynamic functions which to be used based on bot commands. While development, since not all functions are done, I have to define for them an empty functions (if I not define then code won't run) like this:
def c_about():
return
def c_events():
return
def c_currentlocation():
return
etc. many dummy functions.
Question:
it is somehow possible in Python to define same function but with multiple names?
Something like this:
def c_about(), c_events(), c_currentlocation():
return
Yes, it's perfectly possible since defined functions are stored in variables like everything else.
def foo():
pass
baz = bar = foo
There is still some metadata relating to the original function (help(bar) will still mention foo), but it doesn't affect functionality.
Another option is to use lambdas for one-liners:
foo = bar = baz = lambda: None
Functions do not intern in Python (i.e., automatically share multiple references to the same immutable object), but can share the same name:
>>> def a(): pass
...
>>> a
<function a at 0x101c892a8>
>>> def b(): pass
...
>>> b
<function b at 0x101c89320>
>>> c=a
>>> c
<function a at 0x101c892a8> # note the physical address is the same as 'a'
So clearly you can do:
>>> c=d=e=f=g=a
>>> e
<function a at 0x101c892a8>
For the case of functions not yet defined, you can use a try/catch block by catching either a NameError:
def default():
print "default called"
try:
not_defined()
except NameError:
default()
Or use a dict of funcs and catch the KeyError:
funcs={"default": default}
try:
funcs['not_defined']()
except KeyError:
funcs['default']()
Or, you can do funcs.get(not_defined, default)() if you prefer that syntax with a dict of funcs.
Yes, it is possible, and it is even possible to store them in lists using loops. For instance:
l = []
for i in range(10):
l.append(lambda: None)
And you can reference any of them through indices like l[index]()
For example:
c_events = 0
c_about = 1
l[c_events]()
I am trying to produce a better answer to the frequently-asked question "How do I do function-local static variables in Python?" (1, 2, 3, ...) "Better" means completely encapsulated in a decorator, that can be used in any context where a function definition may appear. In particular, it must DTRT when applied to methods and nested functions; it must play nice with other decorators applied to the same function (in any order); it must accept arbitrary initializers for the static variables, and it must not modify the formal parameter list of the decorated function. Basically, if this were to be proposed for inclusion in the standard library, nobody should be able to object on quality-of-implementation grounds.
Ideal surface syntax would be
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
I would also accept
#static_vars(a=0, b=[])
def test():
static.b.append(static.a)
static.a += 1
sys.stdout.write(repr(static.b) + "\n")
or similar, as long as the namespace for the static variables is not the name of the function! (I intend to use this in functions that may have very long names.)
A slightly more motivated example involves precompiled regular expressions that are only relevant to one function:
#static_vars(encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
Now, I already have a mostly-working implementation. The decorator rewrites each function definition as if it had read like so (using the first example):
def _wrap_test_():
a = 0
b = 1
def test():
nonlocal a, b
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test = _wrap_test_()
del _wrap_test_
It seems that the only way to accomplish this is to munge the AST. I have code that works for simple cases (see below) but I strongly suspect it is wrong in more complicated cases. For instance, I think it will break if applied to a method definition, and of course it also breaks in any situation where inspect.getsource() fails.
So the question is, first, what should I do to make it work in more cases, and second, is there a better way to define a decorator with the same black-box effects?
Note 1: I only care about Python 3.
Note 2: Please assume that I have read all of the proposed solutions in all of the linked questions and found all of them inadequate.
#! /usr/bin/python3
import ast
import functools
import inspect
import textwrap
def function_skeleton(name, args):
"""Return the AST of a function definition for a function named NAME,
which takes keyword-only args ARGS, and does nothing. Its
.body field is guaranteed to be an empty array.
"""
fn = ast.parse("def foo(*, {}): pass".format(",".join(args)))
# The return value of ast.parse, as used here, is a Module object.
# We want the function definition that should be the Module's
# sole descendant.
assert isinstance(fn, ast.Module)
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.FunctionDef)
fn = fn.body[0]
# Remove the 'pass' statement.
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.Pass)
fn.body.clear()
fn.name = name
return fn
class static_vars:
"""Decorator which provides functions with static variables.
Usage:
#static_vars(foo=1, bar=2, ...)
def fun():
foo += 1
return foo + bar
The variables are implemented as upvalues defined by a wrapper
function.
Uses introspection to recompile the decorated function with its
context changed, and therefore may not work in all cases.
"""
def __init__(self, **variables):
self._variables = variables
def __call__(self, func):
if func.__name__ in self._variables:
raise ValueError(
"function name {} may not be the same as a "
"static variable name".format(func.__name__))
fname = inspect.getsourcefile(func)
lines, first_lineno = inspect.getsourcelines(func)
mod = ast.parse(textwrap.dedent("".join(lines)), filename=fname)
# The return value of ast.parse, as used here, is a Module
# object. Save that Module for use later and extract the
# function definition that should be its sole descendant.
assert isinstance(mod, ast.Module)
assert len(mod.body) == 1
assert isinstance(mod.body[0], ast.FunctionDef)
inner_fn = mod.body[0]
mod.body.clear()
# Don't apply decorators twice.
inner_fn.decorator_list.clear()
# Fix up line numbers. (Why the hell doesn't ast.parse take a
# starting-line-number argument?)
ast.increment_lineno(inner_fn, first_lineno - inner_fn.lineno)
# Inject a 'nonlocal' statement declaring the static variables.
svars = sorted(self._variables.keys())
inner_fn.body.insert(0, ast.Nonlocal(svars))
# Synthesize the wrapper function, which will take the static
# variableas as arguments.
outer_fn_name = ("_static_vars_wrapper_" +
inner_fn.name + "_" +
hex(id(self))[2:])
outer_fn = function_skeleton(outer_fn_name, svars)
outer_fn.body.append(inner_fn)
outer_fn.body.append(
ast.Return(value=ast.Name(id=inner_fn.name, ctx=ast.Load())))
mod.body.append(outer_fn)
ast.fix_missing_locations(mod)
# The new function definition must be evaluated in the same context
# as the original one. FIXME: supply locals if appropriate.
context = func.__globals__
exec(compile(mod, filename="<static-vars>", mode="exec"),
context)
# extract the function we just defined
outer_fn = context[outer_fn_name]
del context[outer_fn_name]
# and call it, supplying the static vars' initial values; this
# returns the adjusted inner function
adjusted_fn = outer_fn(**self._variables)
functools.update_wrapper(adjusted_fn, func)
return adjusted_fn
if __name__ == "__main__":
import sys
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test()
test()
test()
test()
Isn't this what classes are for?
import sys
class test_class:
a=0
b=[]
def test(self):
test_class.b.append(test_class.a)
test_class.a += 1
sys.stdout.write(repr(test_class.b) + "\n")
t = test_class()
t.test()
t.test()
[0]
[0, 1]
Here is a version of your regexp encoder:
import re
class encode:
encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(self, segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode.encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
e = encode()
print(e.encode_nonascii_and_percents('foo bar'))
foo%20bar
There is always the singleton class.
Is there a simple, elegant way to define Singletons in Python?
Is it possible to do something like the following:
class foo():
def bar(): # a method that doesn't take any args
# slow calculation
return somefloat
b = bar # bar is a function but b just gives you the float attribute
f = foo()
f.b # returns somefloat but doesn't require the empty parentheses
I hope the example is clear since I'm not super clear on what the terminology is for what I want to do. My basic goal is to remove a bunch of parentheses for methods that don't have arguments to make the code cleaner to read.
The function is slow and rarely used so it would be easiest to calculate it real time rather than calculate it once ahead of time and store the variable.
Is this possible? Is it good practice? Is there a better way?
The standard way to achieve this is to use property, which is a decorator:
class Foo():
#property
def bar(self):
# slow calculation
return somefloat
f = Foo()
f.bar # returns somefloat but doesn't require the empty parentheses
A couple of things to notice:
You still need self in the method signature as usual, because sometimes you're going to need to refer to e.g. self.some_attribute inside the method. As you can see, that doesn't affect the use of the property at all.
There's no need to clutter your API with both a f.bar() method and a f.b property - it's better to decide what makes most sense for your class than offer a heap of different ways to do the same thing.
b = bar obviously wouldn't work. However a property would for the simplest "doesn't require the empty parentheses" ask of yours:
b = property(bar)
Now every access to f.b will call f.bar() "behind the curtains".
However this means that if you access f.b twice, f.bar() gets called twice, repeating the computation. If the repetition is irrelevant (i.e if the result doesn't change for repeated computations on the same object) you can do better ("caching" the result in f.b forever once it's first been computed) -- something like:
class foo(object):
def bar(self): # a method that doesn't take any args
# slow calculation
return somefloat
def _cache_bar(self):
result = self.bar()
setattr(self, 'b', result)
return result
b = property(_cache_bar)
By static method, but need to call by parentheses.
class foo(object):
#staticmethod
def bar(): # a method that doesn't take any args
# slow calculation
return "abc"
b = bar # bar is a function but b just gives you the float attribute
f = foo()
print f.b()
output:
$ python test.py
abc
I saw this particular piece of code:
def g(x,y):
return x+y
def g(x,y):
return x*y
x,y=6,7
print (g(x,y))
The output is obviously(but not to me) is 42. Can somebody please explain this behavior? This is method overriding I suppose, but I'm still not getting the flow here.
When you define a function, and you redefine it, it will use the last one you defined, even the parameter is different:
def g(x,y):
return x+y
def g(x,y):
return x*y
x,y=6,7
print (g(x,y))
def hello():
return 'hello'
def hello():
return 'bye'
print hello()
def withone(word):
return word
def withone():
return 1==1
print withone('ok')
Output:
42
bye
TypeError: withone() takes no arguments (1 given)
And function name in Python is more like simple variable:
def hello():
return 'hello'
iamhello = hello # bind to the old one
def hello():
return 'bye'
print hello() # here is the new guy
print iamhello()
OutPut:
bye
hello
The devil is in the order of function definitions.
This is not technically method overriding as that requires class inheritance, instead it's a result of how python declares and references functions.
When declaring a function, python stores a reference to that function in a variable named after the function definition. (e.g. variable would be "foo" for "def foo():")
By declaring the function twice, the value of that variable gets overwritten by the second definition.
A Python script is parsed from top till bottom.
So anytime the same name of a variable or function or class occurs, it overwrites any definitions that where associated with this name before.
def g(x,z):
print('first')
def g():
print('second')
g = 3
print g
print g()
So look at this example which will result in the printout of '3' and then in an Exception: 'TypeError: 'int' object is not callable'
The name g is at first a function with two parameters, then it gets redefined to be a function with no parameters, then it gets redefined to be an int.
Which cannot be called obviously :)
Everything in python is treated as object, whether it be a function name or class name. So, when we define a function using 'def', the memory allocation is done for that method. Then python points the name that we assign to the function, to this allocated memory location. So if we define a method :-
def demo():
print 'hi'
the memory is allocated for the method, and the name 'demo' is pointed to its memory location as follows :-
Now as described by zoosuck in his second example, when you assign the function name to another variable :-
demo2 = demo # bind to the old one
then in that case, the assigned memory location to demo, is assigned to demo2 as well. So both demo and demo2 points to same location 12506.
print id(demo) # will print 12506
print id(demo2) # will print 12506
Now if we modify the above piece of code and in the next line, define a new method with same name demo:-
def demo():
print 'hi'
demo2 = demo # bind to the old one
demo() # Will print hi
def demo():
print "hello"
demo() # Will print hello
demo2() # Will print hi
then a completely new memory location 12534 is allocated for this new method, and now demo will point to this new location 12534 instead of pointing to the old one i.e. to 12506. But demo2 is still pointing to the location 12506.
I hope this will give you a clear idea of what is going on and how the method name is over-written.
Order matters, if names are same,last function you defined is processing. In your case it's;
def g(x,y):
return x*y
g is just a variable. The fact that the object it refers to is a function doesn't make it special in Python, so you can assign and reassign it as you want. In this case, the second assignment (which is what a function definition is) simply replaces the object stored there with a different one.
Functions and methods are normal objects like any others. So in
def g(x, y):
return x + y
def g(x, y):
return x * y
the second object g will override(replace) the first one, just like object a does below:
a = 1
a = 2
The number, type or order of parameters does not make any difference, because Python does not support function/method override and does not allow two functions/methods to have the same name.
If you are familiar with lambda function, also often called anonymous\inline functions, this might clear things up a bit
These two code blocks are essentially equal
def g(x,y):
return x+y
def g(x,y):
return x*y
g = lambda x,y: x+y
g = lambda x,y: x*y