I have the following decorator with parameters:
from functools import wraps
def pdecor(p):
def decorator(fn):
#wraps(fn)
def wrapper(*args, **kwargs):
p -= 1
return fn(*args, **wargs)
return wrapper
return decorator
Trying to use the decorator results in :
>>> #pdecor(1)
... def run(): pass
...
>>> run()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in wrapper
UnboundLocalError: local variable 'p' referenced before assignment
>>>
Why can't I change the p?
Because you assign to p inside wrapper, Python treats the p inside wrapper as local to wrapper. In Python 3 you can use nonlocal p to mark p as referenced from an outer scope. In Python 2 there is no way to assign to the intermediate p, although you can get a reference to the same value by passing it into the nested functions as a keyword argument (e.g., def decorator(fn, p=p)).
However, it's not clear what you're getting at with this anyway. p is already only local to pdecor. No code outside pdecor can access p, so decrementing it won't have any effect on any code elsewhere. So whether you can decrement p or not, it won't really accomplish anything.
Related
I have a file named file.py containing the following script:
def b():
print("b")
def proc():
print("this is main")
b()
proc()
And I have another file named caller.py, which contains this script:
text = open('file.py').read()
exec(text)
When I run it, I get the expected output:
this is main
b
However, if I change caller.py to this:
def main():
text = open('file.py').read()
exec(text)
main()
I get the following error:
this is main
Traceback (most recent call last):
File "./caller.py", line 7, in <module>
main()
File "./caller.py", line 5, in main
exec(text)
File "<string>", line 10, in <module>
File "<string>", line 8, in main
NameError: global name 'b' is not defined
How is function b() getting lost? It looks to me like I'm not violating any scope rules. I need to make something similar to the second version of caller.py work.
exec(text) executes text in the current scope, but modifying that scope (as def b does via the implied assignment) is undefined.
The fix is simple:
def main():
text = open('file.py').read()
exec(text, {})
This causes text to run in an empty global scope (augmented with the default __builtins object), the same way as in a regular Python file.
For details, see the exec documentation. It also warns that modifying the default local scope (which is implied when not specifying any arguments besides text) is unsound:
The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.
Would it work for you if you imported and called the function instead?
myfile.py
def b():
print("b")
def proc():
print("this is main")
b()
caller.py
import myfile
myfile.proc()
The following executes without an error in Python 3:
code = """
import math
def func(x):
return math.sin(x)
func(10)
"""
_globals = {}
exec(code, _globals)
But if I try to capture the local variable dict as well, it fails with a NameError:
>>> _globals, _locals = {}, {}
>>> exec(code, _globals, _locals)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-9-aeda81bf0af1> in <module>()
----> 1 exec(code, {}, {})
<string> in <module>()
<string> in func(x)
NameError: name 'math' is not defined
Why is this happening, and how can I execute this code while capturing both global and local variables?
From the exec() documentation:
Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
You passed in two separate dictionaries, but tried to execute code that requires module-scope globals to be available. import math in a class would produce a local scope attribute, and the function you create won't be able to access that as class scope names are not considered for function closures.
See Naming and binding in the Python execution model reference:
Class definition blocks and arguments to exec() and eval() are special in the context of name resolution. A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution with an exception that unbound local variables are looked up in the global namespace. The namespace of the class definition becomes the attribute dictionary of the class. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods[.]
You can reproduce the error by trying to execute the code in a class definition:
>>> class Demo:
... import math
... def func(x):
... return math.sin(x)
... func(10)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in Demo
File "<stdin>", line 4, in func
NameError: name 'math' is not defined
Just pass in one dictionary.
The following are test classes with methods not taking in cls or self arguments and dont have #staticmethod decorator. They work like normal static methods without complaining about arguments. This seems contrary to my understanding of python methods. Does python automatically treat non-class, non-instance methods as static?
>>> class Test():
... def testme(s):
... print(s)
...
>>> Test.testme('hello')
hello
>>> class Test():
... def testme():
... print('no')
...
>>> Test.testme()
no
P.S: I am using python3.4
It sort of does, yes. However, note that if you call such an "implicit static method" on an instance, you will get an error:
>>> Test().testme()
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
Test().testme()
TypeError: testme() takes 0 positional arguments but 1 was given
This is because the self parameter still gets passed, which doesn't happen with a proper #staticmethod:
>>> class Test:
#staticmethod
def testme():
print('no')
>>> Test.testme()
no
>>> Test().testme()
no
Note that this doesn't work in Python 2:
>>> class Test(object):
... def testme():
... print 'no'
...
>>> Test.testme()
Traceback (most recent call last):
File "<ipython-input-74-09d78063da08>", line 1, in <module>
Test.testme()
TypeError: unbound method testme() must be called with Test instance as first argument (got nothing instead)
But in Python 3, unbound methods were removed, as Alex Martelli points out in this answer. So really all you're doing is calling a plain function that happens to be defined inside the Test class.
UPDATE: As noted by Mr. Fooz, the functional version of the wrapper has a bug, so I reverted to the original class implementation. I've put the code up on GitHub:
https://github.com/nofatclips/timeout/commits/master
There are two commits, one working (using the "import" workaround) the second one broken.
The source of the problem seems to be the pickle#dumps function, which just spits out an identifier when called on an function. By the time I call Process, that identifier points to the decorated version of the function, rather than the original one.
ORIGINAL MESSAGE:
I was trying to write a function decorator to wrap a long task in a Process that would be killed if a timeout expires. I came up with this (working but not elegant) version:
from multiprocessing import Process
from threading import Timer
from functools import partial
from sys import stdout
def safeExecution(function, timeout):
thread = None
def _break():
#stdout.flush()
#print (thread)
thread.terminate()
def start(*kw):
timer = Timer(timeout, _break)
timer.start()
thread = Process(target=function, args=kw)
ret = thread.start() # TODO: capture return value
thread.join()
timer.cancel()
return ret
return start
def settimeout(timeout):
return partial(safeExecution, timeout=timeout)
##settimeout(1)
def calculatePrimes(maxPrimes):
primes = []
for i in range(2, maxPrimes):
prime = True
for prime in primes:
if (i % prime == 0):
prime = False
break
if (prime):
primes.append(i)
print ("Found prime: %s" % i)
if __name__ == '__main__':
print (calculatePrimes)
a = settimeout(1)
calculatePrime = a(calculatePrimes)
calculatePrime(24000)
As you can see, I commented out the decorator and assigned the modified version of calculatePrimes to calculatePrime. If I tried to reassign it to the same variable, I'd get a "Can't pickle : attribute lookup builtins.function failed" error when trying to call the decorated version.
Anybody has any idea of what is happening under the hood? Is the original function being turned into something different when I assign the decorated version to the identifier referencing it?
UPDATE: To reproduce the error, I just change the main part to
if __name__ == '__main__':
print (calculatePrimes)
a = settimeout(1)
calculatePrimes = a(calculatePrimes)
calculatePrimes(24000)
#sleep(2)
which yields:
Traceback (most recent call last):
File "c:\Users\mm\Desktop\ING.SW\python\thread2.py", line 49, in <module>
calculatePrimes(24000)
File "c:\Users\mm\Desktop\ING.SW\python\thread2.py", line 19, in start
ret = thread.start()
File "C:\Python33\lib\multiprocessing\process.py", line 111, in start
self._popen = Popen(self)
File "C:\Python33\lib\multiprocessing\forking.py", line 241, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python33\lib\multiprocessing\forking.py", line 160, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtin
s.function failed
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python33\lib\multiprocessing\forking.py", line 344, in main
self = load(from_parent)
EOFError
P.S. I also wrote a class version of safeExecution, which has exactly the same behaviour.
Move the function to a module that's imported by your script.
Functions are only picklable in python if they're defined at the top level of a module. Ones defined in scripts are not picklable by default. Module-based functions are pickled as two strings: the name of the module, and the name of the function. They're unpickled by dynamically importing the module then looking up the function object by name (hence the restriction on top-level-only functions).
It's possible to extend the pickle handlers to support semi-generic function and lambda pickling, but doing so can be tricky. In particular, it can be difficult to reconstruct the full namespace tree if you want to properly handle things like decorators and nested functions. If you want to do this, it's best to use Python 2.7 or later or Python 3.3 or later (earlier versions have a bug in the dispatcher of cPickle and pickle that's unpleasant to work around).
Is there an easy way to pickle a python function (or otherwise serialize its code)?
Python: pickling nested functions
http://bugs.python.org/issue7689
EDIT:
At least in Python 2.6, the pickling works fine for me if the script only contains the if __name__ block, the script imports calculatePrimes and settimeout from a module, and if the inner start function's name is monkey-patched:
def safeExecution(function, timeout):
...
def start(*kw):
...
start.__name__ = function.__name__ # ADD THIS LINE
return start
There's a second problem that's related to Python's variable scoping rules. The assignment to the thread variable inside start creates a shadow variable whose scope is limited to one evaluation of the start function. It does not assign to the thread variable found in the enclosing scope. You can't use the global keyword to override the scope because you want and intermediate scope and Python only has full support for manipulating the local-most and global-most scopes, not any intermediate ones. You can overcome this problem by placing the thread object in a container that's housed in the intermediate scope. Here's how:
def safeExecution(function, timeout):
thread_holder = [] # MAKE IT A CONTAINER
def _break():
#stdout.flush()
#print (thread)
thread_holder[0].terminate() # REACH INTO THE CONTAINER
def start(*kw):
...
thread = Process(target=function, args=kw)
thread_holder.append(thread) # MUTATE THE CONTAINER
...
start.__name__ = function.__name__ # MAKES THE PICKLING WORK
return start
Not sure really why you get that problem, but to answer your title question: Why does the decorator not work?
When you pass arguments to a decorator, you need to structure the code slightly different. Essentially you have to implement the decorator as a class with an __init__ and an __call__.
In the init, you collect the arguments that you send to the decorator, and in the call, you'll get the function you decorate:
class settimeout(object):
def __init__(self, timeout):
self.timeout = timeout
def __call__(self, func):
def wrapped_func(n):
func(n, self.timeout)
return wrapped_func
#settimeout(1)
def func(n, timeout):
print "Func is called with", n, 'and', timeout
func(24000)
This should get you going on the decorator front at least.
I know how to override an object's getattr() to handle calls to undefined object functions. However, I would like to achieve the same behavior for the builtin getattr() function. For instance, consider code like this:
call_some_undefined_function()
Normally, that simply produces an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'call_some_undefined_function' is not defined
I want to override getattr() so that I can intercept the call to "call_some_undefined_function()" and figure out what to do.
Is this possible?
I can only think of a way to do this by calling eval.
class Global(dict):
def undefined(self, *args, **kargs):
return u'ran undefined'
def __getitem__(self, key):
if dict.has_key(self, key):
return dict.__getitem__(self, key)
return self.undefined
src = """
def foo():
return u'ran foo'
print foo()
print callme(1,2)
"""
code = compile(src, '<no file>', 'exec')
globals = Global()
eval(code, globals)
The above outputs
ran foo
ran undefined
You haven't said why you're trying to do this. I had a use case where I wanted to be capable of handling typos that I made during interactive Python sessions, so I put this into my python startup file:
import sys
import re
def nameErrorHandler(type, value, traceback):
if not isinstance(value, NameError):
# Let the normal error handler handle this:
nameErrorHandler.originalExceptHookFunction(type, value, traceback)
name = re.search(r"'(\S+)'", value.message).group(1)
# At this point we know that there was an attempt to use name
# which ended up not being defined anywhere.
# Handle this however you want...
nameErrorHandler.originalExceptHookFunction = sys.excepthook
sys.excepthook = nameErrorHandler
Hopefully this is helpful for anyone in the future who wants to have a special error handler for undefined names... whether this is helpful for the OP or not is unknown since they never actually told us what their intended use-case was.