This is my code:
import os
if os.path.exists(r'C:\Genisis_AI'):
print("Main File path exists! Continuing with startup")
else:
createDirs()
def createDirs():
os.makedirs(r'C:\Genisis_AI\memories')
When I execute this, it throws an error:
File "foo.py", line 6, in <module>
createDirs()
NameError: name 'createDirs' is not defined
I made sure it's not a typo and I didn't misspell the function's name, so why am I getting a NameError?
You can't call a function unless you've already defined it. Move the def createDirs(): block up to the top of your file, below the imports.
Some languages allow you to use functions before defining them. For example, javascript calls this "hoisting". But Python is not one of those languages.
Note that it's allowable to refer to a function in a line higher than the line that creates the function, as long as chronologically the definition occurs before the usage. For example this would be acceptable:
import os
def doStuff():
if os.path.exists(r'C:\Genisis_AI'):
print("Main File path exists! Continuing with startup")
else:
createDirs()
def createDirs():
os.makedirs(r'C:\Genisis_AI\memories')
doStuff()
Even though createDirs() is called on line 7 and it's defined on line 9, this isn't a problem because def createDirs executes before doStuff() does on line 12.
Related
I have a file named file.py containing the following script:
def b():
print("b")
def proc():
print("this is main")
b()
proc()
And I have another file named caller.py, which contains this script:
text = open('file.py').read()
exec(text)
When I run it, I get the expected output:
this is main
b
However, if I change caller.py to this:
def main():
text = open('file.py').read()
exec(text)
main()
I get the following error:
this is main
Traceback (most recent call last):
File "./caller.py", line 7, in <module>
main()
File "./caller.py", line 5, in main
exec(text)
File "<string>", line 10, in <module>
File "<string>", line 8, in main
NameError: global name 'b' is not defined
How is function b() getting lost? It looks to me like I'm not violating any scope rules. I need to make something similar to the second version of caller.py work.
exec(text) executes text in the current scope, but modifying that scope (as def b does via the implied assignment) is undefined.
The fix is simple:
def main():
text = open('file.py').read()
exec(text, {})
This causes text to run in an empty global scope (augmented with the default __builtins object), the same way as in a regular Python file.
For details, see the exec documentation. It also warns that modifying the default local scope (which is implied when not specifying any arguments besides text) is unsound:
The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.
Would it work for you if you imported and called the function instead?
myfile.py
def b():
print("b")
def proc():
print("this is main")
b()
caller.py
import myfile
myfile.proc()
This is my code:
import os
if os.path.exists(r'C:\Genisis_AI'):
print("Main File path exists! Continuing with startup")
else:
createDirs()
def createDirs():
os.makedirs(r'C:\Genisis_AI\memories')
When I execute this, it throws an error:
File "foo.py", line 6, in <module>
createDirs()
NameError: name 'createDirs' is not defined
I made sure it's not a typo and I didn't misspell the function's name, so why am I getting a NameError?
You can't call a function unless you've already defined it. Move the def createDirs(): block up to the top of your file, below the imports.
Some languages allow you to use functions before defining them. For example, javascript calls this "hoisting". But Python is not one of those languages.
Note that it's allowable to refer to a function in a line higher than the line that creates the function, as long as chronologically the definition occurs before the usage. For example this would be acceptable:
import os
def doStuff():
if os.path.exists(r'C:\Genisis_AI'):
print("Main File path exists! Continuing with startup")
else:
createDirs()
def createDirs():
os.makedirs(r'C:\Genisis_AI\memories')
doStuff()
Even though createDirs() is called on line 7 and it's defined on line 9, this isn't a problem because def createDirs executes before doStuff() does on line 12.
Windows 10, Python 3.5.1 x64 here.
This is weird... Let's say I have this script, called do.py. Please note the import string statement:
import string
# Please note that if the print statement is OUTSIDE 'main()', it works.
# It's like if 'main()' can't see the imported symbols from 'string'
def main():
print(string.ascii_lowercase)
main()
I want to run it from a "launcher script", in a subthread, like this (launcher.py):
import sys
import threading
sys.argv.append('do.py')
def run(script, filename):
exec(compile(script, filename, 'exec'))
with open(sys.argv[1], 'rb') as _:
script = _.read()
# But this WORKS:
# exec(compile(script, sys.argv[1], 'exec'))
thread = threading.Thread(name='Runner', target=run, args=(script, sys.argv[1]))
thread.start()
thread.join()
It dies with the following error:
Exception in thread Runner:
Traceback (most recent call last):
File "C:\Program Files\Python35\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Program Files\Python35\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "tmpgui.py", line 7, in run
exec(compile(script, filename, 'exec'))
File "do.py", line 6, in <module>
main()
File "do.py", line 4, in main
print(string.ascii_lowercase)
NameError: name 'string' is not defined
That is, the exec'ed code is not importing string properly or something like that, and within main() the string module is not visible.
This is not the full code of my project, which is too big to post here, but the bare minimum I've created which mimics the problem.
Just in case someone is curious, I'm rewriting an old program of mine which imported the main() function of a script and ran that function with the standard output streams redirected to a tkinter text box. Instead of importing a function from the script, I want to load the script and run it. I don't want to use subprocess for a whole variety of reasons, I prefer to run the "redirected" code in a thread and communicate with the main thread which is the one handling the GUI. And that part works perfectly, the only problem I have is this and I can't understand why is happening!
My best bet: I should be passing something in globals or locals dictionaries to exec, but I'm at a lost here...
Thanks a lot in advance!
exec(thing) is equivalent to exec(thing, globals(), locals()).
Thus,
the local symbol table of do.py is the local symbol table of the run function
the global symbol table of do.py is the global symbol table of launcher.py
import string imports the module and binds it to the variable in the local space, which is the local space of the run function. You can verify this:
def run(script, filename):
try:
exec(compile(script, filename, 'exec'))
finally:
assert 'string' in locals(), "won't fail because 'import' worked properly"
main has a separate local scope, but it shares the global symbol table with do.py and, consequently, with launcher.py.
Python tried to find the variable named string inside both local (it's empty) and global symbol tables of main, but failed, and raised the NameError.
Pass one empty dictionary in a call to exec:
def run(script, filename):
exec(compile(script, filename, 'exec'), {})
I am attempting to use multiprocessing to call derived class member function defined in a different module. There seem to be several questions dealing with calling class methods from the same module, but none from different modules. For example, if I have the following structure:
main.py
multi/
__init__.py (empty)
base.py
derived.py
main.py
from multi.derived import derived
from multi.base import base
if __name__ == '__main__':
base().multiFunction()
derived().multiFunction()
base.py
import multiprocessing;
# The following two functions wrap calling a class method
def wrapPoolMapArgs(classInstance, functionName, argumentLists):
className = classInstance.__class__.__name__
return zip([className] * len(argumentLists), [functionName] * len(argumentLists), [classInstance] * len(argumentLists), argumentLists)
def executeWrappedPoolMap(args, **kwargs):
classType = eval(args[0])
funcType = getattr(classType, args[1])
funcType(args[2], args[3:], **kwargs)
class base:
def multiFunction(self):
mppool = multiprocessing.Pool()
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
def method(self,args):
print "base.method: " + args.__str__()
derived.py
from base import base
class derived(base):
def method(self,args):
print "derived.method: " + args.__str__()
Output
base.method: (0,)
base.method: (1,)
base.method: (2,)
Traceback (most recent call last):
File "e:\temp\main.py", line 6, in <module>
derived().multiFunction()
File "e:\temp\multi\base.py", line 15, in multiFunction
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 567, in get
raise self._value
NameError: name 'derived' is not defined
I have tried fully qualifying the class name in the wrapPoolMethodArgs method, but that just gives the same error, saying multi is not defined.
Is there someway to achieve this, or must I restructure to have all classes in the same package if I want to use multiprocessing with inheritance?
This is almost certainly caused by the ridiculous eval based approach to dynamically invoking specific code.
In executeWrappedPoolMap (in base.py), you convert a str name of a class to the class itself with classType = eval(args[0]). But eval is executed in the scope of executeWrappedPoolMap, which is in base.py, and can't find derived (because the name doesn't exist in base.py).
Stop passing the name, and pass the class object itself, passing classInstance.__class__ instead of classInstance.__class__.__name__; multiprocessing will pickle it for you, and you can use it directly on the other end, instead of using eval (which is nearly always wrong; it's code smell of the strongest sort).
BTW, the reason the traceback isn't super helpful is that the exception is raised in the worker, caught, pickle-ed, and sent back to the main process and re-raise-ed. The traceback you see is from that re-raise, not where the NameError actually occurred (which was in the eval line).
UPDATE: As noted by Mr. Fooz, the functional version of the wrapper has a bug, so I reverted to the original class implementation. I've put the code up on GitHub:
https://github.com/nofatclips/timeout/commits/master
There are two commits, one working (using the "import" workaround) the second one broken.
The source of the problem seems to be the pickle#dumps function, which just spits out an identifier when called on an function. By the time I call Process, that identifier points to the decorated version of the function, rather than the original one.
ORIGINAL MESSAGE:
I was trying to write a function decorator to wrap a long task in a Process that would be killed if a timeout expires. I came up with this (working but not elegant) version:
from multiprocessing import Process
from threading import Timer
from functools import partial
from sys import stdout
def safeExecution(function, timeout):
thread = None
def _break():
#stdout.flush()
#print (thread)
thread.terminate()
def start(*kw):
timer = Timer(timeout, _break)
timer.start()
thread = Process(target=function, args=kw)
ret = thread.start() # TODO: capture return value
thread.join()
timer.cancel()
return ret
return start
def settimeout(timeout):
return partial(safeExecution, timeout=timeout)
##settimeout(1)
def calculatePrimes(maxPrimes):
primes = []
for i in range(2, maxPrimes):
prime = True
for prime in primes:
if (i % prime == 0):
prime = False
break
if (prime):
primes.append(i)
print ("Found prime: %s" % i)
if __name__ == '__main__':
print (calculatePrimes)
a = settimeout(1)
calculatePrime = a(calculatePrimes)
calculatePrime(24000)
As you can see, I commented out the decorator and assigned the modified version of calculatePrimes to calculatePrime. If I tried to reassign it to the same variable, I'd get a "Can't pickle : attribute lookup builtins.function failed" error when trying to call the decorated version.
Anybody has any idea of what is happening under the hood? Is the original function being turned into something different when I assign the decorated version to the identifier referencing it?
UPDATE: To reproduce the error, I just change the main part to
if __name__ == '__main__':
print (calculatePrimes)
a = settimeout(1)
calculatePrimes = a(calculatePrimes)
calculatePrimes(24000)
#sleep(2)
which yields:
Traceback (most recent call last):
File "c:\Users\mm\Desktop\ING.SW\python\thread2.py", line 49, in <module>
calculatePrimes(24000)
File "c:\Users\mm\Desktop\ING.SW\python\thread2.py", line 19, in start
ret = thread.start()
File "C:\Python33\lib\multiprocessing\process.py", line 111, in start
self._popen = Popen(self)
File "C:\Python33\lib\multiprocessing\forking.py", line 241, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python33\lib\multiprocessing\forking.py", line 160, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtin
s.function failed
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python33\lib\multiprocessing\forking.py", line 344, in main
self = load(from_parent)
EOFError
P.S. I also wrote a class version of safeExecution, which has exactly the same behaviour.
Move the function to a module that's imported by your script.
Functions are only picklable in python if they're defined at the top level of a module. Ones defined in scripts are not picklable by default. Module-based functions are pickled as two strings: the name of the module, and the name of the function. They're unpickled by dynamically importing the module then looking up the function object by name (hence the restriction on top-level-only functions).
It's possible to extend the pickle handlers to support semi-generic function and lambda pickling, but doing so can be tricky. In particular, it can be difficult to reconstruct the full namespace tree if you want to properly handle things like decorators and nested functions. If you want to do this, it's best to use Python 2.7 or later or Python 3.3 or later (earlier versions have a bug in the dispatcher of cPickle and pickle that's unpleasant to work around).
Is there an easy way to pickle a python function (or otherwise serialize its code)?
Python: pickling nested functions
http://bugs.python.org/issue7689
EDIT:
At least in Python 2.6, the pickling works fine for me if the script only contains the if __name__ block, the script imports calculatePrimes and settimeout from a module, and if the inner start function's name is monkey-patched:
def safeExecution(function, timeout):
...
def start(*kw):
...
start.__name__ = function.__name__ # ADD THIS LINE
return start
There's a second problem that's related to Python's variable scoping rules. The assignment to the thread variable inside start creates a shadow variable whose scope is limited to one evaluation of the start function. It does not assign to the thread variable found in the enclosing scope. You can't use the global keyword to override the scope because you want and intermediate scope and Python only has full support for manipulating the local-most and global-most scopes, not any intermediate ones. You can overcome this problem by placing the thread object in a container that's housed in the intermediate scope. Here's how:
def safeExecution(function, timeout):
thread_holder = [] # MAKE IT A CONTAINER
def _break():
#stdout.flush()
#print (thread)
thread_holder[0].terminate() # REACH INTO THE CONTAINER
def start(*kw):
...
thread = Process(target=function, args=kw)
thread_holder.append(thread) # MUTATE THE CONTAINER
...
start.__name__ = function.__name__ # MAKES THE PICKLING WORK
return start
Not sure really why you get that problem, but to answer your title question: Why does the decorator not work?
When you pass arguments to a decorator, you need to structure the code slightly different. Essentially you have to implement the decorator as a class with an __init__ and an __call__.
In the init, you collect the arguments that you send to the decorator, and in the call, you'll get the function you decorate:
class settimeout(object):
def __init__(self, timeout):
self.timeout = timeout
def __call__(self, func):
def wrapped_func(n):
func(n, self.timeout)
return wrapped_func
#settimeout(1)
def func(n, timeout):
print "Func is called with", n, 'and', timeout
func(24000)
This should get you going on the decorator front at least.