Develop Reference

Why must a new global variable be created to reference the current class instance in "exec"?

I have a class that contains ~20 methods, and in def __init__(self, ...): I have to call many of these methods (~9) but I didn't want to have to call each individual method one by one.
So I took the easy way out and created two list list comprehensions, that use exec to call each method:
[exec("self.create%s()" % x) for x in "ArticleObjects SeriesObjects ArticleList SearchList".split(" ")]
[exec("self.compile%sPage(self)" % x) for x in "About Screenshots Search Contact Articles".split(" ")]
When I ran this code using python3 filename.py I got an error, that read:
NameError: name 'self' is not defined
Through trial and error I found that; in order to get this code to work I had to create a copy of self called instance and make the new instance variable a global variable and then call the method using ClassName.methodName(instance) instead of self.methodName():
With the working code being:
global instance; instance = self
[exec("ClassNamecreate%s(instance)" % x) for x in "ArticleObjects SeriesObjects ArticleList SearchList".split(" ")]
[exec("ClassName.compile%sPage(instance)" % x) for x in "About Screenshots Search Contact Articles".split(" ")]
Why is this? Why is the self variable undefined in exec despite it being available to the scope that exec is being called in?
Update: I'm using Python 3.6.7

There's lots of good suggestions here for how to avoid the exec statement (which is generally bad), but to answer your question about why this happens, it's got more to do with the list comprehension. List comprehensions create a new scope, and when you call exec without a globals or locals argument, it uses the locals() function:
Note: The default locals act as described for function locals() below
Source
Here you can see what the results of the locals() function look like from within a list comprehension:
class Sample:
def __init__(self):
k = 4
print(locals())
exec("print(locals())")
[print(locals()) for x in range(1)]
[exec("print(locals())") for x in range(1)]
Sample()
output:
{'k': 4, 'self': <__main__.Sample object at 0x00000000030295C0>}
{'k': 4, 'self': <__main__.Sample object at 0x00000000030295C0>}
{'x': 0, '.0': <range_iterator object at 0x00000000030019F0>}
{'x': 0, '.0': <range_iterator object at 0x00000000030019F0>}
So, locals() is the same inside or outside the exec. It's the list comprehension that changes it. Only, when you're outside an exec statement, the interpreter can fall past the locals of the list comprehension and find self in the outer scope. No such luck once you call exec.

Using getattr is simpler (and usually safer) than exec. Try something along these lines:
def __init__(self):
suffixes = ["ArticleObjects", "SeriesObjects", ...]
for suffix in suffixes:
method = getattr(self, "create" + suffix)
method()

I wouldn't use exec for this. While it may be the shortest version, it might also confuse both collaborators and code analysis tools. I'd use something like this instead:
class Test:
def __init__(self):
for f in (self.createA, self.createB, self.createC):
f()
def createA(self):
print("A")
def createB(self):
print("B")
def createC(self):
print("C")

Python basic nested statements and scope concept

def func():
def nested():
global x
x = 1
x = 2
func()
print(x)
The correct answer would be '2' and the reason why is because func() is not defined. But when I read this it seems that func() is defined as nested(). I would think that when you call func() that would then automatically call nested(). I'm struggling to grasp this and understand why I shouldn't read it that way.

You're defining nested inside func, but you're not calling nested() anywhere in func, so when you call func() it's effectively doing nothing.
To do what you want try defining func as:
def func():
def nested():
global x
x = 1
nested()
UPDATE: After StevenRumbalski's comment, I think a small addition about what exactly is going on in that function can help clarify things around.
Python's functions are themselves objects which can respond to the operator ().
When you define a new function, what you're actually doing is instantiating a function object and giving it a name. In the example above, def func() creates a function instance and gives it name func, so that when you apply operator () to func (i.e. when you call the function with func()) the code of the function associated to that name is executed.
Let's now take one step further and look at what happens with nested.
Nested is defined inside func's scope, so when you exit func's scope the name nested is not defined anymore.
If, however, you return the function object to the caller of func, you can use that object to run nested's code.
A small example:
def returning_func():
def nested():
print("I am nested!")
return nested # note the lack of the () operator!
In this case, if we do
my_reference_to_nested = func()
nothing gets printed, because nested is defined but not executed.
If we call it, however:
my_reference_to_nested()
we execute nested's code and print to the output I am nested!

How can I change a global variable from within a function? [duplicate]

This question already has answers here:
Using global variables in a function
(25 answers)
UnboundLocalError trying to use a variable (supposed to be global) that is (re)assigned (even after first use)
(14 answers)
Closed 7 months ago.
I'm trying to add or subtract from a defined variable, but how can I overwrite the old value with the new one?
a = 15
def test():
a = a + 10
print(a)
test()
Error message:
Traceback (most recent call last):
File "test.py", line 7, in <module>
test()
File "test.py", line 4, in test
a = a +10
UnboundLocalError: local variable 'a' referenced before assignment

The error that you get when you try to run your code is:
UnboundLocalError: local variable 'a' referenced before assignment
… which, on the face of it, seems strange: after all, the first statement in the code above (a = 15) is an assignment. So, what's going on?
Actually, there are two distinct things happening, and neither of them are obvious unless you already know about them.
First of all, you actually have two different variables:
The a in your first line is a global variable (so called because it exists in the global scope, outside of any function definitions).
The a in the other lines is a local variable, meaning that it only exists inside your test() function.
These two variables are completely unrelated to each other, even though they have the same name.
A variable is local to a function if there's a statement assigning to it inside that function - for instance, your a = a +10 line.
Even so, the error still looks strange - after all, the very first thing you do inside test() is assign to a, so how can it be referenced beforehand?
The answer is that, in an assignment statement, Python evaluates everything on the right hand side of the = sign before assigning it to the name on the left hand side – so even though the assignment is written first in your code, a gets referenced first in that right hand side: a +10.
There are two ways you can get around this. The first is to tell Python that you really want the a inside test() to be the same a in the global scope:
def test():
global a
a = a + 10
print(a)
This will work, but it's a pretty bad way to write programs. Altering global variables inside functions gets hard to manage really quickly, because you usually have lots of functions, and none of them can ever be sure that another one isn't messing with the global variable in some way they aren't expecting.
A better way is to pass variables as arguments to functions, like this:
a = 15
def test(x):
x = x + 10
print(x)
test(a)
Notice that the name doesn't have to be the same - your new definition of test() just says that it accepts a value, and then does something with it. You can pass in anything you like – it could be a, or the number 7, or something else. In fact, your code will always be easier to understand if you try to avoid having variables with the same name in different scopes.
If you play with the code above, you'll notice something interesting:
>>> a = 15
>>> test(a)
25
>>> a
15
… a didn't change! That's because although you passed it into test() and it got assigned to x, it was then x that got changed, leaving the original a alone.
If you want to actually change a, you need to return your modified x from the function, and then reassign it back to a on the outside:
>>> a = 15
>>>
>>> def test(x):
... x = x + 10
... print(x)
... return x
...
>>> a = test(a)
25
>>> a
25

You're modifying a variable a created in the scope of the function test(). If you want the outer a to be modified, you could do:
a = 15
def test():
global a
a = a + 1
print(a)
test()

I would do it this way:
def test(a):
a = a +10
return a
print(test(15))
Note that in the version hereby proposed there are some things differing from yours.
First, what I wrote down would create a function that has, as an input, the value a (in this case set to 15 when we call the function -already defined in the last line-), then assigns to the object a the value a (which was 15) plus 10, then returns a (which has been modified and now is 25) and, finally, prints a out thanks to the last line of code:
print(test(15))
Note that what you did wasn't actually a pure function, so to speak. Usually we want functions to get an input value (or several) and return an input value (or several). In your case you had an input value that was actually empty and no output value (as you didn't use return). Besides, you tried to write this input a outside the function (which, when you called it by saying test(a) the value a was not loaded an gave you the error -i.e. in the eyes of the computer it was "empty").
Furthermore, I would encourage you to get used to writing return within the function and then using a print when you call it (just like I wrote in the last coding line: print(test(15))) instead of using it inside the function. It is better to use print only when you call the function and want to see what the function is actually doing.
At least, this is the way they showed me in basic programming lessons. This can be justified as follows: if you are using the return within the function, the function will give you a value that can be later used in other functions (i.e. the function returns something you can work with). Otherwise, you would only get a number displayed on screen with a print, but the computer could not further work with it.
P.S. You could do the same doing this:
def test(a):
a +=10
return a
print(test(15))

a = 15
def test():
global a
a = a + 10
print(a)
test()
If you want to change the values of local variables inside a function you need to use the global keyword.

The scope of the variable is local to the block unless defined explicitly using keyword global. There is another way to access the global variable local to a function using the globals function:
a = 15
def test():
a = globals()['a']
a += 10
print(a)
test()
The above example will print 25 while keeping the global value intact, i.e., 15.

The issue here is scope.
The error basically means you're using the variable a, but it doesn't exist yet. Why? The variable a has not been declared in the test() function. You need to "invite" global variables before using them.
There are several ways to include a variable in the scope of a function. One way is to use it as a parameter like this:
def test(number):
number = number + 10
print(number)
Then implement it as:
test(a)
Alternatively, you could also use the global keyword to show the function that a is a global variable. How? Like this:
def test():
global a
a = a + 10
print(a)
Using the global keyword just tells the test where to look for a.
With the first method, you can't modify the global variable a, because you made a copy of it and you're using the copy. In the second method, though, any changes you make in the function test() to a, will affect the global variable a.

# All the mumbo jumbo aside, here is an experiment
# to illustrate why this is something useful
# in the language of Python:
a = 15 # This could be read-only, should check docs
def test_1():
b = a + 10 # It is perfectly ok to use 'a'
print(b)
def test_2():
a = a + 10 # Refrain from attempting to change 'a'
print(a)
test_1() # No error
test_2() # UnboundLocalError: ...

Your error has nothing to do with it being already defined…
A variable is only valid inside its so-called scope: If you create a variable in a function it is only defined in this function.
def test():
x = 17
print(x) # Returns 17
test()
print(x) # Results in an error.

Returning Variables in Functions Python Not Working Right [duplicate]

This question already has answers here:
Using global variables in a function
(25 answers)
How do I get a result (output) from a function? How can I use the result later?
(4 answers)
Closed last month.
I have been trying to return a variable in a function in a variable and use it outside of it:
test = 0
def testing():
test = 1
return test
testing()
print(test)
But when I run it, the result is 0. How could I fix this problem?

You are messing up a bit the scopes and/or assignment. Try this:
def testing():
test = 1
return test
test = testing()
print(test)
Explanation: The test inside testing is different to the test inside the module. You have to assign it on module-level to get the expected result.

Because you declare test in the function, it is not a global variable, thus, you can not access the variable test you created in the function outside of it as they are different scopes
If you want to return test to a variable, you have to do
result = testing()
print(result)
Or, you can also add a global statement:
test = 0
def testing():
global test
test = 1
return test
testing()
print(test)
By the way, when doing a conditional statement, you don't need the brackets around the 1==1 :).

TLDR: A return value must be assigned to something at the call site.
test = testing()
Functions in Python have their own scope. It is created on entering (calling) the function, and destroyed when leaving it. Assignment to a name inside a scope makes that name local to this scope - causing it to be destroyed along with the scope.
# start outer scope
test = 0 # create name outer:test
def testing():
# start inner scope
test = 1 # create name outer.testing:test
return test
# end inner scope
# destroy name outer.testing:test
testing() # new temporary inner scope outer.testing
print(test) # use name outer:test
# end outer scope
Notably, names in an inner scope may "shadow" names from an outer scope. While the name test exists in both testing and the outer scope, it does not refer to the same thing. This has two important implications:
The assignment to the inner test does not affect the outer test.
At the end of testing, the inner test is destroyed and only the outer test remains.
This is why calling testing() does not have the desired effect: it never modifies the outer test passed to print.
The return statement defines the value returned by calling a function. It does not return the name, only the value pointed to.
def testing():
test = 1 # test refers to the value 1
return test # return test => value 1
The value returned by a function is like any other value - be it from a literal, lookup, or other. Most importantly, the value does not persist unless you assign it to a name or use it directly.
testing() # call test, discard its value
test = testing() # call test, store its value as `test`
print(testing()) # call test, use its value in `print`
So in order to return something from a function for later use, you must store the result to a name. You can then use that name in a later statement. A minimal example for your case looks like this:
# we already can define testing here
# it is overwritten later on, then
def testing():
# all names we use inside of testing are gone at the end
# if we just want a value, we can skip temporary names
return 1
# store the return value of testing() for later use
test = testing()
print(test)
Addendum: It is possible for a function to modify its containing scope. However, names must then be explicitly declared as being from a foreign scope.
The nonlocal and global keywords allow to modify names from outer scopes. A nonlocal is the name in the closest matching function scope. A global is the name at the module scope, regardless of any functions in-between.
test = 0
def increment():
global test # declare test as belonging to a specific scope
test += 1
# no need to return something
# we already modified the outer `test`
print(test) # 0
increment()
print(test) # 1
Note that modifying outer names is often the sign of an anti-pattern, moreso for globals than nonlocals. Beyond small scripts, it gets difficult to trace what is accessing and modifying globals. Often, it is more appropriate to use classes or generators to hold state.
A function can always read names from its containing scope, provided it never writes to the same name. Such closures are very easy to create, and the lack of modification makes them easier to trace. Note that modifying a name anywhere in a function makes it local, unless declared global or nonlocal:
test = 0
def increment():
global test
test += 1
def show_test():
# we never modify `test`, so it is fetched from the outside
print(test)
def show_and_increment1(): # this function is broken!
print(test) # `test` is *not* the outer one, since we modify it in the function
test += 1 # modifying `test` makes it local for the *entire* function
def show_and_increment2(): # this function works!
global test # force `test` to be global
print(test)
test += 1
show_test() # 0
increment()
show_test() # 1
show_and_increment2() # 1
show_and_increment2() # 2
show_and_increment2() # 3
show_test() # 4
show_and_increment1() # UnboundLocalError: local variable 'test' referenced before assignment

Inside the function testing(), you're creating a new variable test, not referring to the one that already exists. If you want to do that, you should use a global statement in the top, as in:
def testing():
global test
...etc...

Your test variable inside the function does not have a global scope. So, if you want to store the return value in a variable and output it after that, you can do something like this:
result = testing()
print(result)

Can a decorated function access variables of the decorator

I'm trying to understand how decorators work, and was wondering if a decorated function can access variables of the decorator. For example, in the following code, how do I make f1 have access to localVariable? Is that possible, and is it even a good way of doing things?
def funcDec(func):
localVariable = "I'm a local string"
def func2Return(*args):
print "Calling localVariable from decorator " + localVariable
func(*args)
print "done with calling f1"
return func2Return
#funcDec
def f1(x, y):
print x + y
print localVariable
f1(2, 3)

Because of Python's scoping rules, a decorated function generally can't access any variables in the decorator. However, since functions can have arbitrary attributes assigned to them, you could do something like the following in the decorator to get a similar effect (due to the same scoping rules):
def funcDec(func):
localVariable = "I'm a local string"
def wrapped(*args):
print("Calling localVariable from funcDec " + localVariable)
func(*args)
print("done with calling f1")
wrapped.attrib = localVariable
return wrapped
#funcDec
def f1(x, y):
print(x + y)
print('f1.attrib: {!r}'.format(f1.attrib))
f1(2, 3)
Which would produce the following output:
Calling localVariable from funcDec I'm a local string
5
f1.attrib: "I'm a local string"
done with calling f1
Someone asked whether this could be applied to methods of a class:
The answer is "yes", but you have to reference the method either through the class itself or the instance of it passed as the self argument. Both techniques are shown below. Using self is preferable since it makes the code independent of the name of the class it's is in.
class Test(object):
#funcDec
def f1(self):
print('{}.f1() called'.format(self.__class__.__name__))
print('self.f1.attrib: {!r}'.format(self.f1.attrib)) # Preferred.
print('Test.f1.attrib: {!r}'.format(Test.f1.attrib)) # Also works.
print()
test = Test()
test.f1()
Output:
Calling localVariable from funcDec I'm a local string
Test.f1() called
self.f1.attrib: "I'm a local string"
Test.f1.attrib: "I'm a local string"
done with calling f1
Update
Another way of doing this that would give the decorated function more direct access to decorator variables would be to temporarily "inject" them into its global namespace (and then remove them afterwards).
I got the idea from #Martijn Pieters' answer to the somewhat related question: How to inject variable into scope with a decorator?
def funcDec(func):
localVariable = "I'm a local string"
# Local variable(s) to inject into wrapped func.
context = {'localVariable': localVariable}
def wrapped(*args):
func_globals = func.__globals__
# Save copy of any global values that will be replaced.
saved_values = {key: func_globals[key] for key in context if key in func_globals}
func_globals.update(context)
print(f'Calling localVariable from funcDec: {localVariable!r}')
try:
func(*args)
finally:
func_globals.update(saved_values) # Restore any replaced globals.
print(f'done with calling {func.__name__}()')
return wrapped
#funcDec
def f1(x, y):
print(x + y)
print(f'Calling funcDec localVariable from f1: {localVariable!r}')
f1(2, 3)
Result from this version:
Calling localVariable from funcDec: "I'm a local string"
5
Calling funcDec localVariable from f1: "I'm a local string"
done with calling f1()

I think it helps if you keep in mind that a decorator
#deco
def f(...): ...
is just syntactic sugar for
def f(...): ...
f = deco(f)
rather than some kind of macro expansion. In Python the scope of a variable is determined statically, so for a global (module-level) function a variable that is neither passed as an argument nor assigned to will be looked up in the global namespace.
Therefore you have to pass on a local variable of func2Return() explicitly. Change the signature of f1 to f1(x, y, localvariable=None) and have the wrapper function fun2Return call it with
f1(*args, localvariable=localvariable)

No, you can't. See this previous question. Just because the function is a decorator doesn't mean functions it calls have special access to its variables. If you do this:
def func():
a = 2
otherFunc()
Then otherFunc doesn't have access to the variable a. That's how it works for all function calls, and it's how it works for decorators too.
Now, the wrapper function you define inside the decorator (func2Return in your example) does have access to the variables, because that function is lexically in the same scope as those variables. So your line print "Calling localVariable from decorator " + localVariable will work. You can use this to some extent to wrap the decorated function with behavior that depends on variables in the decorator. But the function actually being decorated (f1 in your example) doesn't have access to those variables.
A function only has access to local variables from the scope where the function definition actually is. Functions don't get variables from calling scopes. (This is a good thing. If they did, it would be a huge mess.)

I found this question trying to use variables in (I think) the opposite direction, and found the answer from Martineau to apply in that opposite direction. That is, I am using a decorator to wrap some standard error handling around a series of similar functions but want to pass some of that error info up since that's where my logging is.
So, I was able to do something like this:
def error_handler(func):
def wrapper_decorator(*args,**kwargs):
try:
func(*args,**kwargs)
except Exception as e: # I actually handle things better than a general grab here
logger.error(f'func had bad value {error_handler.value}')
return wrapper_decorator
#error_handler
def main():
# blah blah, do stuff, read a dataframe from a web page
error_handler.value = value # value is created within this main() func
# blah blah, do more stuff where we actually end up having an error
This sets the .value attribute on error_handler to a value that is generated within the scope of main(), effectively letting me 'pass up' that value.
In my actual specific example, I'm using a dataframe and getting errors when I save to serve because the data is poorly formatted. main() reads in the dataframe, transforms it, and (tries) to save to server. This setup allows me to pass the dataframe (as the value in the example code) to my actual error_handler function and then save the dataframe as a csv so I can inspect it, and I don't have to have the saving of the dataframe within each individual function.