I was reading a question in StackOverflow where the user applied the "with" statement twice in a row, pipelining the results from a variable declared inside the first with statement into the second. Like so (simple example):
with open('first_file.txt', 'r') as f:
loaded_file = f.readlines()
#...Prepare a csv file somehow - loaded_file is not declared outside with...
with open("second_file.csv", "w") as f:
for line in loaded_file:
f.write(line+"\n")
Considering variable scopes, why does it work?
Only one statement creates a new scope: the def statement. Any other assignment creates a name that is local to the current function body.
The exceptions are:
A name declared global refers to the (module) global scope rather than the local function body.
A name declared nonlocal refers to the name defined in the closest containing function scope (or global scope if no other name is found)
The Python interpreter itself can define names.
In your example, f is either a local variable or a global, depending on the scope where the with statement occurs. In no way is f local to any other particular statement.
No, "with" statements does not create a new scope.
The "With" statement is a resource developed by the Python Dev team to generalize a common (and heavily recommended) practice of closing opened resources even after an exception ocurred. Imagine the following situation:
try:
f = open('file.txt', 'w')
#Do some processing that can raise exceptions and leave 'f' open, eventually locking the file or having trash data loaded into RAM.
#To avoid this situation, a mindful developer will do the following:
finally:
f.close()
It gets verbose easily.
To solve the problem, python dev team proposed the use of some dunder methods which encapsule this process: __enter__() and __exit__() - these are invoked "under the hood" when you use a "with" statement.
You can even implement them in your own classes!
class controlled_execution:
def __enter__(self):
set things up
return thing
def __exit__(self, type, value, traceback):
tear things down
with controlled_execution() as thing:
some code
In the end, a with statement, even though there's identation, is not a separate block of code. It is just an ellegant try...finaly block. It abstracts a "contained" piece of code.
This can be easily comprehended by looking at a try...except statement with variables declared inside the try:
x = 10
try:
y = load_using_failable_function()
z = 5
except FunctionFailureException:
y = 10
#If you treat it properly, there's nothing wrong with doing:
x = x + y + z
I hope it is clear to anyone else looking for this reason.
Consulted websites:
https://www.geeksforgeeks.org/with-statement-in-python/
https://effbot.org/zone/python-with-statement.htm
Related
This question already has answers here:
UnboundLocalError trying to use a variable (supposed to be global) that is (re)assigned (even after first use)
(14 answers)
Closed 6 months ago.
I am using the PyQt library to take a screenshot of a webpage, then reading through a CSV file of different URLs. I am keeping a variable feed that incremements everytime a URL is processed and therefore should increment to the number of URLs.
Here's code:
webpage = QWebPage()
fo = open("C:/Users/Romi/Desktop/result1.txt", "w")
feed = 0
def onLoadFinished(result):
#fo.write( column1[feed])#, column2[feed], urls[feed])
#feed = 0
if not result:
print "Request failed"
fo.write(column1[feed])
fo.write(',')
fo.write(column2[feed])
fo.write(',')
#fo.write(urls[feed])
fo.write(',')
fo.write('404,image not created\n')
feed = feed + 1
sys.exit(1)
save_page(webpage, outputs.pop(0)) # pop output name from list and save
if urls:
url = urls.pop(0) # pop next url to fetch from list
webpage.mainFrame().load(QUrl(url))
fo.write(column1[feed])#,column2[feed],urls[feed],'200','image created','/n')
fo.write(',')
fo.write(column2[feed])
fo.write(',')
#fo.write(urls[feed])
fo.write(',')
fo.write('200,image created\n')
feed = feed + 1
else:
app.quit() # exit after last url
webpage.connect(webpage, SIGNAL("loadFinished(bool)"), onLoadFinished)
webpage.mainFrame().load(QUrl(urls.pop(0)))
#fo.close()
sys.exit(app.exec_())
It gives me the error:
local variable feed referenced before the assignment at fo.write(column1[feed])#,column2[feed],urls[feed],'200','image created','/n')
Any idea why?
When Python parses the body of a function definition and encounters an assignment such as
feed = ...
Python interprets feed as a local variable by default. If you do not wish for it to be a local variable, you must put
global feed
in the function definition. The global statement does not have to be at the beginning of the function definition, but that is where it is usually placed. Wherever it is placed, the global declaration makes feed a global variable everywhere in the function.
Without the global statement, since feed is taken to be a local variable, when Python executes
feed = feed + 1,
Python evaluates the right-hand side first and tries to look up the value of feed. The first time through it finds feed is undefined. Hence the error.
The shortest way to patch up the code is to add global feed to the beginning of onLoadFinished. The nicer way is to use a class:
class Page(object):
def __init__(self):
self.feed = 0
def onLoadFinished(self, result):
...
self.feed += 1
The problem with having functions which mutate global variables is that it makes it harder to grok your code. Functions are no longer isolated units. Their interaction extends to everything that affects or is affected by the global variable. Thus it makes larger programs harder to understand.
By avoiding mutating globals, in the long run your code will be easier to understand, test and maintain.
Put a global statement at the top of your function and you should be good:
def onLoadFinished(result):
global feed
...
To demonstrate what I mean, look at this little test:
x = 0
def t():
x += 1
t()
this blows up with your exact same error where as:
x = 0
def t():
global x
x += 1
t()
does not.
The reason for this is that, inside t, Python thinks that x is a local variable. Furthermore, unless you explicitly tell it that x is global, it will try to use a local variable named x in x += 1. But, since there is no x defined in the local scope of t, it throws an error.
As the Python interpreter reads the definition of a function (or, I think, even a block of indented code), all variables that are assigned to inside the function are added to the locals for that function. If a local does not have a definition before an assignment, the Python interpreter does not know what to do, so it throws this error.
The solution here is to add
global feed
to your function (usually near the top) to indicate to the interpreter that the feed variable is not local to this function.
in my case that exact same error was triggered by a typo !
I thought my my var name was
varAlpha
but in the code i had defined
varalpha
& got the error
UnboundLocalError: local variable 'varAlpha' referenced before assignment
when calling varAlpha
I hope it helps somebody one day searching for that error & wondering (as my search for that error led me here while being unrelated with the use of global or not global which was a head scratcher !)
You can do like this for the function scope
def main()
self.x = 0
def incr():
self.x += 1
for i in range(5):
incr()
print(self.x)
This question already has answers here:
UnboundLocalError trying to use a variable (supposed to be global) that is (re)assigned (even after first use)
(14 answers)
Closed 6 months ago.
I am using the PyQt library to take a screenshot of a webpage, then reading through a CSV file of different URLs. I am keeping a variable feed that incremements everytime a URL is processed and therefore should increment to the number of URLs.
Here's code:
webpage = QWebPage()
fo = open("C:/Users/Romi/Desktop/result1.txt", "w")
feed = 0
def onLoadFinished(result):
#fo.write( column1[feed])#, column2[feed], urls[feed])
#feed = 0
if not result:
print "Request failed"
fo.write(column1[feed])
fo.write(',')
fo.write(column2[feed])
fo.write(',')
#fo.write(urls[feed])
fo.write(',')
fo.write('404,image not created\n')
feed = feed + 1
sys.exit(1)
save_page(webpage, outputs.pop(0)) # pop output name from list and save
if urls:
url = urls.pop(0) # pop next url to fetch from list
webpage.mainFrame().load(QUrl(url))
fo.write(column1[feed])#,column2[feed],urls[feed],'200','image created','/n')
fo.write(',')
fo.write(column2[feed])
fo.write(',')
#fo.write(urls[feed])
fo.write(',')
fo.write('200,image created\n')
feed = feed + 1
else:
app.quit() # exit after last url
webpage.connect(webpage, SIGNAL("loadFinished(bool)"), onLoadFinished)
webpage.mainFrame().load(QUrl(urls.pop(0)))
#fo.close()
sys.exit(app.exec_())
It gives me the error:
local variable feed referenced before the assignment at fo.write(column1[feed])#,column2[feed],urls[feed],'200','image created','/n')
Any idea why?
When Python parses the body of a function definition and encounters an assignment such as
feed = ...
Python interprets feed as a local variable by default. If you do not wish for it to be a local variable, you must put
global feed
in the function definition. The global statement does not have to be at the beginning of the function definition, but that is where it is usually placed. Wherever it is placed, the global declaration makes feed a global variable everywhere in the function.
Without the global statement, since feed is taken to be a local variable, when Python executes
feed = feed + 1,
Python evaluates the right-hand side first and tries to look up the value of feed. The first time through it finds feed is undefined. Hence the error.
The shortest way to patch up the code is to add global feed to the beginning of onLoadFinished. The nicer way is to use a class:
class Page(object):
def __init__(self):
self.feed = 0
def onLoadFinished(self, result):
...
self.feed += 1
The problem with having functions which mutate global variables is that it makes it harder to grok your code. Functions are no longer isolated units. Their interaction extends to everything that affects or is affected by the global variable. Thus it makes larger programs harder to understand.
By avoiding mutating globals, in the long run your code will be easier to understand, test and maintain.
Put a global statement at the top of your function and you should be good:
def onLoadFinished(result):
global feed
...
To demonstrate what I mean, look at this little test:
x = 0
def t():
x += 1
t()
this blows up with your exact same error where as:
x = 0
def t():
global x
x += 1
t()
does not.
The reason for this is that, inside t, Python thinks that x is a local variable. Furthermore, unless you explicitly tell it that x is global, it will try to use a local variable named x in x += 1. But, since there is no x defined in the local scope of t, it throws an error.
As the Python interpreter reads the definition of a function (or, I think, even a block of indented code), all variables that are assigned to inside the function are added to the locals for that function. If a local does not have a definition before an assignment, the Python interpreter does not know what to do, so it throws this error.
The solution here is to add
global feed
to your function (usually near the top) to indicate to the interpreter that the feed variable is not local to this function.
in my case that exact same error was triggered by a typo !
I thought my my var name was
varAlpha
but in the code i had defined
varalpha
& got the error
UnboundLocalError: local variable 'varAlpha' referenced before assignment
when calling varAlpha
I hope it helps somebody one day searching for that error & wondering (as my search for that error led me here while being unrelated with the use of global or not global which was a head scratcher !)
You can do like this for the function scope
def main()
self.x = 0
def incr():
self.x += 1
for i in range(5):
incr()
print(self.x)
I'm a bit confused about a code in the book "Learning Python", p. 539.
As far as I know assignments within a function are only in this local scope. So if I want to change a global one I first have to declare it global. But why does the following code change the builtin.open() to custom completely once called?
import builtins
def makeopen(id):
original = builtins.open
def custom(*pargs, **kargs):
print('Custom open call %r: ' % id, pargs, kargs)
return original(*pargs, **kargs)
builtins.open = custom
If I call makeopen('spam') and a F = open('text.txt') afterwards I get the custom call. So the builtin.open() has been changed in the whole script after the makeopen('spam'). Why?
And if I would make some more makeopen('xx') one builtin.open('text.txt') would print the custom call for every created makeopen. Why?
Comparing this code to
x = 99
def changing():
x = 88
changing()
print(x)
doesnt even help me. Isn't it the same but with an x instead of builtin.open()?
A variable is considered local if you assign to it anywhere in the function, unless you declare it global.
In your first piece of code, you never assign anything to builtins, so it's not considered local. You just change one of its attributes, open.
The rule is respected!
In your second piece of code, you assign something to x in x = 88, so it is considered local.
When you call makeopen, you replace the original, global open with custom. custom, when executed, prints its name and calls the original open.
If you call makeopen a second time, it will create a second, different custom function, and make the name builtins.open refer to it. When you call this function, it will print its name, then call original, which is what builtins.open referred to when it was created - and that is your first custom function, which will print its name and call the original open.
So, successive calls to makeopen create a chain of functions, and calling open will make each of them run and call its predecessor.
I realize the question is a bit specific, sorry.
I'm trying to solve a python puzzle where what I input get exec'ed. My aim is to store something of arbitrary length in the global name space (e.g, change the variable target). You'll see below that my input is limited to 35 chars.
Here's the program:
#!/usr/bin/env python2
from sys import modules
modules.clear()
del modules
raw_input2 = raw_input
exception2 = Exception
__builtins__.__dict__.clear()
__builtins__ = None
target = None # change this !
while 1:
try:
scope = {'r':0}
exec 'r = ' + raw_input2()[:35] in scope
print 'Result:', scope['r']
except exception2, e:
print "Error: ", e
As said, my point is to store data somewhere, since vars get reset each loop.
I know I could use globals()['target']=xxx, but the builtins were disabled.
In theory I could use global target;target=xxx but this executes in scope, not in the global scope (also I think the global keyword must come at the beginning of the anonymous function)
I know all variables are stored in some __dict__ object, but the only way I know to write in it is via globals()['target']=xxx which is disabled
I know you can painfully access the super-object with ().__class__.__base__, but with the limit of 35 characters, it doesn't seem to be the way (this string alone is 21 chars already, you need two more to start with 0;, so only 11 char remaining to assign something...).
Any ideas ?
Thanks!
The answer is: write in __builtins__.
Example input:
1;__builtins__['a']="string1"
Result: 1
1;__builtins__['a']+="string2"
... which actually seems very simple since it is right above the variable marked # change this in the question. I do not fully understand yet why the __builtins__ var is passed in the exec scope, especially since it should be None (and hence not a dictionary), but is assignable like a dictionary.
But it works, tested in python 2.7
I have a utilities.py file for my python project. It contains only util functions, for example is_float(string), is_empty(file), etc.
Now I want to have a function is_valid(number), which has to:
read from a file, valid.txt, which contains all numbers which are valid, and load them onto a map/set.
check the map for the presence of number and return True or False.
This function is called often, and running time should be as small as possible. I don't want to read open and read valid.txt everytime the function is called. The only solution I have come up with is to use a global variable, valid_dict, which is loaded once from valid.txt when utilities.py is imported. The loading code is written as main in utilities.py.
My question is how do I do this without using a global variable, as it is considered bad practice? What is a good design pattern for doing such a task without using globals? Also note again that this is a util file, so there should ideally be no main as such, just functions.
The following is a simple example of a closure. The dictionary, cache, is encapsulated within the outer function (load_func), but remains in scope of the inner, even when it is returned. Notice that load_func returns the inner function as an object, it does not call it.
In utilities.py:
def _load_func(filename):
cache = {}
with open(filename) as fn:
for line in fn:
key, value = line.split()
cache[int(key)] = value
def inner(number):
return number in cache
return inner
is_valid = _load_func('valid.txt')
In __main__:
from utilities import is_valid # or something similar
if is_valid(42):
print(42, 'is valid')
else:
print(42, 'is not valid')
The dictionary (cache) creation could have been done using a dictionary comprehension, but I wanted you to concentrate on the closure.
The variable valid_dict would not be global but local to utilities.py. It would only become global if you did something like from utilities import *. Now that is considered bad practice when you're developing a package.
However, I have used a trick in cases like this that essentially requires a static variable: Add an argument valid_dict={} to is_valid(). This dictionary will be instantiated only once and each time the function is called the same dict is available in valid_dict.
def is_valid(number, valid_dict={}):
if not valid_dict:
# first call to is_valid: load valid.txt into valid_dict
# do your check
Do NOT assign to valid_dict in the if-clause but only modify it: e.g., by setting keys valid_dict[x] = y or using something like valid_dict.update(z).
(PS: Let me know if this is considered "dirty" or "un-pythonic".)