when to release function stack data in python? - python

I have a question about this on testing the following code:
1,
def file_close_test():
f = open('/tmp/test', 'w+')
if __name__ == '__main__':
file_close_test()
# wait to see whether file closed.
import time
time.sleep(30)
2,
def file_close_on_exc_test():
f = open('/tmp/test', 'w+')
raise Exception()
def exception_wrapper():
try:
file_close_on_exc_test()
except:
pass
# wait to see whether file closed.
import time
time.sleep(10)
if __name__ == '__main__':
exception_wrapper()
import time
time.sleep(30)
The file object closed when the file_close_test exits because no reference to it.
After the exception raised,the file object not closed.so i think the related stack data not released.
After exception_wrapper exit,the file closed automatically.
can you explain this for me? thanks.

The exception includes a traceback object which can be used to access all of the local variables in any of the stack frames active when the exception was thrown. That means you can still access the file until the exception context is cleared.
Even after the sleep() at the end of exception_wrapper you could use sys.exc_info to get at the open file like this:
tb = sys.exc_info()[2]
print tb.tb_next.tb_frame.f_locals['f']
All of this is of course specific to the particular Python implementation you are using. Other implementations may not implicitly close files at all until they are garbage collected.
The bottom line is you should never depend on Python's reference counting or garbage collection to clean up resources like open files, always do it explicitly.

Related

Stopping a program in if statement [duplicate]

How do I exit a script early, like the die() command in PHP?
import sys
sys.exit()
details from the sys module documentation:
sys.exit([arg])
Exit from Python. This is implemented by raising the
SystemExit exception, so cleanup actions specified by finally clauses
of try statements are honored, and it is possible to intercept the
exit attempt at an outer level.
The optional argument arg can be an integer giving the exit status
(defaulting to zero), or another type of object. If it is an integer,
zero is considered “successful termination” and any nonzero value is
considered “abnormal termination” by shells and the like. Most systems
require it to be in the range 0-127, and produce undefined results
otherwise. Some systems have a convention for assigning specific
meanings to specific exit codes, but these are generally
underdeveloped; Unix programs generally use 2 for command line syntax
errors and 1 for all other kind of errors. If another type of object
is passed, None is equivalent to passing zero, and any other object is
printed to stderr and results in an exit code of 1. In particular,
sys.exit("some error message") is a quick way to exit a program when
an error occurs.
Since exit() ultimately “only” raises an exception, it will only exit
the process when called from the main thread, and the exception is not
intercepted.
Note that this is the 'nice' way to exit. #glyphtwistedmatrix below points out that if you want a 'hard exit', you can use os._exit(*errorcode*), though it's likely os-specific to some extent (it might not take an errorcode under windows, for example), and it definitely is less friendly since it doesn't let the interpreter do any cleanup before the process dies. On the other hand, it does kill the entire process, including all running threads, while sys.exit() (as it says in the docs) only exits if called from the main thread, with no other threads running.
A simple way to terminate a Python script early is to use the built-in quit() function. There is no need to import any library, and it is efficient and simple.
Example:
#do stuff
if this == that:
quit()
Another way is:
raise SystemExit
You can also use simply exit().
Keep in mind that sys.exit(), exit(), quit(), and os._exit(0) kill the Python interpreter. Therefore, if it appears in a script called from another script by execfile(), it stops execution of both scripts.
See "Stop execution of a script called with execfile" to avoid this.
While you should generally prefer sys.exit because it is more "friendly" to other code, all it actually does is raise an exception.
If you are sure that you need to exit a process immediately, and you might be inside of some exception handler which would catch SystemExit, there is another function - os._exit - which terminates immediately at the C level and does not perform any of the normal tear-down of the interpreter; for example, hooks registered with the "atexit" module are not executed.
I've just found out that when writing a multithreadded app, raise SystemExit and sys.exit() both kills only the running thread. On the other hand, os._exit() exits the whole process. This was discussed in "Why does sys.exit() not exit when called inside a thread in Python?".
The example below has 2 threads. Kenny and Cartman. Cartman is supposed to live forever, but Kenny is called recursively and should die after 3 seconds. (recursive calling is not the best way, but I had other reasons)
If we also want Cartman to die when Kenny dies, Kenny should go away with os._exit, otherwise, only Kenny will die and Cartman will live forever.
import threading
import time
import sys
import os
def kenny(num=0):
if num > 3:
# print("Kenny dies now...")
# raise SystemExit #Kenny will die, but Cartman will live forever
# sys.exit(1) #Same as above
print("Kenny dies and also kills Cartman!")
os._exit(1)
while True:
print("Kenny lives: {0}".format(num))
time.sleep(1)
num += 1
kenny(num)
def cartman():
i = 0
while True:
print("Cartman lives: {0}".format(i))
i += 1
time.sleep(1)
if __name__ == '__main__':
daemon_kenny = threading.Thread(name='kenny', target=kenny)
daemon_cartman = threading.Thread(name='cartman', target=cartman)
daemon_kenny.setDaemon(True)
daemon_cartman.setDaemon(True)
daemon_kenny.start()
daemon_cartman.start()
daemon_kenny.join()
daemon_cartman.join()
from sys import exit
exit()
As a parameter you can pass an exit code, which will be returned to OS. Default is 0.
I'm a total novice but surely this is cleaner and more controlled
def main():
try:
Answer = 1/0
print Answer
except:
print 'Program terminated'
return
print 'You wont see this'
if __name__ == '__main__':
main()
...
Program terminated
than
import sys
def main():
try:
Answer = 1/0
print Answer
except:
print 'Program terminated'
sys.exit()
print 'You wont see this'
if __name__ == '__main__':
main()
...
Program terminated Traceback (most recent call last): File "Z:\Directory\testdieprogram.py", line 12, in
main() File "Z:\Directory\testdieprogram.py", line 8, in main
sys.exit() SystemExit
Edit
The point being that the program ends smoothly and peacefully, rather than "I'VE STOPPED !!!!"
Problem
In my practice, there was even a case when it was necessary to kill an entire multiprocessor application from one of those processes.
The following functions work well if your application uses the only main process. But no one of the following functions didn't work in my case as the application had many other alive processes.
quit()
exit(0)
os._exit(0)
sys.exit(0)
os.kill(os.getppid(), 9) - where os.getppid() is the pid of parent process
The last one killed the main process and itself but the rest processes were still alive.
Solution
I had to kill it by external command and finally found the solution using pkill.
import os
# This can be called even in process worker and will kill
# whole application included correlated processes as well
os.system(f"pkill -f {os.path.basename(__file__)}")
In Python 3.5, I tried to incorporate similar code without use of modules (e.g. sys, Biopy) other than what's built-in to stop the script and print an error message to my users. Here's my example:
## My example:
if "ATG" in my_DNA:
## <Do something & proceed...>
else:
print("Start codon is missing! Check your DNA sequence!")
exit() ## as most folks said above
Later on, I found it is more succinct to just throw an error:
## My example revised:
if "ATG" in my_DNA:
## <Do something & proceed...>
else:
raise ValueError("Start codon is missing! Check your DNA sequence!")
My two cents.
Python 3.8.1, Windows 10, 64-bit.
sys.exit() does not work directly for me.
I have several nexted loops.
First I declare a boolean variable, which I call immediateExit.
So, in the beginning of the program code I write:
immediateExit = False
Then, starting from the most inner (nested) loop exception, I write:
immediateExit = True
sys.exit('CSV file corrupted 0.')
Then I go into the immediate continuation of the outer loop, and before anything else being executed by the code, I write:
if immediateExit:
sys.exit('CSV file corrupted 1.')
Depending on the complexity, sometimes the above statement needs to be repeated also in except sections, etc.
if immediateExit:
sys.exit('CSV file corrupted 1.5.')
The custom message is for my personal debugging, as well, as the numbers are for the same purpose - to see where the script really exits.
'CSV file corrupted 1.5.'
In my particular case I am processing a CSV file, which I do not want the software to touch, if the software detects it is corrupted. Therefore for me it is very important to exit the whole Python script immediately after detecting the possible corruption.
And following the gradual sys.exit-ing from all the loops I manage to do it.
Full code: (some changes were needed because it is proprietory code for internal tasks):
immediateExit = False
start_date = '1994.01.01'
end_date = '1994.01.04'
resumedDate = end_date
end_date_in_working_days = False
while not end_date_in_working_days:
try:
end_day_position = working_days.index(end_date)
end_date_in_working_days = True
except ValueError: # try statement from end_date in workdays check
print(current_date_and_time())
end_date = input('>> {} is not in the list of working days. Change the date (YYYY.MM.DD): '.format(end_date))
print('New end date: ', end_date, '\n')
continue
csv_filename = 'test.csv'
csv_headers = 'date,rate,brand\n' # not real headers, this is just for example
try:
with open(csv_filename, 'r') as file:
print('***\nOld file {} found. Resuming the file by re-processing the last date lines.\nThey shall be deleted and re-processed.\n***\n'.format(csv_filename))
last_line = file.readlines()[-1]
start_date = last_line.split(',')[0] # assigning the start date to be the last like date.
resumedDate = start_date
if last_line == csv_headers:
pass
elif start_date not in working_days:
print('***\n\n{} file might be corrupted. Erase or edit the file to continue.\n***'.format(csv_filename))
immediateExit = True
sys.exit('CSV file corrupted 0.')
else:
start_date = last_line.split(',')[0] # assigning the start date to be the last like date.
print('\nLast date:', start_date)
file.seek(0) # setting the cursor at the beginnning of the file
lines = file.readlines() # reading the file contents into a list
count = 0 # nr. of lines with last date
for line in lines: #cycling through the lines of the file
if line.split(',')[0] == start_date: # cycle for counting the lines with last date in it.
count = count + 1
if immediateExit:
sys.exit('CSV file corrupted 1.')
for iter in range(count): # removing the lines with last date
lines.pop()
print('\n{} lines removed from date: {} in {} file'.format(count, start_date, csv_filename))
if immediateExit:
sys.exit('CSV file corrupted 1.2.')
with open(csv_filename, 'w') as file:
print('\nFile', csv_filename, 'open for writing')
file.writelines(lines)
print('\nRemoving', count, 'lines from', csv_filename)
fileExists = True
except:
if immediateExit:
sys.exit('CSV file corrupted 1.5.')
with open(csv_filename, 'w') as file:
file.write(csv_headers)
fileExists = False
if immediateExit:
sys.exit('CSV file corrupted 2.')
In Python 3.9, you can also use: raise SystemExit("Because I said so").
Just put at the end of your code quit() and that should close a python script.
use exit and quit in .py files
and sys.exit for exe files

How to stop execution of the main module, if an exception occurs in a sub module

Have a situation here where I have a main module which calls in a number of sub modules, which in turn calls sub modules and its methods.
Eg:
Main module:
import submodule1
import submodule2
var1=submodule1.test(2,0)
var2=submodule2.verify(2,"zero")
Submodule1:
import blah
def test(x,y):
try:
return x/y
except:
#some code to print the error to log file
#some code to determine if this is a critical error
submodule2:
import blah
def verify(x,y):
try:
return x*y
except:
#some code to print the error to log file
#some code to determine if this is a critical error
Now, in the above case, the first call to method "submodule1.test(2,0)" will throw an exception which is logged into a log file and then I try determine if the error is a critical one or not. If it is a critical error, I want to stop execution and close all files, conenctions, modules etc (basically cleanup).
With the above code the control returns to the main module, the execution proceeds to the next line.
My main module might have number of methods or object instantiations. I do not want to check for a condition for every single statement.
Any suggestions how this can be achieved? Thank you!
So the best way here would be to just let the exception propagate up to the main module.
try:
# Code
except:
# Print
if is_critical():
raise # This will re-raise the exception you just caught.

Does finally ensure some code gets run atomically, no matter what?

Assume I'm going to write a Python script that catches the KeyboardInterrupt exception to be able to get terminated by the user using Ctrl+C safely
However, I can't put all critical actions (like file writes) into the catch block because it relies on local variables and to make sure a subsequent Ctrl+C does not break it anyway.
Would it work and be good practice to use a try-catch block with empty (pass) try part and all the code inside the finally part to define this snippet as "atomic, interrupt-safe code" which may not get interrupted mid-way?
Example:
try:
with open("file.txt", "w") as f:
for i in range(1000000):
# imagine something useful that takes very long instead
data = str(data ** (data ** data))
try:
pass
finally:
# ensure that this code is not interrupted to prevent file corruption:
f.write(data)
except KeyboardInterrupt:
print("User aborted, data created so far saved in file.txt")
exit(0)
In this example I don't care for the currently produced data string, i.e. that creation could be interrupted and no write would be triggered. But once the write started, it must be completed, that's all I want to ensure. Also, what would happen if an exception (or KeyboardInterrupt) happened while performing the write inside the finally clause?
Code in finally can still be interrupted too. Python makes no guarantees about this; all it guarantees is that execution will switch to the finally suite after the try suite completed or if an exception in the try suite was raised. A try can only handle exceptions raised within its scope, not outside of it, and finally is outside of that scope.
As such there is no point in using try on a pass statement. The pass is a no-op, it won't ever be interrupted, but the finally suite can easily be interrupted still.
You'll need to pick a different technique. You could write to a separate file and move that into place on successful completion; the OS guarantees that a file move is atomic, for example. Or record your last successful write position, and truncate the file to that point if a next write is interrupted. Or write markers in your file that signal a successful record, so that reads know what to ignore.
In your case, there is no problem, because file writes are atomic, but if you have some file object implementetion, that is more complex, your try-except is in the wrong place. You have to place exception handling around the write:
try:
f.write(data)
except:
#do some action to restore file integrity
raise
For example, if you write binary data, you could to the following:
filepos = f.tell()
try:
f.write(data)
except:
# remove the already written data
f.seek(filepos)
f.truncate()
raise

Python file.read() Callback

I currently have code that reads raw content from a file chosen by the user:
def choosefile():
filec = tkFileDialog.askopenfile()
# Wait a few to prevent attempting to displayng the file's contents before the entire file was read.
time.sleep(1)
filecontents = filec.read()
But, sometimes people open big files that take more than 2 seconds to open. Is there a callback for FileObject.read([size])? For people who don't know what a callback is, it's a operation executed once another operation has executed.
Slightly modified from the docs:
#!/usr/bin/env python
import signal, sys
def handler(signum, frame):
print "You took too long"
sys.exit(1)
f = open(sys.argv[1])
# Set the signal handler and a 2-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(2)
contents = f.read()
signal.alarm(0) # Disable the alarm
print contents
Answer resolved by asker
Hm, I made a mistake at first. tkFileDialog.askopenfile() does not read the file, but FileObject.read() reads the file, and blocks the code. I found the solution according to #kindall. I'm not a complete expert at Python, though.
Your question seems to assume that Python will somehow start reading your file while some other code executes, and therefore you need to wait for the read to catch up. This is not even slightly true; both open() and read() are blocking calls and will not return until the operation has completed. Your sleep() is not necessary and neither is your proposed workaround. Simply open the file and read it. Python won't do anything else while that is happening.
Thanks kindall! Resolved code:
def choosefile():
filec = tkFileDialog.askopenfile()
filecontents = filec.read()

Why won't my script write to a file?

import time
import traceback
import sys
import tools
from BeautifulSoup import BeautifulSoup
f = open("randomwords.txt","w")
while 1:
try:
page = tools.download("http://wordnik.com/random")
soup = BeautifulSoup(page)
si = soup.find("h1")
w = si.string
print w
f.write(w)
f.write("\n")
time.sleep(3)
except:
traceback.print_exc()
continue
f.close()
It prints just fine. It just won't write to the file. It's 0 bytes.
You can never leave the while loop, hence the f.close() call will never be called and the stream buffer to the file will never be flushed.
Let me explain a little bit further, in your exception catch statement you've included continue so there's no "exit" to the loop condition. Perhaps you should add some sort of indicator that you've reached the end of the page instead of a static 1. Then you'd see the close call and information printed to the file.
A bare except is almost certainly a bad idea; you should only handle the exception you expect to see. Then if it does something totally unexpected you will still get a useful error trace about it.
import time
import tools
from BeautifulSoup import BeautifulSoup
def scan_file(url, logf):
try:
page = tools.download(url)
except IOError:
print("Couldn't read url {0}".format(url))
return
try:
soup = BeautifulSoup(page)
w = soup.find("h1").string
except AttributeError:
print("Couldn't find <h1> tag")
return
print(w)
logf.write(w)
logf.write('\n')
def main():
with open("randomwords.txt","a") as logf:
try:
while True:
time.sleep(3)
scan_file("http://wordnik.com/random", logf)
except KeyboardInterrupt:
break
if __name__=="__main__":
main()
Now you can close the program by typing Ctrl-C, and the "with" clause will ensure that the log file is closed properly.
From what i understand, you want to output a random number every three second into a file. But caching will take place, so you will not see your numbers until the cache has grown too large, typically in the order of 4K bytes.
i suggest that in your loop, you add a f.flush() before the sleep() line.
Also, like wheaties sugessted, you should have proper exception handling (if i want to stop your program, i will likely do a SIGINT using Ctrl+C, and your program won't stop in this case) and a proper exit path.
I'm sure that when you test your program, you will kill it hard to stop it, and any random number it has written will not be written because the file is not properly closed. If you program could exit normally, you would have close()d the file, and close() triggers a flush(), and so you would have something written in your file.
Read the answer posted by wheaties.
And, if you want to force to write the file's buffer to the disk, read:
http://docs.python.org/library/stdtypes.html#file.flush

Categories

Resources