Develop Reference

"break" not terminating script python

I am trying to open an Excel file and to take the columns according to the names (headings). If it doesn't find those headings, I want the script to stop and display a message that the column was not found.
I first assign the current numbers to the columns with those names, and then I search for the name in the headings to make sure there were no columns inserted in between that will alter the column numbers. If the number is not the same, it will take whatever column number the name is in.
I have been trying to put an "else" under the "elif" and asking it to print an error and then I put a "break". The problem is that it doesn't stop the script if a column name is not found, it just takes the default assignment. I do understand why that break won't stop the whole script, but I don't know how to put it in order to stop it. The code is a bit more complex than just this, but I hope this will give a good understanding.
I also tried removing the default assignment but then it says that it can't find "i" so it gives me an error.
I would really appreciate some help.
Code:
colTPNumber = 2
colTDNumber = 3
rows = sheet.iter_rows(min_row=1, max_row=1)
firstRow = next(rows)
for i in firstRow:
if str(i.value) == "TP":
colTPNumber = i.column
elif str(i.value) == "TD":
colTDNumber = i.column
else:
print ("Column name not found")
break

break exits the loop, not the whole program.
Use exit() or sys.exit() (don't forget to import sys for the second option) to exit the program
Example:
import sys #unnessary if you're using exit() only
colTPNumber = 2
colTDNumber = 3
rows = sheet.iter_rows(min_row=1, max_row=1)
firstRow = next(rows)
for i in firstRow:
if str(i.value) == "TP":
colTPNumber = i.column
elif str(i.value) == "TD":
colTDNumber = i.column
else:
print ("Column name not found")
#break
sys.exit()
#OR
exit()
Some more info:
You can also use quit()
quit() raises the SystemExit exception behind the scenes.
Furthermore, if you print it, it will give a message:
>>> print (quit) 
Use quit() or Ctrl-Z plus Return to exit
This functionality was included to help people who do not know Python. After all, one of the most likely things a newbie will try to exit Python is typing in quit.
Nevertheless, quit should not be used in production code. This is because it only works if the site module is loaded. Instead, this function should only be used in the interpreter.
exit() is an alias for quit (or vice-versa). They exist together simply to make Python more user-friendly.
Furthermore, it too gives a message when printed:
>>> print (exit) 
Use exit() or Ctrl-Z plus Return to exit
However, like quit, exit is considered bad to use in production code and should be reserved for use in the interpreter. This is because it too relies on the site module.
sys.exit raises the SystemExit exception in the background. This means that it is the same as quit and exit in that respect.
Unlike those two however, sys.exit() is considered good to use in production code. This is because the sys module will always be there.
os._exit exits the program without calling cleanup handlers, flushing stdio buffers, etc. Thus, it is not a standard way to exit and should only be used in special cases. The most common of these is in the child process(es) created by os.fork.
Note that, of the four methods given, only this one is unique in what it does.
Summed up, all four methods exit the program. However, the first two are considered bad to use in production code and the last is a non-standard, dirty way that is only used in special scenarios.
So, if you want to exit a program normally, go with the third method: sys.exit.
Or, even better in my opinion, you can just do directly what sys.exit does behind the scenes and run:
raise SystemExit
This way, you do not need to import sys first.
However, this choice is simply one on style and is purely up to you
Adapted from: https://www.edureka.co/community/16988/python-exit-commands-why-so-many-and-when-should-each-be-used
More info on sys.exit():
sys.exit([arg])
Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions specified by finally clauses of try statements are honored, and it is possible to intercept the exit attempt at an outer level.
The optional argument arg can be an integer giving the exit status (defaulting to zero), or another type of object. If it is an integer, zero is considered “successful termination” and any nonzero value is considered “abnormal termination” by shells and the like. Most systems require it to be in the range 0–127, and produce undefined results otherwise. Some systems have a convention for assigning specific meanings to specific exit codes, but these are generally underdeveloped; Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors. If another type of object is passed, None is equivalent to passing zero, and any other object is printed to stderr and results in an exit code of 1. In particular, sys.exit("some error message") is a quick way to exit a program when an error occurs.
Since exit() ultimately “only” raises an exception, it will only exit the process when called from the main thread, and the exception is not intercepted.
Changed in version 3.6: If an error occurs in the cleanup after the Python interpreter has caught SystemExit (such as an error flushing buffered data in the standard streams), the exit status is changed to 120.
From https://docs.python.org/3/library/sys.html

Give the variables invalid default values, so you can check them after the loop.
Then you can call exit() if they're not set correctly, this will terminate the entire script.
colTPNumber = None
colTDNumber = None
rows = sheet.iter_rows(min_row=1, max_row=1)
firstRow = next(rows)
for i in firstRow:
if str(i.value) == "TP":
colTPNumber = i.column
elif str(i.value) == "TD":
colTDNumber = i.column
if colTPNumber is None or colTDNumber is None:
print("Column name not found")
exit(1)
See Terminating a Python script

Try using exit() instead of break. break ends a loop, but exit() actually exits your program.

What's a good way to detect when a Python program exits or crashes?

I have the following Python program running in a Docker container.
Basically, if the Python process exits gracefully (ex. when I manually stop the container) or if the Python process crashes (while inside some_other_module.do_work()) then I need to do some cleanup and ping my DB telling it that process has exited.
What's the best way to accomplish this? I saw one answer where they did a try catch on main(), but that seems a bit odd.
My code:
def main():
some_other_module.do_work()
if __name__ == '__main__':
main()

I assume that the additional cleanup will be done by a different process, since the main process has likely crashed in a not recoverable way (I understood the question in this way).
The simplest way would be that the main process sets a flag somewhere (maybe creates a file in a specified location, or a column value in a database table; could also include the PID of the main process that sets the flag) when it starts and removes (or un-sets) that same flag if it finishes gracefully.
The cleanup process just needs to check the flag:
if the flag is set but the main process has ended already (the flag could contain the PID of the main process, so the cleanup process uses that to find if the main process is still running or not), then a cleanup is in order.
if the flag is set and the main process is running, then nothing is to be done.
if the flag is not set, then nothing is to be done.

Try-catch on main seems simplest, but doesn't/may not work for most things (please see comments below). You can always except specific exceptions:
def main():
some_other_module.do_work()
if __name__ == '__main__':
try:
main()
except Exception as e:
if e == "<INSERT GRACEFUL INTERRUPT HERE>":
# finished gracefully
else:
print(e)
# crash

Use a try/except
def thing_that_crashes():
exit()
try:
thing_that_crashes()
except:
print('oh and by the way, that thing tried to kill me')
I think it is impossible to catch a process with advanced suicidal behaviour (I don't know sending a SYGKILL to itself or something) so if you need your main process to live whatever happens, maybe run the other one in a subprocess.

You could wrap your script with another subprocess script and check the returncode. Inspired by this Relevant question.
from subprocess import Popen
script = Popen("python abspath/to/your/script.py")
script.communicate()
if script.returncode <> 0:
# something went wrong
# do something about it

Does 'finally' always execute in Python?

For any possible try-finally block in Python, is it guaranteed that the finally block will always be executed?
For example, let’s say I return while in an except block:
try:
1/0
except ZeroDivisionError:
return
finally:
print("Does this code run?")
Or maybe I re-raise an Exception:
try:
1/0
except ZeroDivisionError:
raise
finally:
print("What about this code?")
Testing shows that finally does get executed for the above examples, but I imagine there are other scenarios I haven't thought of.
Are there any scenarios in which a finally block can fail to execute in Python?

"Guaranteed" is a much stronger word than any implementation of finally deserves. What is guaranteed is that if execution flows out of the whole try-finally construct, it will pass through the finally to do so. What is not guaranteed is that execution will flow out of the try-finally.
A finally in a generator or async coroutine might never run, if the object never executes to conclusion. There are a lot of ways that could happen; here's one:
def gen(text):
try:
for line in text:
try:
yield int(line)
except:
# Ignore blank lines - but catch too much!
pass
finally:
print('Doing important cleanup')
text = ['1', '', '2', '', '3']
if any(n > 1 for n in gen(text)):
print('Found a number')
print('Oops, no cleanup.')
Note that this example is a bit tricky: when the generator is garbage collected, Python attempts to run the finally block by throwing in a GeneratorExit exception, but here we catch that exception and then yield again, at which point Python prints a warning ("generator ignored GeneratorExit") and gives up. See PEP 342 (Coroutines via Enhanced Generators) for details.
Other ways a generator or coroutine might not execute to conclusion include if the object is just never GC'ed (yes, that's possible, even in CPython), or if an async with awaits in __aexit__, or if the object awaits or yields in a finally block. This list is not intended to be exhaustive.
A finally in a daemon thread might never execute if all non-daemon threads exit first.
os._exit will halt the process immediately without executing finally blocks.
os.fork may cause finally blocks to execute twice. As well as just the normal problems you'd expect from things happening twice, this could cause concurrent access conflicts (crashes, stalls, ...) if access to shared resources is not correctly synchronized.
Since multiprocessing uses fork-without-exec to create worker processes when using the fork start method (the default on Unix), and then calls os._exit in the worker once the worker's job is done, finally and multiprocessing interaction can be problematic (example).
A C-level segmentation fault will prevent finally blocks from running.
kill -SIGKILL will prevent finally blocks from running. SIGTERM and SIGHUP will also prevent finally blocks from running unless you install a handler to control the shutdown yourself; by default, Python does not handle SIGTERM or SIGHUP.
An exception in finally can prevent cleanup from completing. One particularly noteworthy case is if the user hits control-C just as we're starting to execute the finally block. Python will raise a KeyboardInterrupt and skip every line of the finally block's contents. (KeyboardInterrupt-safe code is very hard to write).
If the computer loses power, or if it hibernates and doesn't wake up, finally blocks won't run.
The finally block is not a transaction system; it doesn't provide atomicity guarantees or anything of the sort. Some of these examples might seem obvious, but it's easy to forget such things can happen and rely on finally for too much.

Yes. Finally always wins.
The only way to defeat it is to halt execution before finally: gets a chance to execute (e.g. crash the interpreter, turn off your computer, suspend a generator forever).
I imagine there are other scenarios I haven't thought of.
Here are a couple more you may not have thought about:
def foo():
# finally always wins
try:
return 1
finally:
return 2
def bar():
# even if he has to eat an unhandled exception, finally wins
try:
raise Exception('boom')
finally:
return 'no boom'
Depending on how you quit the interpreter, sometimes you can "cancel" finally, but not like this:
>>> import sys
>>> try:
... sys.exit()
... finally:
... print('finally wins!')
...
finally wins!
$
Using the precarious os._exit (this falls under "crash the interpreter" in my opinion):
>>> import os
>>> try:
... os._exit(1)
... finally:
... print('finally!')
...
$
I'm currently running this code, to test if finally will still execute after the heat death of the universe:
try:
while True:
sleep(1)
finally:
print('done')
However, I'm still waiting on the result, so check back here later.

According to the Python documentation:
No matter what happened previously, the final-block is executed once the code block is complete and any raised exceptions handled. Even if there's an error in an exception handler or the else-block and a new exception is raised, the code in the final-block is still run.
It should also be noted that if there are multiple return statements, including one in the finally block, then the finally block return is the only one that will execute.

Well, yes and no.
What is guaranteed is that Python will always try to execute the finally block. In the case where you return from the block or raise an uncaught exception, the finally block is executed just before actually returning or raising the exception.
(what you could have controlled yourself by simply running the code in your question)
The only case I can imagine where the finally block will not be executed is when the Python interpretor itself crashes for example inside C code or because of power outage.

I found this one without using a generator function:
import multiprocessing
import time
def fun(arg):
try:
print("tried " + str(arg))
time.sleep(arg)
finally:
print("finally cleaned up " + str(arg))
return foo
list = [1, 2, 3]
multiprocessing.Pool().map(fun, list)
The sleep can be any code that might run for inconsistent amounts of time.
What appears to be happening here is that the first parallel process to finish leaves the try block successfully, but then attempts to return from the function a value (foo) that hasn't been defined anywhere, which causes an exception. That exception kills the map without allowing the other processes to reach their finally blocks.
Also, if you add the line bar = bazz just after the sleep() call in the try block. Then the first process to reach that line throws an exception (because bazz isn't defined), which causes its own finally block to be run, but then kills the map, causing the other try blocks to disappear without reaching their finally blocks, and the first process not to reach its return statement, either.
What this means for Python multiprocessing is that you can't trust the exception-handling mechanism to clean up resources in all processes if even one of the processes can have an exception. Additional signal handling or managing the resources outside the multiprocessing map call would be necessary.

You can use a finally with an if statement, below example is checking for network connection and if its connected it will run the finally block
try:
reader1, writer1 = loop.run_until_complete(self.init_socket(loop))
x = 'connected'
except:
print("cant connect server transfer") #open popup
x = 'failed'
finally :
if x == 'connected':
with open('text_file1.txt', "r") as f:
file_lines = eval(str(f.read()))
else:
print("not connected")

Methods to restart died process

There's a little app named logivew that I'm writing a script to monitor, along with some other tasks. In the main while loop (which will exit when the app I'm most concerned about closes), I check to see if logview needs restarting. The code I have presently is roughly as follows:
#a good old global
logview = "/usr/bin/logview"
#a function that starts logview:
port = 100
log_file = "/foo/bar"
logview_process = subprocess.Popen([logview, log_file, port],
stdout = subprocess.DEVNULL,
stderr = subprocess.STDOUT)
#a separate function that monitors in the background:
while True:
time.sleep(1)
logview_status = 0
try:
logview_status = psutil.Process(logview_process.pid).status()
except psutil.NoSuchProcess:
pass
if(logview_status == psutil.STATUS_STOPPED or
logview_status == psutil.STATUS_ZOMBIE or
logview_status == psutil.STATUS_DEAD or
logview_status == 0):
print("Logview died; restarting")
logview_cli_list = [logview]
logview_cli_list.extend(logview_process.args)
logview_process = subprocess.Popen(logview_cli_list,
stdout = subprocess.DEVNULL,
stderr = subprocess.STDOUT)
if(some_other_condition): break
However, if I test-kill logview, the condition triggers and I do see the printed message, but then I see it again, and again, and again. It seems that the condition triggers every single iteration of the loop if logview does die. And, it never does get restarted properly.
So clearly... I'm doing something wrong. =)
Any help (or better methods!) would be greatly appreciated.

I don't know your logview program but the problem is here:
logview_cli_list = [logview]
logview_cli_list.extend(logview_process.args)
When you're creating the argument list, you're putting logview twice in your command, because logview_process.args also contains the name of the launched command, so the program probably fails immediately because of bad args, and is run again and again...
The fix is then obvious:
logview_cli_list = logview_process.args
a better fix would be to create the process in the loop if a given flag is set and set the flag at the start.
When process dies, set the flag to trigger the process creation again. Would have avoided this copy/almost paste mistake.

Exit Python program if data directory is empty

UPDATE: I think I may have just realized what I need to figure out re: the below, which is the correct error type to specify with the except clause (new to this, obviously)
Starting in a specified root directory, my program iterates through subdirectories, then files within those, identifying valid (.csv) data files and then aggregating and performing calculations on the data.
In cases where the root directory happens to be empty, can someone suggest a clean/graceful way to simply exit the program at the start without further processing?
I tried to adapt a suggestion I found here, but it didn't work as I expected:
Exit gracefully if file doesn't exist
That person's code was:
def main():
try:
file = open('file.txt', 'r')
except IOError:
print('There was an error opening the file!')
return
I gave the above a try, and it works for the particular case above. However, when I tried to adapt it as follows, it 'broke' and I got an "Index out of range error", instead dropping down to the except code.
dir = os.listdir(masterDirPath)
def main():
try:
item = dir[0]
except IOError:
print('The data area is empty.')
return
(Also/instead, I very much welcome suggestions for some completely other approach to the task overall)

To exit your program immediately you should use either exit() or quit(). Instead of throwing an error you could use the fact that many objects in Python are truthy; an empty list is False, and a list with one or more elements is True.
import os
dir_contents = os.listdir('.')
if dir_contents:
do_stuff
else:
print('Directory was empty. Exiting')
exit()
If you prefer explicitness to implicitness you could also check the length of your list using len(dir_contents) before indexing into it.
You might also want to avoid using dir in Python as a variable name as it will shadow the builtin function dir().

An empty list has no elements so you should catch IndexError instead of IOError.
def main():
try:
item = dir[0]
except IndexError:
print('The data area is empty.')
return

Why are you using an exception?
if not dir:
print('The data area is empty')
exit(0)
item = dir[0]

#!/usr/bin/env python
import os
path = 'path-to-your-dir'
if os.listdir(path) == []:
exit(0)
else:
print "Your other code goes here."

These are the exit commands and its definitions
exit(0) means a clean exit without any errors / problems
exit(1) means there was some issue / error / problem and that is why the program is exiting.
os._exit() for child processes
quit() the SystemExit exception behind the scenes.
sys.exit() kill the interpreter