Develop Reference

How to print the reason for program exit / program termination / exit errors?

I managed to catch the termination with:
atexit.register(exitHandler)
But how to print out what happened on that point? I want to see if the program aborts due to an error, due to Crtl-C or normal stopping...

It's not the complete solution - but you could wrap code in try except
try:
YOUR CODE HERE
except Exception as e:
print(e)

You should catch KeyboardInterrupt for Ctrl-C. E.g.:
import sys
try:
# your code
except KeyboardInterrupt:
sys.exit('Abort by user interrupt')
except Exception as exc:
sys.exit(f'Abort on error: {exc}')

Hive UDF with Python - Print exception on shell

In the code below, Whenever exception is caught, I would like to exit from the program & print the exception on shell
#stream.py
import hashlib
import sys
import os
import importlib
for line in sys.stdin.readlines():
try:
inpFile = "temp.py"
execfile(inpFile)
line1 = line.strip('\n').split('\t')
print "\t".join(line1)
except:
#exception expected "temp.py:File Not Found", how do I exit the code & print the exception on console ?
sys.exit(1)
Here is the Transform query to call the UDF:
Create table newtable as Select TRANSFORM(id,name) USING
'python stream.py' as (id,name) from mytable;
ideas appreciated.

if you want to catch a specific type of exception (for examle an IOError) you can use
except IOError as e:
and access the error string with e.strerror
if you want to catch all exception you can use sys.exc_info()[0] to access the last error message
for example
try:
1/0
except:
print sys.exc_info()[0]
would print
<type 'exceptions.ZeroDivisionError'>

If you are expecting a specific exception then you should trap that, not everything, which is what you are doing. But there is nothing in your code to generate a "File Not Found" anyway!
EDIT: question code changed! We really do wish to trap "everything". I am using Exception, which traps "almost" everything, see https://docs.python.org/2/library/exceptions.html
This uses Python 2 syntax:
import sys
for line in sys.stdin.readlines():
try:
inpFile = "temp.py"
execfile(inpFile)
line1 = line.strip('\n').split('\t')
print "\t".join(line1)
except Exception as err:
print >> sys.stderr, err # print to stderr (python 2 syntax)
sys.exit(1)

How to avoid a Broken Pipe error when printing a large amount of formatted data?

I am trying to print a list of tuples formatted in my stdout. For this, I use the str.format method. Everything works fine, but when I pipe the output to see the
first lines using the head command a IOError occurs.
Here is my code:
# creating the data
data = []$
for i in range(0, 1000):
pid = 'pid%d' % i
uid = 'uid%d' % i
pname = 'pname%d' % i
data.append( (pid, uid, pname) )
# find max leghed string for each field
pids, uids, pnames = zip(*data)
max_pid = len("%s" % max( pids) )
max_uid = len("%s" % max( uids) )
max_pname = len("%s" % max( pnames) )
# my template for the formatted strings
template = "{0:%d}\t{1:%d}\t{2:%d}" % (max_pid, max_uid, max_pname)
# print the formatted output to stdout
for pid, uid, pname in data:
print template.format(pid, uid, pname)
And here is the error I get after running the command: python myscript.py | head
Traceback (most recent call last):
File "lala.py", line 16, in <module>
print template.format(pid, uid, pname)
IOError: [Errno 32] Broken pipe
Can anyone help me on this?
I tried to put print in a try-except block to handle the error,
but after that there was another message in the console:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
I also tried to flush immediately the data through a two consecutive
sys.stdout.write and sys.stdout.flush calls, but nothing happend..

head reads from stdout then closes it. This causes print to fail, internally it writes to sys.stdout, now closed.
You can simply catch the IOError and exit silently:
try:
for pid, uid, pname in data:
print template.format(pid, uid, pname)
except IOError:
# stdout is closed, no point in continuing
# Attempt to close them explicitly to prevent cleanup problems:
try:
sys.stdout.close()
except IOError:
pass
try:
sys.stderr.close()
except IOError:
pass

The behavior you are seeing is linked to the buffered output implementation in Python3. The problem can be avoided using the -u option or setting environmental variable PYTHONUNBUFFERED=x. See the man pages for more information on -u.
$ python2.7 testprint.py | echo
Exc: <type 'exceptions.IOError'>
$ python3.5 testprint.py | echo
Exc: <class 'BrokenPipeError'>
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
$ python3.5 -u testprint.py | echo
Exc: <class 'BrokenPipeError'>
$ export PYTHONUNBUFFERED=x
$ python3.5 testprint.py | echo
Exc: <class 'BrokenPipeError'>

In general, I try to catch the most specific error I can get away with. In this case it is BrokenPipeError:
try:
# I usually call a function here that generates all my output:
for pid, uid, pname in data:
print template.format(pid, uid, pname)
except BrokenPipeError as e:
pass # Ignore. Something like head is truncating output.
finally:
sys.stderr.close()
If this is at the end of execution, I find I only need to close sys.stderr. If I don't close sys.stderr, I'll get a BrokenPipeError but without a stack trace.
This seems to be the minimum fix for writing tools that output to pipelines.

Had this problem with Python3 and debug logging piped into head as well. If your script talks to the network or does file IO, simply dropping IOError's is not a good solution. Despite mentions here, I was not able to catch BrokenPipeError for some reason.
Found a blog post talking about restoring the default signal handler for sigpipe: http://newbebweb.blogspot.com/2012/02/python-head-ioerror-errno-32-broken.html
In short, you add the following to your script before the bulk of the output:
if log.isEnabledFor(logging.DEBUG): # optional
# set default handler to no-op
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE, SIG_DFL)
This seems to happen with head, but not other programs such as grep---as mentioned head closes stdout. If you don't use head with the script often, it may not be worth worrying about.

How to read named FIFO non-blockingly?

I create a FIFO, and periodically open it in read-only and non-blockingly mode from a.py:
os.mkfifo(cs_cmd_fifo_file, 0777)
io = os.open(fifo, os.O_RDONLY | os.O_NONBLOCK)
buffer = os.read(io, BUFFER_SIZE)
From b.py, open the fifo for writing:
out = open(fifo, 'w')
out.write('sth')
Then a.py will raise an error:
buffer = os.read(io, BUFFER_SIZE)
OSError: [Errno 11] Resource temporarily unavailable
Anyone know what's wrong?

According to the manpage of read(2):
EAGAIN or EWOULDBLOCK
The file descriptor fd refers to a socket and has been marked
nonblocking (O_NONBLOCK), and the read would block.
POSIX.1-2001 allows either error to be returned for this case,
and does not require these constants to have the same value, so
a portable application should check for both possibilities.
So what you're getting is that there is no data available for reading. It is safe to handle the error like this:
try:
buffer = os.read(io, BUFFER_SIZE)
except OSError as err:
if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
buffer = None
else:
raise # something else has happened -- better reraise
if buffer is None:
# nothing was received -- do something else
else:
# buffer contains some received data -- do something with it
Make sure you have the errno module imported: import errno.

out = open(fifo, 'w')
Who will close it for you?
Replace your open+write by this:
with open(fifo, 'w') as fp:
fp.write('sth')
UPD:
Ok, than just make this:
out = os.open(fifo, os.O_NONBLOCK | os.O_WRONLY)
os.write(out, 'tetet')

exit failed script run (python)

I have seen several questions about exiting a script after a task is successfully completed, but is there a way to do the same for a script which has failed? I am writing a testing script which just checks that a camera is functioning correctly. If the first test fails it is more than likely that the following tests will also fail; therefore, I want the first failure to invoke an exit and provide output to screen letting me know that there was an error.
I hope this is enough information; let me know if more details are required to help me.

Are you just looking for the exit() function?
import sys
if 1 < 0:
print >> sys.stderr, "Something is seriously wrong."
sys.exit(1)
The (optional) parameter of exit() is the return code the script will return to the shell. Usually values different than 0 signal an error.

You can use sys.exit() to exit. However, if any code higher up catches the SystemExit exception, it won't exit.

You can raise exceptions to identify error conditions. Your top-level code can catch those exceptions and handle them appropriately. You can use sys.exit to exit. E.g., in Python 2.x:
import sys
class CameraInitializationError(StandardError):
pass
def camera_test_1():
pass
def camera_test_2():
raise CameraInitializationError('Failed to initialize camera')
if __name__ == '__main__':
try:
camera_test_1()
camera_test_2()
print 'Camera successfully initialized'
except CameraInitializationError, e:
print >>sys.stderr, 'ERROR: %s' % e
sys.exit(1)

You want to check the return code from the c++ program you are running, and exit if it indicates failure. In the code below, /bin/false and /bin/true are programs that exit with error and success codes, respectively. Replace them with your own program.
import os
import sys
status = os.system('/bin/true')
if status != 0:
# Failure occurred, exit.
print 'true returned error'
sys.exit(1)
status = os.system('/bin/false')
if status != 0:
# Failure occurred, exit.
print 'false returned error'
sys.exit(1)
This assumes that the program you're running exits with zero on success, nonzero on failure.