NB: I have not attempted to reproduce the problem described below under Windows, or with versions of Python other than 2.7.3.
The most reliable way to elicit the problem in question is to pipe the output of the following test script through : (under bash):
try:
for n in range(20):
print n
except:
pass
I.e.:
% python testscript.py | :
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
My question is:
How can I modify the test script above to avoid the error message when the script is run as shown (under Unix/bash)?
(As the test script shows, the error cannot be trapped with a try-except.)
The example above is, admittedly, highly artificial, but I'm running into the same problem sometimes when the output of a script of mine is piped through some 3rd party software.
The error message is certainly harmless, but it is disconcerting to end-users, so I would like to silence it.
EDIT: The following script, which differs from the original one above only in that it redefines sys.excepthook, behaves exactly like the one given above.
import sys
STDERR = sys.stderr
def excepthook(*args):
print >> STDERR, 'caught'
print >> STDERR, args
sys.excepthook = excepthook
try:
for n in range(20):
print n
except:
pass
How can I modify the test script above to avoid the error message when the script is run as shown (under Unix/bash)?
You will need to prevent the script from writing anything to standard output. That means removing any print statements and any use of sys.stdout.write, as well as any code that calls those.
The reason this is happening is that you're piping a nonzero amount of output from your Python script to something which never reads from standard input. This is not unique to the : command; you can get the same result by piping to any command which doesn't read standard input, such as
python testscript.py | cd .
Or for a simpler example, consider a script printer.py containing nothing more than
print 'abcde'
Then
python printer.py | python printer.py
will produce the same error.
When you pipe the output of one program into another, the output produced by the writing program gets backed up in a buffer, and waits for the reading program to request that data from the buffer. As long as the buffer is nonempty, any attempt to close the writing file object is supposed to fail with an error. This is the root cause of the messages you're seeing.
The specific code that triggers the error is in the C language implementation of Python, which explains why you can't catch it with a try/except block: it runs after the contents of your script has finished processing. Basically, while Python is shutting itself down, it attempts to close stdout, but that fails because there is still buffered output waiting to be read. So Python tries to report this error as it would normally, but sys.excepthook has already been removed as part of the finalization procedure, so that fails. Python then tries to print a message to sys.stderr, but that has already been deallocated so again, it fails. The reason you see the messages on the screen is that the Python code does contain a contingency fprintf to write out some output to the file pointer directly, even if Python's output object doesn't exist.
Technical details
For those interested in the details of this procedure, let's take a look at the Python interpreter's shutdown sequence, which is implemented in the Py_Finalize function of pythonrun.c.
After invoking exit hooks and shutting down threads, the finalization code calls PyImport_Cleanup to finalize and deallocate all imported modules. The next-to-last task performed by this function is removing the sys module, which mainly consists of calling _PyModule_Clear to clear all the entries in the module's dictionary - including, in particular, the standard stream objects (the Python objects) such as stdout and stderr.
When a value is removed from a dictionary or replaced by a new value, its reference count is decremented using the Py_DECREF macro. Objects whose reference count reaches zero become eligible for deallocation. Since the sys module holds the last remaining references to the standard stream objects, when those references are unset by _PyModule_Clear, they are then ready to be deallocated.1
Deallocation of a Python file object is accomplished by the file_dealloc function in fileobject.c. This first invokes the Python file object's close method using the aptly-named close_the_file function:
ret = close_the_file(f);
For a standard file object, close_the_file(f) delegates to the C fclose function, which sets an error condition if there is still data to be written to the file pointer. file_dealloc then checks for that error condition and prints the first message you see:
if (!ret) {
PySys_WriteStderr("close failed in file object destructor:\n");
PyErr_Print();
}
else {
Py_DECREF(ret);
}
After printing that message, Python then attempts to display the exception using PyErr_Print. That delegates to PyErr_PrintEx, and as part of its functionality, PyErr_PrintEx attempts to access the Python exception printer from sys.excepthook.
hook = PySys_GetObject("excepthook");
This would be fine if done in the normal course of a Python program, but in this situation, sys.excepthook has already been cleared.2 Python checks for this error condition and prints the second message as a notification.
if (hook && hook != Py_None) {
...
} else {
PySys_WriteStderr("sys.excepthook is missing\n");
PyErr_Display(exception, v, tb);
}
After notifying us about the missing excepthook, Python then falls back to printing the exception info using PyErr_Display, which is the default method for displaying a stack trace. The very first thing this function does is try to access sys.stderr.
PyObject *f = PySys_GetObject("stderr");
In this case, that doesn't work because sys.stderr has already been cleared and is inaccessible.3 So the code invokes fprintf directly to send the third message to the C standard error stream.
if (f == NULL || f == Py_None)
fprintf(stderr, "lost sys.stderr\n");
Interestingly, the behavior is a little different in Python 3.4+ because the finalization procedure now explicitly flushes the standard output and error streams before builtin modules are cleared. This way, if you have data waiting to be written, you get an error that explicitly signals that condition, rather than an "accidental" failure in the normal finalization procedure. Also, if you run
python printer.py | python printer.py
using Python 3.4 (after putting parentheses on the print statement of course), you don't get any error at all. I suppose the second invocation of Python may be consuming standard input for some reason, but that's a whole separate issue.
1Actually, that's a lie. Python's import mechanism caches a copy of each imported module's dictionary, which is not released until _PyImport_Fini runs, later in the implementation of Py_Finalize, and that's when the last references to the standard stream objects disappear. Once the reference count reaches zero, Py_DECREF deallocates the objects immediately. But all that matters for the main answer is that the references are removed from the sys module's dictionary and then deallocated sometime later.
2Again, this is because the sys module's dictionary is cleared completely before anything is really deallocated, thanks to the attribute caching mechanism. You can run Python with the -vv option to see all the module's attributes being unset before you get the error message about closing the file pointer.
3This particular piece of behavior is the only part that doesn't make sense unless you know about the attribute caching mechanism mentioned in previous footnotes.
I ran into this sort of issue myself today and went looking for an answer. I think a simple workaround here is to ensure you flush stdio first, so python blocks instead of failing during script shutdown. For example:
--- a/testscript.py
+++ b/testscript.py
## -9,5 +9,6 ## sys.excepthook = excepthook
try:
for n in range(20):
print n
+ sys.stdout.flush()
except:
pass
Then with this script nothing happens, as the exception (IOError: [Errno 32] Broken pipe) is suppressed by the try...except.
$ python testscript.py | :
$
In your program throws an exception that can not be caught using try/except block. To catch him, override function sys.excepthook:
import sys
sys.excepthook = lambda *args: None
From documentation:
sys.excepthook(type, value, traceback)
When an exception is raised and uncaught, the interpreter calls
sys.excepthook with three arguments, the exception class, exception
instance, and a traceback object. In an interactive session this
happens just before control is returned to the prompt; in a Python
program this happens just before the program exits. The handling of
such top-level exceptions can be customized by assigning another
three-argument function to sys.excepthook.
Illustrative example:
import sys
import logging
def log_uncaught_exceptions(exception_type, exception, tb):
logging.critical(''.join(traceback.format_tb(tb)))
logging.critical('{0}: {1}'.format(exception_type, exception))
sys.excepthook = log_uncaught_exceptions
I realize that this is an old question, but I found it in a Google search for the error. In my case it was a coding error. One of my last statements was:
print "Good Bye"
The solution was simply fixing the syntax to:
print ("Good Bye")
[Raspberry Pi Zero, Python 2.7.9]
Related
I am passing "sys.stdout" as an argument to a process, and the process then writes to the "sys.stdout" while it does its stuff.
import multiprocessing
import sys
def worker_with(stream):
stream.write('In the process\n')
if __name__ == '__main__':
sys.stdout.write('In the main\n')
lock = multiprocessing.Lock()
w = multiprocessing.Process(target=worker_with, args=(sys.stdout,))
w.start()
w.join()
The code above does not work, it returns the following error: "ValueError: operation on closed file".
I tried running the same code but calling the function directly instead of spawning a process and it works, it prints out to the console.
I also tried running the same code but calling directly sys.stdout inside the function, spawn it as a process and it works.
The problem seems to be passing sys.stout as a parameter of the process.
Does someone haveany idea why ?
Note: this code is inspired by the tutorial PYMOTW - communication between processes.
EDIT : i am running Python 2.7.10, 32 bits on Windows7.
When you pass arguments to a Process, they are pickled in the parent, transmitted to the child, and unpickled there. Unfortunately, it looks like the round trip through pickle silently misbehaves for file objects; with protocol 0, it errors out, but with protocol 2 (the highest Python 2 protocol, and the one used for multiprocessing), it silently produces a junk file object:
>>> import pickle, sys
>>> pickle.loads(pickle.dumps(sys.stdout, pickle.HIGHEST_PROTOCOL))
<closed file '<uninitialized file>', mode '<uninitialized file>' at 0xDEADBEEF>
Same problem occurs for named files too; it's not unique to the standard handles. Basically, pickle can't round trip a file object; even when it claims to succeed, the result is garbage.
Generally, multiprocessing isn't really expected to handle a scenario like this; usually, Processes are worker tasks, and I/O is performed through the main process (because if they all wrote independently to the same file handle, you'd have issues with interleaved writes).
In Python 3.5 at least, they fixed this so the error is immediate and obvious (the file-like objects returned by open, TextIOWrapper and Buffered*, will error out when pickled with any protocol).
The best you could do on Windows would be to send the known file descriptor as an argument:
sys.stdout.flush() # Precaution to minimize output interleaving
w = multiprocessing.Process(target=worker_with, args=(sys.stdout.fileno(),))
then reopen it on the other side using os.fdopen. For fds not part of the standard handles (0, 1 and 2), since Windows uses the "spawn" method of making new Processes, you'd need to make sure any such fd was opened as a consequence of importing the __main__ module when __name__ != "__main__" (Windows simulates a fork by importing the __main__ module, setting the __name__ to something else). Of course, if it's a named file, not a standard handle, you could just pass the name and reopen that. For example, to make this work, you'd change:
def worker_with(stream):
stream.write('In the process\n')
to:
import os
def worker_with(toopen):
opener = open if isinstance(toopen, basestring) else os.fdopen
with opener(toopen, 'a') as stream:
stream.write('In the process\n')
Note: As written, if the fd is for one of the standard handles, os.fdopen will close the underlying file descriptor when the with statement exits, which may not be what you want. If you need file descriptors to survive the close of the with block, when passed a file descriptor, you may want to use os.dup to duplicate the handle before calling os.fdopen, so the two handles are independent of one another.
Other solutions would include writing results back to the main process over a multiprocessing.Pipe (so the main process is responsible for passing the data along to sys.stdout, possibly launching a thread to perform this work asynchronously), or using higher level constructs (e.g. multiprocessing.Pool().*map*) that return data using return statement instead of explicit file I/O.
If you're really desperate to make this work in general for all file descriptors (and don't care about portability), not just the standard handles and descriptors created on import of __main__, you can use the undocumented Windows utility function multiprocessing.forking.duplicate that is used to explicitly duplicate a file descriptor from one process to another; it would be incredibly hacky (you'd need to look at the rest of the Windows definition of multiprocessing.forking.Popen there to see how it would be used), but it would at least allow passing along arbitrary file descriptors, not just statically opened ones.
I want to embed C++ in python application. I don't want to use Boost library.
If C++ function do assertion, I want to catch it and print error in my python application or get some detailed information like line number in python script that caused error. and main thing is "I want to proceed further in python execution flow"
How can I do it? I can't find any functions to get detailed assertion information in Python API or C++.
C++ Code
void sum(int iA, int iB)
{
assert(iA + iB >10);
}
Python Code
from ctypes import *
mydll = WinDLL("C:\\Users\\cppwrapper.dll")
try:
mydll.sum(10,3)
catch:
print "exception occurred"
# control should go to user whether exceptions occurs, after exception occurs if he provide yes then continue with below or else abort execution, I need help in this part as well
import re
for test_string in ['555-1212', 'ILL-EGAL']:
if re.match(r'^\d{3}-\d{4}$', test_string):
print test_string, 'is a valid US local phone number'
else:
print test_string, 'rejected'
Thanks in advance.
This can't really be done in exactly the way you say, (as was also pointed out in the comments).
Once the assertion happens and SIGABRT is sent to the process, it's in the operating system's hands what will happen, and generally the process will be killed.
The simplest way to recover from a process being killed, is to have the process launched by an external process. Like, a secondary python script, or a shell script. It's easy in bash scripting, for instance, to launch another process, check if it terminates normally or is aborted, log it, and continue.
For instance here's some bash code that executes command line $command, logs the standard error channel to a log file, checks the return code (which will be 130 or something for an SIGABRT) and does something in the various cases:
$command 2> error.log
error_code="$?"
if check_errs $error_code; then
# Do something...
return 0
else
# Do something else...
return 1
fi
Where check_errs is some other subroutine that you would write.
I have a ruby script that gets executed by a python script. From within the python script I want to access to return value of the ruby function.
Imagine, I would have this ruby script test.rb:
class TestClass
def self.test_function(some_var)
if case1
puts "This may take some time"
# something is done here with some_var
puts "Finished"
else
# just do something short with some_var
end
return some_var
end
end
Now, I want to get the return value of that function into my python script, the printed output should go to stdout.
I tried the following (example 1):
from subprocess import call
answer = call(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
However, this gives me the whole output on stdout and answer is just the exit code.
Therefore i tried this (example 2):
from subprocess import check_output
answer = check_output(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
This gives me the return value of the function in the else case (see test.rb) almost immediately. However, if case1 is true, answer contains the whole output, but while running test.rb nothing gets printed.
Is there any way to get the return value of the ruby function and the statements printed to stdout? Ideally, the solution requires no additional modules to install. Furthermore, I can't change the ruby code.
Edit:
Also tried this, but this also gives no output on stdout while running the ruby script (example 3):
import subprocess
process = subprocess.Popen(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"], stdout=subprocess.PIPE)
answer = process.communicate()
I also think that this is no matter of flushing the output to stdout in the ruby script. Example 1 gives me the output immediately.
Another way of doing this, without trying to call the ruby script as an external process is to set up a xmlrpc (or jsonrpc) server with the Ruby script, and call the remote functions from Python jsonrpc client (or xmlrpc)- the value would be available inside the Python program, nad even the sntax used would be just like you were dealing with a normal Python function.
Setting up such a server to expose a couple of functions remotely is very easy in Python, and should be the same from Ruby, but I had never tried it.
Check out http://docs.python.org/library/subprocess.html#popen-constructor and look into the ruby means of flushing stdout.
I would like to get the output (print statements) and return status from a python subprocess along with displaying logging info. I am using Python 2.5 on Windows XP. How do I get the return code as well? Here in this extract, I am reading output so that I can redirect it to display in a wxPython TextCtrl in a separate thread.
self.myproc.poll()
if self.myproc.returncode is None:
# Still running so check stdout
txt = self.myproc.stdout.read(self._readblock)
if txt:
# Add to UI's update queue
self._parent.AppendUpdate(txt)
else:
break
The returncode gets returned by the poll call, and is also (once poll did not return None) accessible via the returncode attribute. You are already using this in your code, so I'm not sure what your problem with that is.
When you want to update your control while your process is running, use readline instead of read: the latter will wait for the entire output to be present, while the former will wait on a newline character. To give a full example using your variable names:
from subprocess import Popen, PIPE
self.myproc = Popen('dir /s', shell=True, stdout=PIPE, stderr=PIPE)
while self.myproc.poll() == None:
self._parent.AppendUpdate(self.myproc.stdout.readline())
self._parent.AppendUpdate('Return code was ' + self.myproc.returncode)
Note that you might want to .strip() the readline result as it will contain the newline character.
Edit: To address your confusion between the script output and its return code, the following. First, a Python script at module level cannot return a value: that would yield a syntax error. So what you should differentiate between is the output of the script, and the return code it gives.
The script's output is in the above example read by using the self.myproc.stdout.readline method. Every time the external process has produced a line of outputted text, calling that function would retrieve it.
The return code however (or exit status) is an integer that gets passed from a child process to the parent (your) process, indicating the state in which the child process exited. In Python, you do this with the sys.exit function. Most commonly, when this value is zero it indicates success; a non-zero value depicts some kind of error.
Say your child process script looks like this:
import sys
# Do some stuff
print 'pass' # Gets to stdout
sys.exit(0) # Return code
Executing this external file (let's call it test.py) with the Popen class, we will get pass when we read out self.myproc.stdout, and 0 when we read out self.myproc.poll (or self.myproc.returncode after the first poll).
The purpose of this return code is that you do not have to parse all of the child process' output to determine if it succeeded in its job: you are free to define your own exit codes. For example, you could consider 0 to be a success, 1 to be a failure, 2 to be some kind of invalid input given, 9 an unknown error, and so on. That way, you can just keep polling the process, and based on the exit code returned by that poll you directly know if it was successful. Note that this is a bit less applicable to your case as you need the output of the child process anyway, but still it is easier to parse a number than a string to determine success.
I'd like to find a way to log every error that forces the python interpreter to quit to be saved to a file as well as being printed to the screen. The reason I would like to do this is that I want to keep stats on the types of errors I make while writing code, with an eye towards finding ways to avoid mistakes I make commonly in the future.
I've been attempting to do this by writing a wrapper for the python interpreter using the subprocess module. Basically, it runs the python interpreter, captures any output, parse and saves it to a file, prints the output, and use matplotlib to make some summary figures. However, I'm having a problem getting output from my wrapper script in real time. For example, if the script I'm running is:
import os
import time
for x in range(10):
print "testing"
time.sleep(10)
and I'm using subprocess.Popen() with p.communicate(), the wrapper will wait 100 seconds, and then print all of the output. I'd like the wrapper to be as invisible as possible - ideally in this case it would print "testing" once every ten seconds.
If someone could point me towards a good way of doing this, I'd greatly appreciate it.
Thanks!
I believe you can simply replace sys.excepthook with your own function. You can read about it in the Python documentation.
Basically, it allows you to customize what happens when an exception percolates up to the point of forcing the Python interpreter to quit. You use it like this:
import sys
def my_excepthook(type, value, tb):
# you can log the exception to a file here
print 'In My Exception Handler'
# the following line does the default (prints it to err)
sys.__excepthook__(type, value, tb)
sys.excepthook = my_excepthook
You'll probably also want to look at the traceback module, for formatting the traceback you get.