Develop Reference

How to log error messages arrising from scripts in snakemake

I am trying to build a snakemake pipeline with custom python scripts.
Some of my scripts run into errors, leading to a shutdown of the pipeline.
However, while in the shell output I can barely see the end of the python error message that leads to the shutdown, this error is not logged anywhere. It is not logged in the snakemake.log that gets automatically created (only stating which script failed without giving the error message), and adding a "log: " with a folder to the rule that fails only creates an empty log.
Is there a way to access the error message, so I can solve the underlying issue?
Edit:
my current snakemake rule looks like this:
rule do_X:
input: "{Wildcard}_a"
output: "{wildcard}_b"
log: out = "{Wildcard}_stdout.log"
err = "{Wildcardd}_stderr.err"
shell: python script x.py {input}{output}
If the script fails, I recive empty logs,

The link provided by KeyboardCat's comment seems to work well.
Demonstration adapted from your code and the suggested solution:
Save this as script_x.py:
#script_x.py
print("test output in stdout made my script")
#import sys # WITH THIS COMMENTED OUT, YOU'LL GET TRACEBACK IN `test_stderr.err`
sys.stderr.write("This was made from std.err")
with open("test_b", "w") as out_file:
out_file.write("result\n")
Then your Snakefile is:
with open("test_a", "w") as out_file:
out_file.write("#trigger file")
rule all:
input: "test_b"
rule do_X:
input: "{wildcard}_a"
output: "{wildcard}_b"
log: out = "{wildcard}_stdout.log",
err = "{wildcard}_stderr.err"
shell: 'python script_x.py {input}{output} 2> {log.err} 1> {log.out}'
Execute snakemake so it uses the Snakefile. It will cause an issue because script_x.py has an error in it and fails.
In test_stderr.err, you'll see the traceback from when the script_x.py errored because sys wasn't imported as the script was written:
Traceback (most recent call last):
File "script_x.py", line 3, in <module>
sys.stderr.write("This was made from std.err")
NameError: name 'sys' is not defined
If you delete the files test_a and remove the # from in front of the import sys line inside script_x.py (line #3), it should run without error now and result in the text This was made from std.err showing up in the file test_stderr.err.
test_stdout.log will always end up with test output in stdout made by script in it because it gets created before the line containing the error in the script.
You say you tried the solution suggested at the link provided by KeyboardCat's comment; however, what did you actually try? Always best to include the variation you tried in the comment or add an update section to your OP.

Pylint doesnt execute

I have test.py file that I want to check using pylint in checking.py file with escaping refactor (R) and convention (C) messages. What I did wrong?
test.py
print("hello!")
checking.py
import pylint.lint
from pylint import epylint as lint
pylint_opts = ['--disable=R,C']
pylint.lint.Run(pylint_opts)
(pylint_stdout, pylint_stderr) = lint.py_run('test.py', return_std=True)
It returns me helping message assuming that I have typo when writing pylint options, but I wrote it ok as documentation says:
Messages control
-d <msg ids>, --disable=<msg ids>
Disable the message, report, category or checker with
the given id(s). You can either give multiple
identifiers separated by comma (,) or put this option
multiple times (only on the command line, not in the
configuration file where it should appear only once).
You can also use "--disable=all" to disable everything
first and then reenable specific checks. For example,
if you want to run only the similarities checker, you
can use "--disable=all --enable=similarities". If you
want to run only the classes checker, but have no
Warning level messages displayed, use "--disable=all
--enable=classes --disable=W".
There are 5 kind of message types :
* (C) convention, for programming standard violation
* (R) refactor, for bad code smell
* (W) warning, for python specific problems
* (E) error, for probable bugs in the code
* (F) fatal, if an error occurred which prevented pylint from doing
further processing.

epylint() does its own initialisation (calling Run) with sys.argv, ie. the arguments used when calling your checking.py
I guess you are just calling python checking.py, so sys.argv[1:] == [], which means in bash, you would be executing pylint (with any argument).
I think your solution is over-complicated. why don't you call pylint --disable R,C test.py on the command line ?
You could even write a .pylintrc :
[MESSAGES CONTROL]
disable=R,C
and just call pylint test.py

Strange Issue Using Logging Module in Python

I seem to be running into a problem when I am logging data after invoking another module in an application I am working on. I'd like assistance in understanding what may be happening here.
To replicate the issue, I have developed the following script...
#!/usr/bin/python
import sys
import logging
from oletools.olevba import VBA_Parser, VBA_Scanner
from cloghandler import ConcurrentRotatingFileHandler
# set up logger for application
dbg_h = logging.getLogger('dbg_log')
dbglog = '%s' % 'dbg.log'
dbg_rotateHandler = ConcurrentRotatingFileHandler(dbglog, "a")
dbg_h.addHandler(dbg_rotateHandler)
dbg_h.setLevel(logging.ERROR)
# read some document as a buffer
buff = sys.stdin.read()
# generate issue
dbg_h.error('Before call to module....')
vba = VBA_Parser('None', data=buff)
dbg_h.error('After call to module....')
When I run this, I get the following...
cat somedocument.doc | ./replicate.py
ERROR:dbg_log:After call to module....
For some reason, my last dbg_h logger write attempt is getting output to the console as well as getting written to my dbg.log file? This only appears to happen AFTER the call to VBA_Parser.
cat dbg.log
Before call to module....
After call to module....
Anyone have any idea as to why this might be happening? I reviewed the source code of olevba and did not see anything that stuck out to me specifically.
Could this be a problem I should raise with the module author? Or am I doing something wrong with how I am using the cloghandler?

The oletools codebase is littered with calls to the root logger though calls to logging.debug(...), logging.error(...), and so on. Since the author didn't bother to configure the root logger, the default behavior is to dump to sys.stderr. Since sys.stderr defaults to the console when running from the command line, you get what you're seeing.
You should contact the author of oletools since they're not using the logging system effectively. Ideally they would use a named logger and push the messages to that logger. As a work-around to suppress the messages you could configure the root logger to use your handler.
# Set a handler
logger.root.addHandler(dbg_rotateHandler)
Be aware that this may lead to duplicated log messages.

Python: NameError - what should I do with this?

I am working on a large-scale software system that is written in Python right now.
The thing is, I am not sure how to make sure if each individual .py file in the system is correct. The only way for me to run the software is to run the main.py file, which uses all the other .py files.
So either everything works, or one thing doesn't (causing everything to not work).
I keep getting a NameError even when importing the correct file. I think this may have to do with the fact that the class associated with that name in the NameError may have errors in it. Any suggestions? NameError is giving me this:
File "<string>", line 1, in <module>
NameError: name 'RGBox' is not defined
It's not a very helpful error message, and I'm not sure why it's giving "string" and 'module' instead of actual values.....
[EDIT]- I am working through ssh into a remote unix machine

This is a straight-forward error message which indicates that the execution flow has not yet encountered class/module/variable RGBox prior to it being called.
RGBox is either being called out of sequence or has been mispelt.
Perform a commandline search through the app files for the name 'RGBox' or its regex equivalents. for example with grep you can do a case-insensitive search:
$ grep -lsri 'rgbox' ./my_project_folder
which will output any file which contains the patterns 'RGBox', 'rgBox', etc.
If you are unfamiliar with the code and its structure, then you may as well insert strategic logging (or print) statements at significant locations in the code to understand its flow and execution logic.

Showing the stack trace from a running Python application

I have this Python application that gets stuck from time to time and I can't find out where.
Is there any way to signal Python interpreter to show you the exact code that's running?
Some kind of on-the-fly stacktrace?
Related questions:
Print current call stack from a method in Python code
Check what a running process is doing: print stack trace of an uninstrumented Python program

I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons. Its a bit hacky, and only works on unix (requires signals):
import code, traceback, signal
def debug(sig, frame):
"""Interrupt running process, and provide a python prompt for
interactive debugging."""
d={'_frame':frame} # Allow access to frame object.
d.update(frame.f_globals) # Unless shadowed by global
d.update(frame.f_locals)
i = code.InteractiveConsole(d)
message = "Signal received : entering python shell.\nTraceback:\n"
message += ''.join(traceback.format_stack(frame))
i.interact(message)
def listen():
signal.signal(signal.SIGUSR1, debug) # Register handler
To use, just call the listen() function at some point when your program starts up (You could even stick it in site.py to have all python programs use it), and let it run. At any point, send the process a SIGUSR1 signal, using kill, or in python:
os.kill(pid, signal.SIGUSR1)
This will cause the program to break to a python console at the point it is currently at, showing you the stack trace, and letting you manipulate the variables. Use control-d (EOF) to continue running (though note that you will probably interrupt any I/O etc at the point you signal, so it isn't fully non-intrusive.
I've another script that does the same thing, except it communicates with the running process through a pipe (to allow for debugging backgrounded processes etc). Its a bit large to post here, but I've added it as a python cookbook recipe.

The suggestion to install a signal handler is a good one, and I use it a lot. For example, bzr by default installs a SIGQUIT handler that invokes pdb.set_trace() to immediately drop you into a pdb prompt. (See the bzrlib.breakin module's source for the exact details.) With pdb you can not only get the current stack trace (with the (w)here command) but also inspect variables, etc.
However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in ~/.gdbinit, then:
Attach gdb: gdb -p PID
Get the python stack trace: pystack
It's not totally reliable unfortunately, but it works most of the time. See also https://wiki.python.org/moin/DebuggingWithGdb
Finally, attaching strace can often give you a good idea what a process is doing.

I am almost always dealing with multiple threads and main thread is generally not doing much, so what is most interesting is to dump all the stacks (which is more like the Java's dump). Here is an implementation based on this blog:
import threading, sys, traceback
def dumpstacks(signal, frame):
id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
code = []
for threadId, stack in sys._current_frames().items():
code.append("\n# Thread: %s(%d)" % (id2name.get(threadId,""), threadId))
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
if line:
code.append(" %s" % (line.strip()))
print("\n".join(code))
import signal
signal.signal(signal.SIGQUIT, dumpstacks)

Getting a stack trace of an unprepared python program, running in a stock python without debugging symbols can be done with pyrasite. Worked like a charm for me in on Ubuntu Trusty:
$ sudo pip install pyrasite
$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
$ sudo pyrasite 16262 dump_stacks.py # dumps stacks to stdout/stderr of the python program
(Hat tip to #Albert, whose answer contained a pointer to this, among other tools.)

>>> import traceback
>>> def x():
>>> print traceback.extract_stack()
>>> x()
[('<stdin>', 1, '<module>', None), ('<stdin>', 2, 'x', None)]
You can also nicely format the stack trace, see the docs.
Edit: To simulate Java's behavior, as suggested by #Douglas Leeder, add this:
import signal
import traceback
signal.signal(signal.SIGUSR1, lambda sig, stack: traceback.print_stack(stack))
to the startup code in your application. Then you can print the stack by sending SIGUSR1 to the running Python process.

The traceback module has some nice functions, among them: print_stack:
import traceback
traceback.print_stack()

You can try the faulthandler module. Install it using pip install faulthandler and add:
import faulthandler, signal
faulthandler.register(signal.SIGUSR1)
at the beginning of your program. Then send SIGUSR1 to your process (ex: kill -USR1 42) to display the Python traceback of all threads to the standard output. Read the documentation for more options (ex: log into a file) and other ways to display the traceback.
The module is now part of Python 3.3. For Python 2, see http://faulthandler.readthedocs.org/

What really helped me here is spiv's tip (which I would vote up and comment on if I had the reputation points) for getting a stack trace out of an unprepared Python process. Except it didn't work until I modified the gdbinit script. So:
download https://svn.python.org/projects/python/trunk/Misc/gdbinit and put it in ~/.gdbinit
edit it, changing PyEval_EvalFrame to PyEval_EvalFrameEx [edit: no longer needed; the linked file already has this change as of 2010-01-14]
Attach gdb: gdb -p PID
Get the python stack trace: pystack

python -dv yourscript.py
That will make the interpreter to run in debug mode and to give you a trace of what the interpreter is doing.
If you want to interactively debug the code you should run it like this:
python -m pdb yourscript.py
That tells the python interpreter to run your script with the module "pdb" which is the python debugger, if you run it like that the interpreter will be executed in interactive mode, much like GDB

It can be done with excellent py-spy. It's a sampling profiler for Python programs, so its job is to attach to a Python processes and sample their call stacks. Hence, py-spy dump --pid $SOME_PID is all you need to do to dump call stacks of all threads in the $SOME_PID process. Typically it needs escalated privileges (to read the target process' memory).
Here's an example of how it looks like for a threaded Python application.
$ sudo py-spy dump --pid 31080
Process 31080: python3.7 -m chronologer -e production serve -u www-data -m
Python v3.7.1 (/usr/local/bin/python3.7)
Thread 0x7FEF5E410400 (active): "MainThread"
_wait (cherrypy/process/wspbus.py:370)
wait (cherrypy/process/wspbus.py:384)
block (cherrypy/process/wspbus.py:321)
start (cherrypy/daemon.py:72)
serve (chronologer/cli.py:27)
main (chronologer/cli.py:84)
<module> (chronologer/__main__.py:5)
_run_code (runpy.py:85)
_run_module_as_main (runpy.py:193)
Thread 0x7FEF55636700 (active): "_TimeoutMonitor"
run (cherrypy/process/plugins.py:518)
_bootstrap_inner (threading.py:917)
_bootstrap (threading.py:885)
Thread 0x7FEF54B35700 (active): "HTTPServer Thread-2"
accept (socket.py:212)
tick (cherrypy/wsgiserver/__init__.py:2075)
start (cherrypy/wsgiserver/__init__.py:2021)
_start_http_thread (cherrypy/process/servers.py:217)
run (threading.py:865)
_bootstrap_inner (threading.py:917)
_bootstrap (threading.py:885)
...
Thread 0x7FEF2BFFF700 (idle): "CP Server Thread-10"
wait (threading.py:296)
get (queue.py:170)
run (cherrypy/wsgiserver/__init__.py:1586)
_bootstrap_inner (threading.py:917)
_bootstrap (threading.py:885)

I would add this as a comment to haridsv's response, but I lack the reputation to do so:
Some of us are still stuck on a version of Python older than 2.6 (required for Thread.ident), so I got the code working in Python 2.5 (though without the thread name being displayed) as such:
import traceback
import sys
def dumpstacks(signal, frame):
code = []
for threadId, stack in sys._current_frames().items():
code.append("\n# Thread: %d" % (threadId))
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
if line:
code.append(" %s" % (line.strip()))
print "\n".join(code)
import signal
signal.signal(signal.SIGQUIT, dumpstacks)

Take a look at the faulthandler module, new in Python 3.3. A faulthandler backport for use in Python 2 is available on PyPI.

On Solaris, you can use pstack(1) No changes to the python code are necessary. eg.
# pstack 16000 | grep : | head
16000: /usr/bin/python2.6 /usr/lib/pkg.depotd --cfg svc:/application/pkg/serv
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:282 (_wait) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:295 (wait) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:242 (block) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/_init_.py:249 (quickstart) ]
[ /usr/lib/pkg.depotd:890 (<module>) ]
[ /usr/lib/python2.6/threading.py:256 (wait) ]
[ /usr/lib/python2.6/Queue.py:177 (get) ]
[ /usr/lib/python2.6/vendor-packages/pkg/server/depot.py:2142 (run) ]
[ /usr/lib/python2.6/threading.py:477 (run)
etc.

If you're on a Linux system, use the awesomeness of gdb with Python debug extensions (can be in python-dbg or python-debuginfo package). It also helps with multithreaded applications, GUI applications and C modules.
Run your program with:
$ gdb -ex r --args python <programname>.py [arguments]
This instructs gdb to prepare python <programname>.py <arguments> and run it.
Now when you program hangs, switch into gdb console, press Ctr+C and execute:
(gdb) thread apply all py-list
See example session and more info here and here.

I was looking for a while for a solution to debug my threads and I found it here thanks to haridsv. I use slightly simplified version employing the traceback.print_stack():
import sys, traceback, signal
import threading
import os
def dumpstacks(signal, frame):
id2name = dict((th.ident, th.name) for th in threading.enumerate())
for threadId, stack in sys._current_frames().items():
print(id2name[threadId])
traceback.print_stack(f=stack)
signal.signal(signal.SIGQUIT, dumpstacks)
os.killpg(os.getpgid(0), signal.SIGQUIT)
For my needs I also filter threads by name.

I hacked together some tool which attaches into a running Python process and injects some code to get a Python shell.
See here: https://github.com/albertz/pydbattach

You can use the hypno package, like so:
hypno <pid> "import traceback; traceback.print_stack()"
This would print a stack trace into the program's stdout.
Alternatively, if you don't want to print anything to stdout, or you don't have access to it (a daemon for example), you could use the madbg package, which is a python debugger that allows you to attach to a running python program and debug it in your current terminal. It is similar to pyrasite and pyringe, but newer, doesn't require gdb, and uses IPython for the debugger (which means colors and autocomplete).
To see the stack trace of a running program, you could run:
madbg attach <pid>
And in the debugger shell, enter:
bt
Disclaimer - I wrote both packages

It's worth looking at Pydb, "an expanded version of the Python debugger loosely based on the gdb command set". It includes signal managers which can take care of starting the debugger when a specified signal is sent.
A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb.

Since Austin 3.3, you can use the -w/--where option to emit the current stack trace. See https://stackoverflow.com/a/70905181/1838793
If you want to peek at a running Python application to see the "live" call stack in a top-like fashon you can use austin-tui (https://github.com/p403n1x87/austin-tui). You can install it from PyPI with e.g.
pipx install austin-tui
Note that it requires the austin binary to work (https://github.com/p403n1x87/austin), but then you can attach to a running Python process with
austin-tui -p <pid>

pyringe is a debugger that can interact with running python processes, print stack traces, variables, etc. without any a priori setup.
While I've often used the signal handler solution in the past, it can still often be difficult to reproduce the issue in certain environments.

You can use PuDB, a Python debugger with a curses interface to do this. Just add
from pudb import set_interrupt_handler; set_interrupt_handler()
to your code and use Ctrl-C when you want to break. You can continue with c and break again multiple times if you miss it and want to try again.

I am in the GDB camp with the python extensions. Follow https://wiki.python.org/moin/DebuggingWithGdb, which means
dnf install gdb python-debuginfo or sudo apt-get install gdb python2.7-dbg
gdb python <pid of running process>
py-bt
Also consider info threads and thread apply all py-bt.

How to debug any function in console:
Create function where you use pdb.set_trace(), then function you want debug.
>>> import pdb
>>> import my_function
>>> def f():
... pdb.set_trace()
... my_function()
...
Then call created function:
>>> f()
> <stdin>(3)f()
(Pdb) s
--Call--
> <stdin>(1)my_function()
(Pdb)
Happy debugging :)

I don't know of anything similar to java's response to SIGQUIT, so you might have to build it in to your application. Maybe you could make a server in another thread that can get a stacktrace on response to a message of some kind?

There is no way to hook into a running python process and get reasonable results. What I do if processes lock up is hooking strace in and trying to figure out what exactly is happening.
Unfortunately often strace is the observer that "fixes" race conditions so that the output is useless there too.

use the inspect module.
import inspect
help(inspect.stack)
Help on function stack in module inspect:
stack(context=1)
Return a list of records for the stack above the caller's frame.
I find it very helpful indeed.

In Python 3, pdb will automatically install a signal handler the first time you use c(ont(inue)) in the debugger. Pressing Control-C afterwards will drop you right back in there. In Python 2, here's a one-liner which should work even in relatively old versions (tested in 2.7 but I checked Python source back to 2.4 and it looked okay):
import pdb, signal
signal.signal(signal.SIGINT, lambda sig, frame: pdb.Pdb().set_trace(frame))
pdb is worth learning if you spend any amount of time debugging Python. The interface is a bit obtuse but should be familiar to anyone who has used similar tools, such as gdb.

In case you need to do this with uWSGI, it has Python Tracebacker built-in and it's just matter of enabling it in the configuration (number is attached to the name for each worker):
py-tracebacker=/var/run/uwsgi/pytrace
Once you have done this, you can print backtrace simply by connecting to the socket:
uwsgi --connect-and-read /var/run/uwsgi/pytrace1

At the point where the code is run, you can insert this small snippet to see a nicely formatted printed stack trace. It assumes that you have a folder called logs at your project's root directory.
# DEBUG: START DEBUG -->
import traceback
with open('logs/stack-trace.log', 'w') as file:
traceback.print_stack(file=file)
# DEBUG: END DEBUG --!