Develop Reference

Python Redirect command output to log file [duplicate]

This question already has answers here:
Redirect stdout to a file in Python?
(14 answers)
Closed 2 years ago.
I have written a Python code where I call below Scappy API:
sendp(packet, iface=adapter_name)
The code works fine but issue is that it prints below line on console where the program is executing as many times the above command is called:
Sent 1 packets.
.
I need to suppress the console output so I tried:
old_stdout = sys.stdout
sys.stdout = log_file_name
But then I get exception:
sendp(packet, iface=adapter_name)
File "C:\Python36\lib\site-packages\scapy\sendrecv.py", line 315, in sendp
verbose=verbose, realtime=realtime, return_packets=return_packets)
File "C:\Python36\lib\site-packages\scapy\sendrecv.py", line 289, in __gen_send
print("\nSent %i packets." % n)
AttributeError: 'str' object has no attribute 'write'
How can achieve the objective?

You have this previously answered question if you want to redirect to file:
Redirect stdout to a file in Python?
The following example for redirecting to a memory string (from python docs):
import io
from contextlib import redirect_stdout
f = io.StringIO()
with redirect_stdout(f):
help(pow)
s = f.getvalue()
And the latter example (if you just want to suppress the sys.stdout), trying to
understand how it works: print will try to access the write method of sys.stdout, so I will just replace it with a class that has a write method that does nothing.
>>> class R:
... def write(*args, **kwargs):
... pass
>>> from contextlib import redirect_stdout
>>> with redirect_stdout(R()):
... print('ciao')

Not allowing the developer to use the print method

I have developed a python framework that is being used by others. In order to print any data to the output, the developer should use a Log class (Log.print(...)) and should not use the print() method directly. Is there any ways to force this rule throughout the code? For example, by throwing an error when a developer uses the print method directly like this:
Error: print method cannot be called directly. Please use Log.print().
Suppressing print (as discussed here) is not a good idea as the developer might get confused.

Actullay, below two line code are the same:
sys.stdout.write('hello'+'\n')
print('hello')
so, you can redirect sys.stdout to a class which raise a exception at calling print.
import sys
class BlockPrint():
call_print_exception = Exception('Error: print method cannot be called directly. Please use Log.print().')
def write(self, str):
raise self.call_print_exception
bp = BlockPrint()
sys.stdout=bp
print('aaa')
Output:
Traceback (most recent call last):
File "p.py", line 12, in <module>
print('aaa')
File "p.py", line 7, in write
raise self.call_print_exception
Exception: Error: print method cannot be called directly. Please use Log.print().

How to capture the stdout/stderr of a unittest in a variable?

How to capture the stdout/stderr of a unittest in a variable? I need to capture the entire output output of the following unit test and send it to SQS. I have tried this:
import unittest, io
from contextlib import redirect_stdout, redirect_stderr
class LogProcessorTests(unittest.TestCase):
def setUp(self):
self.var = 'this value'
def test_var_value(self):
with io.StringIO() as buf, redirect_stderr(buf):
print('Running LogProcessor tests...')
print('Inside test_var_value')
self.assertEqual(self.var, 'that value')
print('-----------------------')
print(buf.getvalue())
However, it doesn't work and the following output appears only on stdout/stderr.
Testing started at 20:32 ...
/Users/myuser/Documents/virtualenvs/app-venv3/bin/python3 "/Applications/PyCharm CE.app/Contents/helpers/pycharm/_jb_unittest_runner.py" --path /Users/myuser/Documents/projects/application/LogProcessor/tests/test_processor_tests.py
Launching unittests with arguments python -m unittest /Users/myuser/Documents/projects/application/LogProcessor/tests/test_processor_tests.py in /Users/myuser/Documents/projects/application/LogProcessor/tests
Running LogProcessor tests...
Inside test_var_value
that value != this value
Expected :this value
Actual :that value
<Click to see difference>
Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/helpers/pycharm/teamcity/diff_tools.py", line 32, in _patched_equals
old(self, first, second, msg)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 839, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 1220, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 680, in fail
raise self.failureException(msg)
AssertionError: 'this value' != 'that value'
- this value
? ^^
+ that value
? ^^
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 615, in run
testMethod()
File "/Users/myuser/Documents/projects/application/LogProcessor/tests/test_processor_tests.py", line 15, in test_var_value
self.assertEqual(self.var, 'that value')
Ran 1 test in 0.004s
FAILED (failures=1)
Process finished with exit code 1
Assertion failed
Assertion failed
Any idea? Please let me know if more info is needed.

Based on the contextlib.redirect_stdout documentation, this is how you'd redirect stderr or stdout:
import io
import contextlib
f = io.StringIO()
with contextlib.redirect_stderr(f):
parser = target.parse_args([])
self.assertTrue("error: one of the arguments -p/--propagate -cu/--cleanup is required" in f.getvalue())
You can also combine that with another context manager (like assertRaises) like this:
f = io.StringIO()
with self.assertRaises(SystemExit) as cm, contextlib.redirect_stderr(f):
parser = target.parse_args([])
self.assertEqual(cm.exception.code, 2)
self.assertTrue("error: one of the arguments -p/--propagate -cu/--cleanup is required" in f.getvalue())

If you manually instantiate the test runner (e.g. unittest.TextTestRunner), you can specify the (file) stream it writes to. By default this is sys.stderr, but you can use a StringIO instead. That will capture the output of the unittest itself. The output of your own print-statements will not be captured, but you can use the redirect_stdout context manager for that, using the same StringIO object.
Note that I would recommend to avoid using print-statements, since they will interfere with the output of the unittest framework (your test output will break the output lines of the unittest framework) and it's a bit of a hack to redirect the stdout/stderr streams. A better solution would be to use the logging module instead. You could then add a logging handler that writes all log messages into a StringIO for further processing (in your case: sending to SQS).
Below is example code based on your code using print-statements.
#!/usr/bin/env python3
import contextlib
import io
import unittest
class LogProcessorTests(unittest.TestCase):
def setUp(self):
self.var = 'this value'
def test_var_value(self):
print('Running LogProcessor tests...')
print('Inside test_var_value')
self.assertEqual(self.var, 'that value')
print('-----------------------')
if __name__ == '__main__':
# find all tests in this module
import __main__
suite = unittest.TestLoader().loadTestsFromModule(__main__)
with io.StringIO() as buf:
# run the tests
with contextlib.redirect_stdout(buf):
unittest.TextTestRunner(stream=buf).run(suite)
# process (in this case: print) the results
print('*** CAPTURED TEXT***:\n%s' % buf.getvalue())
This prints:
*** CAPTURED TEXT***:
Running LogProcessor tests...
Inside test_var_value
F
======================================================================
FAIL: test_var_value (__main__.LogProcessorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 16, in test_var_value
self.assertEqual(self.var, 'that value')
AssertionError: 'this value' != 'that value'
- this value
? ^^
+ that value
? ^^
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
This confirms all output (from the unittest framework and the testcase itself) were captured in the StringIO object.

Honestly the easiest way is probably to redirect your output at the OS level--run the test from the command line and > it to a file.
If you are using a build system to execute these, then the build system should be capturing the output for you and you can extract the output from it's build artifacts.

Log stack trace for python warning

A package that I'm using in my python program is throwing a warning that I'd like to understand the exact cause of. I've set logging.captureWarning(True) and am capturing the warning in my logging, but still have no idea where it is coming from. How do I also log the stack trace so I can see where in my code the warning is coming from? Do I use traceback?

I've ended up going with the below:
import warnings
import traceback
_formatwarning = warnings.formatwarning
def formatwarning_tb(*args, **kwargs):
s = _formatwarning(*args, **kwargs)
tb = traceback.format_stack()
s += ''.join(tb[:-1])
return s
warnings.formatwarning = formatwarning_tb
logging.captureWarnings(True)

It's a little hackish, but you can monkeypatch the warnings.warn method to this:
import traceback
import warnings
def g():
warnings.warn("foo", Warning)
def f():
g()
warnings.warn("bar", Warning)
_old_warn = warnings.warn
def warn(*args, **kwargs):
tb = traceback.extract_stack()
_old_warn(*args, **kwargs)
print("".join(traceback.format_list(tb)[:-1]))
warnings.warn = warn
f()
print("DONE")
This is the output:
/tmp/test.py:14: Warning: foo
_old_warn(*args, **kwargs)
File "/tmp/test.py", line 17, in <module>
f()
File "/tmp/test.py", line 8, in f
g()
File "/tmp/test.py", line 5, in g
warnings.warn("foo", Warning)
/tmp/test.py:14: Warning: bar
_old_warn(*args, **kwargs)
File "/tmp/test.py", line 17, in <module>
f()
File "/tmp/test.py", line 9, in f
warnings.warn("bar", Warning)
DONE
See that calling the original warnings.warn function does not report the line you'd want, bu the stack trace is indeed correct (you could print the warning message yourself).

If you do not know what data/instruction is causing the warning throw, you can use tools like the standard Python Debugger.
The documentation is really good and detailed, but some quickly examples that may help should be:
Without modifying source code: invoking the debbugger as script:
$ python -m pdb myscript.py
Modifying source code: you can make use of calls to pdb.set_trace(), that work like breakpoints; For example, consider I have the following example code:
x = 2
x = x * 10 * 100
y = x + 3 + y
return y
And I would like to know what value does x and y have before the return, or what does the stack contains, I would add the following line between those statements:
pdb.set_trace()
And I will be promted to the (Pdb) prompt, that will allow you to go through the code line by line. Useful commands for the (Pdb) prompt are:
n: executes the next statement.
q: quits the whole program.
c: quits the (Pdb) prompt and stops debugging.
p varname: prints the value of varname
As you do not provide more information, I do not know if that should be enough, but I think that at least, it may be a good start.
BONUS EDIT
Based on this answer, I have found there is a nice and friendly GUI debugging tool, that you can simply install by:
$ pip install pudb
And run the debugger with your script with:
$ python -m pudb.run myscript.py
EDIT: Adding the postmortem debugging
If we do not even know if the code is going to crash or not, we can enter in postmortem debugging if there has been a crash. From the Pbd documentation:
The typical usage to inspect a crashed program is:
>>> import pdb
>>> import mymodule
>>> mymodule.test()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "./mymodule.py", line 4, in test
test2()
File "./mymodule.py", line 3, in test2
print spam
NameError: spam
>>> pdb.pm()
> ./mymodule.py(3)test2()
-> print spam
(Pdb)
As postmortem looks at sys.last_traceback, to enter only if there is a traceback (and so on, a warning or crash):
if sys.last_traceback:
pdb.pm()

You can turn warnings into exceptions, which means you will get a stack trace automatically:
warnings.filterwarnings("error")
See https://docs.python.org/3.4/library/warnings.html#the-warnings-filter

If it was me, I'd go with #Lluís Vilanova's quick & dirty hack, just to find something. But if that's not an option...
If you really want a "logging" solution, you could try something like this (fully working source).
Basic steps are:
Create a custom logging.Formatter subclass that includes the current stack where the logging record is formatted
Use that formatter on the class of the warning
The meat of the code is the custom formatter:
class Formatter(logging.Formatter):
def format(self, record):
record.stack_info = ''.join(traceback.format_stack())
return super().format(record)
Per the docs:
New in version 3.2: The stack_info parameter was added.

For python 3.2 and above, using the optional stack_info keyword argument is the easiest way to get stack trace info along with the log message.
In the example below, "Server.py" is using "lib2.py", which is in turn using "lib.py".
On enabling the stack_info argument the complete trace back is logged along with every logging.log() call. This works the same with logging.info() and other convenience methods as well.
Usage :-
logging.log(DEBUG, "RWL [{}] : acquire_read()".format(self._ownerName), stack_info=True)
Output :-
2018-10-06 10:59:55,726|DEBUG|MainThread|lib.py|acquire_read|RWL [Cache] : acquire_read()
Stack (most recent call last):
File "./Server.py", line 41, in <module>
logging.info("Found {} requests for simulation".format(simdata.count()))
File "<Path>\lib2.py", line 199, in count
with basics.ReadRWLock(self.cacheLock):
File "<Path>\lib.py", line 89, in __enter__
self.rwLock.acquire_read()
File "<Path>\lib.py", line 34, in acquire_read
logging.log(DEBUG, "RWL [{}] : acquire_read()".format(self._ownerName), stack_info=True)

Print current call stack from a method in code

In Python, how can I print the current call stack from within a method (for debugging purposes).

Here's an example of getting the stack via the traceback module, and printing it:
import traceback
def f():
g()
def g():
for line in traceback.format_stack():
print(line.strip())
f()
# Prints:
# File "so-stack.py", line 10, in <module>
# f()
# File "so-stack.py", line 4, in f
# g()
# File "so-stack.py", line 7, in g
# for line in traceback.format_stack():
If you really only want to print the stack to stderr, you can use:
traceback.print_stack()
Or to print to stdout (useful if want to keep redirected output together), use:
traceback.print_stack(file=sys.stdout)
But getting it via traceback.format_stack() lets you do whatever you like with it.

import traceback
traceback.print_stack()

for those who need to print the call stack while using pdb, just do
(Pdb) where

inspect.stack() returns the current stack rather than the exception traceback:
import inspect
print inspect.stack()
See https://gist.github.com/FredLoney/5454553 for a log_stack utility function.

If you use python debugger, not only interactive probing of variables but you can get the call stack with the "where" command or "w".
So at the top of your program
import pdb
Then in the code where you want to see what is happening
pdb.set_trace()
and you get dropped into a prompt

Here's a variation of #RichieHindle's excellent answer which implements a decorator that can be selectively applied to functions as desired. Works with Python 2.7.14 and 3.6.4.
from __future__ import print_function
import functools
import traceback
import sys
INDENT = 4*' '
def stacktrace(func):
#functools.wraps(func)
def wrapped(*args, **kwds):
# Get all but last line returned by traceback.format_stack()
# which is the line below.
callstack = '\n'.join([INDENT+line.strip() for line in traceback.format_stack()][:-1])
print('{}() called:'.format(func.__name__))
print(callstack)
return func(*args, **kwds)
return wrapped
#stacktrace
def test_func():
return 42
print(test_func())
Output from sample:
test_func() called:
File "stacktrace_decorator.py", line 28, in <module>
print(test_func())
42

Install Inspect-it
pip3 install inspect-it --user
Code
import inspect;print(*['{:40}| {}:{}\n'.format(x.function, x.filename, x.lineno) for x in inspect.stack()])
you can Make a snippet of this line
it will show you a list of the function call stack with a filename and line number
list from start to where you put this line