Develop Reference

Pythonic way of not printing a error message up

I am trying to suppress a error/warning in my log while calling a library. Assume i have this code
try:
kazoo_client.start()
except:
pass
This is calling a zookeeper client which throws some exception which bubble up, now i don't want the warn/error in my logs when i call kazoo_client.start() is there a way to get this suppressed when you call the client

Assuming python 2.7.17
Try this approach:
import sys, StringIO
def funky() :
"1" + 1 # This should raise an error
sys.stderr = StringIO.StringIO()
funky() # this should call the funky function
And your code should look something like this:
import sys, StringIO
# import kazoo somehere around here
sys.stderr = StringIO.StringIO()
kazoo_client.start()
And lastly the Python 3 example:
import sys
from io import StringIO
# import kazoo somehere around here
sys.stderr = StringIO()
kazoo_client.start()

If you know the exception, try contextlib.suppress:
>>> from contextlib import suppress
>>> x = (i for i in range(10))
>>> with suppress(StopIteration):
... for i in range(11):
... print(next(x))
0
1
2
3
4
5
6
7
8
9
Without suppress it throws StopIteration error at last iteration.
>>> x = (i for i in range(10))
>>> for i in range(11):
... print(next(x))
0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):
File "<ipython-input-10-562798e05ad5>", line 2, in <module>
print(next(x))
StopIteration
Suppress is Pythonic, safe and explicit.
So in your case:
with suppress(SomeError):
kazoo_client.start()
EDIT:
To suppress all exceptions:
with suppress(Exception):
kazoo_client.start()

I would like to suggest a more generic approach, which can be used in general.
I leave you an example of how to create an decorator who ignore errors.
import functools
# Use the Decorator Design Pattern
def ignore_error_decorator(function_reference):
#functools.wraps(function_reference) # the decorator inherits the original function signature
def wrapper(*args):
try:
result = function_reference(*args) # execute the function
return result # If the function executed correctly, return
except Exception as e:
pass # Default ignore; You can also log the error or do what ever you want
return wrapper # Return the wrapper reference
if __name__ == '__main__':
# First alternative to use. Compose the decorator with another function
def my_first_function(a, b):
return a + b
rez_valid = ignore_error_decorator(my_first_function)(1, 3)
rez_invalid = ignore_error_decorator(my_first_function)(1, 'a')
print("Alternative_1 valid: {result}".format(result=rez_valid))
print("Alternative_1 invalid: {result}".format(result=rez_invalid)) # None is return by the exception bloc
# Second alternative. Decorating a function
#ignore_error_decorator
def my_second_function(a, b):
return a + b
rez_valid = my_second_function(1, 5)
rez_invalid = my_second_function(1, 'a')
print("Alternative_2 valid: {result}".format(result=rez_valid))
print("Alternative_2 invalid: {result}".format(result=rez_invalid)) # None is return by the exception bloc
Getting back to your problem, using my alternative you have to run
ignore_error_decorator(kazoo_client.start)()

Adding an OverflowError side effect with patch - Python

I'd like to mock an OverflowError because I'd like to test the value of a variable is after that exception is raised. However, I do not know how to replicate an OverflowError with the libraries I am using. Libraries I am using in this specific test is pysolar.solar specifically the get_altitude, get_azimuth and radiation methods
After mindlessly trying out different numbers to try to simulate an OverflowError I decided to try out Mocking the function and introducing a side effect.
Code I am testing sunposition.py
import numpy as np
import pandas as pd
from pysolar.solar import get_altitude, get_azimuth, radiation as radiation_module
def sun_position(lat: float, lon: float, time: pd.Timestamp = None) -> List[float]:
if time is None:
time = pd.Timestamp.now(tz='UTC')
dt = time.to_pydatetime()
altitude = get_altitude(lat, lon, dt)
azimuth = get_azimuth(lat, lon, dt)
try:
radiation = radiation_module.get_radiation_direct(dt, altitude)
except OverflowError:
radiation = np.nan
return pd.Series([altitude, azimuth, radiation], index=['Alt', 'Azi', 'Rad'])
**What I started to do with patch **
"""Test sunposition module"""
import unittest
import numpy as np
import pandas as pd
from unittest.mock import MagicMock, patch, Mock
from bigfolder.sun import sunposition
class TestSunposition(unittest.TestCase):
"""Test functions in sunposition."""
def test_sun_position_overflow_error(self):
error_lat = 23
error_lon = 12
error_time = pd.Timestamp('2007-02-18 15:13:05', tz="UTC")
mock_args = {'side_effect': OverflowError}
with patch('bigfolder.sun.sunposition.sun_position', **mock_args):
# run the test
self.assertRaises(OverflowError, sunposition.sun_position(lat=error_lat, lon=error_lon, time=error_time))
if __name__ == '__main__':
unittest.main()
I was expecting it to give me and OverFlow error... and it did, however my assertion failed anyways with an OverflowError My guess is that I patched at the wrong place? I don't really understand why the test failed regardless when the error was still an OverFlow Error
This is what was printed out
_mock_self = <MagicMock name='sun_position' id='102333856'>, args = ()
kwargs = {'lat': 23, 'lon': 12, 'time': Timestamp('2007-02-18 15:13:05+0000', tz='UTC')}
self = <MagicMock name='sun_position' id='102333856'>, _new_name = ''
_new_parent = None
_call = call(lat=23, lon=12, time=Timestamp('2007-02-18 15:13:05+0000', tz='UTC'))
seen = set(), skip_next_dot = False, do_method_calls = False
name = 'sun_position'
def _mock_call(_mock_self, *args, **kwargs):
self = _mock_self
self.called = True
self.call_count += 1
_new_name = self._mock_new_name
_new_parent = self._mock_new_parent
_call = _Call((args, kwargs), two=True)
self.call_args = _call
self.call_args_list.append(_call)
self.mock_calls.append(_Call(('', args, kwargs)))
seen = set()
skip_next_dot = _new_name == '()'
do_method_calls = self._mock_parent is not None
name = self._mock_name
while _new_parent is not None:
this_mock_call = _Call((_new_name, args, kwargs))
if _new_parent._mock_new_name:
dot = '.'
if skip_next_dot:
dot = ''
skip_next_dot = False
if _new_parent._mock_new_name == '()':
skip_next_dot = True
_new_name = _new_parent._mock_new_name + dot + _new_name
if do_method_calls:
if _new_name == name:
this_method_call = this_mock_call
else:
this_method_call = _Call((name, args, kwargs))
_new_parent.method_calls.append(this_method_call)
do_method_calls = _new_parent._mock_parent is not None
if do_method_calls:
name = _new_parent._mock_name + '.' + name
_new_parent.mock_calls.append(this_mock_call)
_new_parent = _new_parent._mock_new_parent
# use ids here so as not to call __hash__ on the mocks
_new_parent_id = id(_new_parent)
if _new_parent_id in seen:
break
seen.add(_new_parent_id)
ret_val = DEFAULT
effect = self.side_effect
if effect is not None:
if _is_exception(effect):
> raise effect
E OverflowError
So then I thought I must have patched in the wrong place and introduced the side effect earlier than I should have? So I instead patched the method int the try block. This was my following code.
def test_sun_position_overflow_error(self):
error_lat = 23
error_lon = 12
error_time = pd.Timestamp('2007-02-18 15:13:05', tz="UTC")
mock_args = {'side_effect': OverflowError}
with patch('bigfolder.sun.sunposition.sun_position.radiation_module.get_radiation_direct', **mock_args):
# run the test
self.assertRaises(OverflowError, sunposition.sun_position(lat=error_lat, lon=error_lon, time=error_time))
And now my error is " ModuleNotFoundError: No module named 'bigfolder.sun.sunposition.sun_position'; 'bigfolder.sun.sunposition' is not a package"
I then just changed the path to 'sun_position.radiation_module.get_radiation_direct' but there's no module found.
So my questions are: How to replicate an OverflowError so that then I can check the value of a variable I set once that Exception is raised. Why is the first OverflowError I introduce still not pass my assertion?
Thanks
UPDATE TEST passes
After following #Gang 's suggestion, the OverFlowError was reproduced. I realized that in order to test the block for the exception, specifically that radiation was then np.nan I had to patch the method that I wanted to have the OverFlowError and not the whole method of sun_position. When I tried doing that I imported it incorrectly, as I thought the external library was part of the code. So I changed bigfolder.sun.sunposition.sun_position.radiation_module.get_radiation_direct to pysolar.solar.radiation.get_radiation_direct which is the external library that has the get_radiation_direct method I wanted to mock.
def test_sun_position_overflow_error(self):
lat = 23
lon = 12
time = pd.Timestamp('2007-02-18 15:13:05', tz="UTC")
# get_radiation_direct will now produce an OverFlowError(regardless of coordinates)
mock_args = {'side_effect': OverflowError}
# mock get_radiation_direct and produce OverFlowError
with patch('pysolar.solar.radiation.get_radiation_direct', **mock_args):
# Check radiation column is nan value
assert math.isnan(sunposition.sun_position(lat=lat, lon=lon, time=time)[2])

Why is the first OverflowError I introduce still not pass my assertion?
Almost there. The right way to assertRaises:
def test_sun_position_overflow_error(self):
# This has to be here first and without func call
with self.assertRaises(OverflowError):
# patch the function to have an exception no matter what
mock_args = {'side_effect': OverflowError}
with patch('bigfolder.sun.sunposition.sun_position', **mock_args):
# call this func to trigger an exception
sunposition.sun_position(lat=error_lat, lon=error_lon, time=error_time)
After review the document, it is how to call a func inside assertRaises
assertRaises(exception, callable, *args, **kwds)
fun(*args, **kwds) raises exc
This usage is wrong:
self.assertRaises(OverflowError, sunposition.sun_position(lat=error_lat, lon=error_lon, time=error_time))
It should be a func name with kwargs:
self.assertRaises(OverflowError, sunposition.sun_position, lat=error_lat, lon=error_lon, time=error_time)

Python ctypes-wrapped function taking int fails to reject argument larger than INT_MAX

Tested using Python 2.7.13 on 64-bit Linux.
I have a C function inside a shared object. This function takes two ints and returns an int. When I try to pass an integer from the Python side that is larger than INT_MAX but still representable as an unsigned int, I don't get a Python-side exception like I would expect.
// add.c
int add(int a, int b)
{
return a + b;
}
This file is compiled into a shared objects using the following options. (snippet from makefile)
gcc -shared -fPIC add.c -o add.so
I'm using the following python script test_add.c to test the code. I'm probably doing more work than necessary, manually supplying argument types of ctypes.c_int and manually coercing the integers generated by hypothesis, but it doesn't seem like any of those operations should be causing the behavior I'm seeing.
Here's the main test that I'm trying with different values of x and y. One example of a failing test case is x=0 and y=2147483648, a number one larger than INT_MAX.
try:
s = add(ctypes.c_int(x), ctypes.c_int(y))
except Exception:
return
# if no exception was thrown, python and C should agree
# about the result of the addition
assert s == (x + y)
And here's the script in full for the sake of completeness:
import ctypes
import os.path
import unittest
from hypothesis import given
import hypothesis.strategies as st
# import shared object from this directory
addso_path = os.path.dirname(os.path.abspath(__file__)) + os.path.sep + "add.so"
addso = ctypes.cdll.LoadLibrary(addso_path)
# extract add symbol, explicitly force its argument types to be
# (int, int)
# and return type to be
# int
add = addso.add
add.argtypes = [ctypes.c_int, ctypes.c_int]
add.restype = ctypes.c_int
class TestAddition(unittest.TestCase):
#given(x=st.integers(), y=st.integers())
def test_add(self, x, y):
# if we catch any error on the python side,
# then that counts as successfully preventing an
# out-of-bounds integer before it is passed to C
try:
s = add(ctypes.c_int(x), ctypes.c_int(y))
except Exception:
return
# if no exception was thrown, python and C should
# agree about the result of the addition
assert s == (x + y)
if __name__ == "__main__":
unittest.main()
Here's the output of the test suite:
Falsifying example: test_add(self=<__main__.TestAddition testMethod=test_add>, x=0, y=2147483648)
F
======================================================================
FAIL: test_add (__main__.TestAddition)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_add.py", line 22, in test_add
def test_add(self, x, y):
File "/.../opt/c-add/venv/lib/python2.7/site-packages/hypothesis/core.py", line 525, in wrapped_test
print_example=True, is_final=True
File "/...opt/c-add/venv/lib/python2.7/site-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/.../opt/c-add/venv/lib/python2.7/site-packages/hypothesis/core.py", line 112, in run
return test(*args, **kwargs)
File "test_add.py", line 33, in test_add
assert s == (x + y)
AssertionError

Try function not running correctly

try:
left_break = signs_pos[dave - 1]
except IndexError:
left_error = True
try:
right_break = signs_pos[dave + 1]
except IndexError:
right_error = True
if left_error == True:
current_cal = user_input[:right_break]
elif right_error == True:
current_cal = user_input[left_break:]
else:
current_cal = user_input[left_break:right_break]
I've only started looking at try functions and I need some help with this. What I would like to happen is that if when it tries to find left_break and it gives an error it will set left_error to be true. But if it does not give an error left_break will be set properly.
When the code runs and either right or left does not give an error it does not set left_break or right_break properly.
Traceback (most recent call last):
File "C:\Users\Max\Desktop\MaxsCal.py", line 170, in <module>
current_cal = user_input[:right_break]
NameError: name 'right_break' is not defined
This is the error I get without the try function.
Traceback (most recent call last):
File "C:\Users\Max\Desktop\MaxsCal.py", line 157, in <module>
right_break = signs_pos[dave + 1]
IndexError: list index out of range
Both right_error and left_error will not be true.

The reason why this is happening, is that because you are trying to assign something to a variable inside a try/except, it will not actually exist if you raised an exception.
Here is a simple example to clarify this:
try:
x = 6 / 0
except ZeroDivisionError:
print('this failed')
print(x)
>>> print(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
Now, to remedy this and you are looking to actually use the variable even if it fails in the try/except, you want to declare it before you are calling the thing that might fail.
x = 0
try:
x = 6 / 0
except ZeroDivisionError:
print('this failed')
print(x)
>>> print(x)
>>> 0
Or even inside your try works, but before what you are calling:
try:
x = 0
x = 6 / 0
except ZeroDivisionError:
print('this failed')
print(x)
>>> print(x)
>>> 0
As mentioned to me in the comments, you could also set a default in your except:
try:
x = 6 / 0
except ZeroDivisionError:
print('this failed')
x = 0
print(x)
>>> print(x)
>>> 0

left_break and right_break are only available within the scope of the try block. You can either define them before the try/except block or add an else block to the exception, as discussed in this thread.

Your error is coming from the fact that the variable is out of scope. Right break is only in scope within the try block, change it to
right_break = None
try:
right_break = signs_pos[dave + 1]
except IndexError:
right_error = True

locals() and globals() in stack trace on exception (Python)

While stack traces are useful in Python, most often the data at the root of the problem are missing - is there a way of making sure that at least locals() (and possibly globals()) are added to printed stacktrace?

You can install your own exception hook and output what you need from there:
import sys, traceback
def excepthook(type, value, tb):
traceback.print_exception(type, value, tb)
while tb.tb_next:
tb = tb.tb_next
print >>sys.stderr, 'Locals:', tb.tb_frame.f_locals
print >>sys.stderr, 'Globals:', tb.tb_frame.f_globals
sys.excepthook = excepthook
def x():
y()
def y():
foo = 1
bar = 0
foo/bar
x()
To print vars from each frame in a traceback, change the above loop to
while tb:
print >>sys.stderr, 'Locals:', tb.tb_frame.f_locals
print >>sys.stderr, 'Globals:', tb.tb_frame.f_globals
tb = tb.tb_next

This is a Box of Pandora. Values can be very large in printed form; printing all locals in a stack trace can easily lead to new problems just due to error output. That's why this is not implemented in general in Python.
In small examples, though, i. e. if you know that your values aren't too large to be printed properly, you can step along the traceback yourself:
import sys
import traceback
def c():
clocal = 1001
raise Exception("foo")
def b():
blocal = 23
c()
def a():
alocal = 42
b()
try:
a()
except Exception:
frame = sys.exc_info()[2]
formattedTb = traceback.format_tb(frame)
frame = frame.tb_next
while frame:
print formattedTb.pop(0), '\t', frame.tb_frame.f_locals
frame = frame.tb_next
The output will be sth like this:
File "/home/alfe/tmp/stacktracelocals.py", line 19, in <module>
a()
{'alocal': 42}
File "/home/alfe/tmp/stacktracelocals.py", line 16, in a
b()
{'blocal': 23}
File "/home/alfe/tmp/stacktracelocals.py", line 12, in b
c()
{'clocal': 1001}
And you can, of course, install your own except hook as thg435 suggested in his answer.

if you didn't know about this already, use the pdb post-mortem feature:
x = 3.0
y = 0.0
print x/y
def div(a, b):
return a / b
print div(x,y)
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
<ipython-input-3-d03977de5fc3> in div(a, b)
1 def div(a, b):
----> 2 return a / b
ZeroDivisionError: float division
import pdb
pdb.pm()
> <ipython-input-3-148da0dcdc9e>(2)div()
0 return a/b
ipdb> l
1 def div(a,b):
----> 2 return a/b
ipdb> a
3.0
ipdb> b
0.0
etc.
there are cases where you really need the prints though, of course. you're better off instrumenting the code (via try/except) to print out extra information around a specific weird exception you are debugging than putting this for everything though, imho.

Try traceback-with-variables package.
Usage:
from traceback_with_variables import traceback_with_variables
def main():
...
with traceback_with_variables():
...your code...
Exceptions with it:
Traceback with variables (most recent call last):
File "./temp.py", line 7, in main
return get_avg_ratio([h1, w1], [h2, w2])
sizes_str = '300 200 300 0'
h1 = 300
w1 = 200
h2 = 300
w2 = 0
File "./temp.py", line 10, in get_avg_ratio
return mean([get_ratio(h, w) for h, w in [size1, size2]])
size1 = [300, 200]
size2 = [300, 0]
File "./temp.py", line 10, in <listcomp>
return mean([get_ratio(h, w) for h, w in [size1, size2]])
.0 = <tuple_iterator object at 0x7ff61e35b820>
h = 300
w = 0
File "./temp.py", line 13, in get_ratio
return height / width
height = 300
width = 0
builtins.ZeroDivisionError: division by zero
Installation:
pip install traceback-with-variables