Develop Reference

Pl\Python: issues when importing module

I have successfully setup plpyton3u extension in Postgresql 10 (64 bit) on my windows 10 (64 bit) machine. However, when i try to make a http request by calling requests module I am getting attribute error AttributeError: 'module' object has no attribute 'get' . Here is the code that I am using
CREATE OR REPLACE FUNCTION from_url(
-- The URL to download.
IN url text,
-- Should any errors (like HTTP transport errors)
-- throw an exception or simply return default_response
IN should_throw boolean DEFAULT true,
-- The default response if any errors are found.
-- Only used when should_throw is set to true
IN default_response text DEFAULT E''
)
RETURNS text
AS $$
# We will use traceback so we get decent error reporting.
import requests
import traceback
# Either throws an error or returns the defeault response
# depending on the should_throw parameter of the from_url() function
def on_error():
if should_throw:
# plpy.error() throws an exception which stops the current transaction
plpy.error("Error downloading '{0}'\n {1}".format(url, traceback.format_exc()))
else:
return default_response
try:
response = requests.get(url)
return response.data
except:
# Log and re-throw the error or return the default response
return on_error()
$$ LANGUAGE plpython3u VOLATILE;
select from_url(<SOME_DATA_FETCHING_URL>);
Where SOME_DATA_FETCHING_URL is the url of the server serving the data. When I run this code it throws following error
ERROR: plpy.Error: Error downloading SOME_DATA_FETCHING_URL
Traceback (most recent call last):
File "", line 16, in __plpython_procedure_from_url_24640
AttributeError: 'module' object has no attribute 'get'
CONTEXT: Traceback (most recent call last):
PL/Python function "from_url", line 19, in
return on_error()
PL/Python function "from_url", line 11, in on_error
plpy.error("Error downloading '{0}'\n {1}".format(url, traceback.format_exc()))
PL/Python function "from_url"
SQL state: XX000
my PYTHONPATH is set to C:\Python34\;C:\Python34\Scripts;C:\Python34\Lib;
I checked on python command prompt, I am able to import requests and run the get command successfully.
Am I missing any settings in PostGRESQL? which will allow me to run this code successfully?

(In case anyone else stumbles on this question) I had a similar problem, in my case it turned out to be caused by inadequate access permissions for user 'postgres' to the directory containing my Python modules. One thing I have learnt is that to check a PL/Python problem using the Python command prompt, it's always necessary to run it as the same user as the Postgres server (in my case, 'postgres')

newbiee query: mocking in python

I'm trying to learn Python and mocking infrastructure in Python at the same time (Due to requirement at my work place). I should also mention that I'm also not familiar with mocking feature in C++ or any other language.
So far, from what I've understood is that, with mocking, I can exercise the application code that makes OS. networking etc related calls, without actually invoking those operation.
Let's say I've an application, implemented as network.py
#!/usr/bin/env python
import sys
import socket
class NetworkService(object):
def sock_create(self):
try:
s = socket.socket()
s.close()
print "closed socket"
except Exception, err:
print "error creating socket"
sys.exit(1)
Things that I'd like to achieve with my unit test is:
Make sure that both normal and failure paths get tested.
In this case, to achieve, this I'm trying to come up with a sample unit test case that exercises the sock_create method, as below:
#!/usr/bin/env python
import unittest
import mock
from network import NetworkService
class NetworkServiceTest(unittest.TestCase):
#mock.patch('network.socket')
def test_01_sock_create(self, mock_sock):
reference = NetworkService()
mock_sock.return_value = False
# NetworkService::sock_create::s.close() should NOT get called
reference.sock_create()
self.assertFalse(mock_sock.close.called, "Failed to not call close")
mock_sock.socket.return_value = True
# NetworkService::sock_create::s.close() should get called
reference.sock_create()
# how to test this ???
#mock_sock.close.assert_called_with("")
if __name__ == '__main__':
unittest.main()
As you can see above, the last 'assert' statement is currently commented out; I'm not sure, how to check this? The following gives me error:
#!/usr/bin/env python
import unittest
import mock
from network import NetworkService
class NetworkServiceTest(unittest.TestCase):
#mock.patch('network.socket')
def test_01_sock_create(self, mock_sock):
reference = NetworkService()
mock_sock.return_value = False
reference.sock_create()
self.assertFalse(mock_sock.close.called, "Failed to not call close")
mock_sock.socket.return_value = True
reference.sock_create()
self.assertTrue(mock_sock.close.called, "Should have called s.close")
if __name__ == '__main__':
unittest.main()
and the error:
$ python tester.py
F
======================================================================
FAIL: test_01_sock_create (__main__.NetworkServiceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "tester.py", line 17, in test_01_sock_create
self.assertTrue(mock_sock.close.called, "Should have called s.close")
AssertionError: Should have called s.close
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
closed socket
error creating socket
NOTE that I'm using mocking in Python 2.7 (mock need to be installed as a separate module)

In network.py you are printing out a string. If you instead would print out the actual error you would see the reason why it's failing. What you would see in this case is that it's failing because of an AttributeError. AttributeError("'bool' object has no attribute 'close'",)
The reason this is happening is because you're giving the mock object the return value of True or False. Since a bool doesn't have any open or close method it'll throw that error.
A few other tips. You're using Exception which will catch all exceptions. Instead, only catch exceptions that you know will happen. Find out what exceptions socket might throw.
That way you would have discovered this earlier.

Can't catch mocked exception because it doesn't inherit BaseException

I'm working on a project that involves connecting to a remote server, waiting for a response, and then performing actions based on that response. We catch a couple of different exceptions, and behave differently depending on which exception is caught. For example:
def myMethod(address, timeout=20):
try:
response = requests.head(address, timeout=timeout)
except requests.exceptions.Timeout:
# do something special
except requests.exceptions.ConnectionError:
# do something special
except requests.exceptions.HTTPError:
# do something special
else:
if response.status_code != requests.codes.ok:
# do something special
return successfulConnection.SUCCESS
To test this, we've written a test like the following
class TestMyMethod(unittest.TestCase):
def test_good_connection(self):
config = {
'head.return_value': type('MockResponse', (), {'status_code': requests.codes.ok}),
'codes.ok': requests.codes.ok
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.SUCCESS
)
def test_bad_connection(self):
config = {
'head.side_effect': requests.exceptions.ConnectionError,
'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError
}
with mock.patch('path.to.my.package.requests', **config):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
If I run the function directly, everything happens as expected. I even tested by adding raise requests.exceptions.ConnectionError to the try clause of the function. But when I run my unit tests, I get
ERROR: test_bad_connection (test.test_file.TestMyMethod)
----------------------------------------------------------------
Traceback (most recent call last):
File "path/to/sourcefile", line ###, in myMethod
respone = requests.head(address, timeout=timeout)
File "path/to/unittest/mock", line 846, in __call__
return _mock_self.mock_call(*args, **kwargs)
File "path/to/unittest/mock", line 901, in _mock_call
raise effect
my.package.requests.exceptions.ConnectionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "Path/to/my/test", line ##, in test_bad_connection
mypackage.myMethod('some_address',
File "Path/to/package", line ##, in myMethod
except requests.exceptions.ConnectionError:
TypeError: catching classes that do not inherit from BaseException is not allowed
I tried to change the exception I was patching in to BaseException and I got a more or less identical error.
I've read https://stackoverflow.com/a/18163759/3076272 already, so I think it must be a bad __del__ hook somewhere, but I'm not sure where to look for it or what I can even do in the mean time. I'm also relatively new to unittest.mock.patch() so it's very possible that I'm doing something wrong there as well.
This is a Fusion360 add-in so it is using Fusion 360's packaged version of Python 3.3 - as far as I know it's a vanilla version (i.e. they don't roll their own) but I'm not positive of that.

I could reproduce the error with a minimal example:
foo.py:
class MyError(Exception):
pass
class A:
def inner(self):
err = MyError("FOO")
print(type(err))
raise err
def outer(self):
try:
self.inner()
except MyError as err:
print ("catched ", err)
return "OK"
Test without mocking :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
a = foo.A()
self.assertEquals("OK", a.outer())
Ok, all is fine, both test pass
The problem comes with the mocks. As soon as the class MyError is mocked, the expect clause cannot catch anything and I get same error as the example from the question :
class FooTest(unittest.TestCase):
def test_inner(self):
a = foo.A()
self.assertRaises(foo.MyError, a.inner)
def test_outer(self):
with unittest.mock.patch('foo.MyError'):
a = exc2.A()
self.assertEquals("OK", a.outer())
Immediately gives :
ERROR: test_outer (__main__.FooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "...\foo.py", line 11, in outer
self.inner()
File "...\foo.py", line 8, in inner
raise err
TypeError: exceptions must derive from BaseException
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#78>", line 8, in test_outer
File "...\foo.py", line 12, in outer
except MyError as err:
TypeError: catching classes that do not inherit from BaseException is not allowed
Here I get a first TypeErrorthat you did not have, because I am raising a mock while you forced a true exception with 'requests.exceptions.ConnectionError': requests.exceptions.ConnectionError in config. But the problem remains that the except clause tries to catch a mock.
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this but in your example, you only need to patch requests.head. So I think that this should work :
def test_bad_connection(self):
with mock.patch('path.to.my.package.requests.head',
side_effect=requests.exceptions.ConnectionError):
self.assertEqual(
mypackage.myMethod('some_address',
mypackage.successfulConnection.FAILURE
)
That is : only patch the head method with the exception as side effect.

I just ran into the same issue while trying to mock sqlite3 (and found this post while looking for solutions).
What Serge said is correct:
TL/DR: as you mock the full requests package, the except requests.exceptions.ConnectionError clause tries to catch a mock. As the mock is not really a BaseException, it causes the error.
The only solution I can imagine is not to mock the full requests but only the parts that are not exceptions. I must admit I could not find how to say to mock mock everything except this
My solution was to mock the entire module, then set the mock attribute for the exception to be equal to the exception in the real class, effectively "un-mocking" the exception. For example, in my case:
#mock.patch(MyClass.sqlite3)
def test_connect_fail(self, mock_sqlite3):
mock_sqlite3.connect.side_effect = sqlite3.OperationalError()
mock_sqlite3.OperationalError = sqlite3.OperationalError
self.assertRaises(sqlite3.OperationalError, MyClass, self.db_filename)
For requests, you could assign exceptions individually like this:
mock_requests.exceptions.ConnectionError = requests.exceptions.ConnectionError
or do it for all of the requests exceptions like this:
mock_requests.exceptions = requests.exceptions
I don't know if this is the "right" way to do it, but so far it seems to work for me without any issue.

For those of us who need to mock an exception and can't do that by simply patching head, here is an easy solution that replaces the target exception with an empty one:
Say we have a generic unit to test with an exception we have to have mocked:
# app/foo_file.py
def test_me():
try:
foo()
return "No foo error happened"
except CustomError: # <-- Mock me!
return "The foo error was caught"
We want to mock CustomError but because it is an exception we run into trouble if we try to patch it like everything else. Normally, a call to patch replaces the target with a MagicMock but that won't work here. Mocks are nifty, but they do not behave like exceptions do. Rather than patching with a mock, let's give it a stub exception instead. We'll do that in our test file.
# app/test_foo_file.py
from mock import patch
# A do-nothing exception we are going to replace CustomError with
class StubException(Exception):
pass
# Now apply it to our test
#patch('app.foo_file.foo')
#patch('app.foo_file.CustomError', new_callable=lambda: StubException)
def test_foo(stub_exception, mock_foo):
mock_foo.side_effect = stub_exception("Stub") # Raise our stub to be caught by CustomError
assert test_me() == "The error was caught"
# Success!
So what's with the lambda? The new_callable param calls whatever we give it and replaces the target with the return of that call. If we pass our StubException class straight, it will call the class's constructor and patch our target object with an exception instance rather than a class which isn't what we want. By wrapping it with lambda, it returns our class as we intend.
Once our patching is done, the stub_exception object (which is literally our StubException class) can be raised and caught as if it were the CustomError. Neat!

I faced a similar issue while trying to mock the sh package. While sh is very useful, the fact that all methods and exceptions are defined dynamically make it more difficult to mock them. So following the recommendation of the documentation:
import unittest
from unittest.mock import Mock, patch
class MockSh(Mock):
# error codes are defined dynamically in sh
class ErrorReturnCode_32(BaseException):
pass
# could be any sh command
def mount(self, *args):
raise self.ErrorReturnCode_32
class MyTestCase(unittest.TestCase):
mock_sh = MockSh()
#patch('core.mount.sh', new=mock_sh)
def test_mount(self):
...

I just ran into the same problem when mocking struct.
I get the error:
TypeError: catching classes that do not inherit from BaseException is not allowed
When trying to catch a struct.error raised from struct.unpack.
I found that the simplest way to get around this in my tests was to simply set the value of the error attribute in my mock to be Exception. For example
The method I want to test has this basic pattern:
def some_meth(self):
try:
struct.unpack(fmt, data)
except struct.error:
return False
return True
The test has this basic pattern.
#mock.patch('my_module.struct')
def test_some_meth(self, struct_mock):
'''Explain how some_func should work.'''
struct_mock.error = Exception
self.my_object.some_meth()
struct_mock.unpack.assert_called()
struct_mock.unpack.side_effect = struct_mock.error
self.assertFalse(self.my_object.some_meth()
This is similar to the approach taken by #BillB, but it is certainly simpler as I don't need to add imports to my tests and still get the same behavior. To me it would seem this is the logical conclusion to the general thread of reasoning in the answers here.

Use patch.object to partially mock a class.
My use case:
import unittest
from unittest import mock
import requests
def test_my_function(self):
response = mock.MagicMock()
response.raise_for_status.side_effect = requests.HTTPError
with mock.patch.object(requests, 'get', return_value=response):
my_function()

Python – How to mock a single function

From the Mock docs, I wasn't able to understand how to implement the following type of pattern successfully. fetch_url does not exist inside of a class.
My function in the auth.py file:
def fetch_url(url, method=urlfetch.GET, data=''):
"""Send a HTTP request"""
result = urlfetch.fetch(url=url, method=method, payload=data,
headers={'Access-Control-Allow-Origin': '*'})
return result.content
My test:
import unittest
from mock import Mock
class TestUrlFetch(unittest.TestCase):
def test_fetch_url(self):
from console.auth import fetch_url
# Create a mock object based on the fetch_url function
mock = Mock(spec=fetch_url)
# Mock the fetch_url function
content = mock.fetch_url('https://google.com')
# Test that content is not empty
self.assertIsNotNone(content)
If what I'm doing is completely in the wrong direction, please shed some light on the correct solution.
The test is not working, and is producing the following error:
======================================================================
ERROR: test_fetch_url (console.tests.test_auth.TestUrlFetch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/bengrunfeld/Desktop/Work/code/wf-ghconsole/console/tests/test_auth.py", line 34, in test_fetch_url
content = mock.fetch_url('https://google.com')
File "/Users/bengrunfeld/.virtualenvs/env2/lib/python2.7/site-packages/mock.py", line 658, in __getattr__
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'fetch_url'
-------------------- >> begin captured logging << --------------------
root: DEBUG: Using threading.local
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 1 test in 0.277s
FAILED (errors=1)

First of all, as univerio's comment suggests you should call you mock like this:
mock('https://google.com')
Your test should pass after that fix, but probably that mock doesn't do what you really want. I've encountered a few problems with spec and autospec.
Mocks created with Mock(spec=) don't check number of arguments they are called with. I've just looked through the docs and they don't state that, but for some reason I expected it to work. Autospecced mocks do check the arguments.
By default both spec and autospec function mocks return mock objects when you call them. This may be not what you want when you mock a function that does not return anything. In this case you can set the return_value manually:
def foo():
pass
mock_foo = Mock(spec=foo, return_value=None)
mock_foo()

How do I use unittest to catch CapabilityDisabledError exceptions?

I have a Flask project on GAE and I'd like to start adding try/except blocks around database writes in case the datastore has problems, which will definitely fire when there's a real error, but I'd like to mimic that error in a unittest so I can have confidence of what will really happen during an outage.
For example, my User model:
class User(ndb.Model):
guser = ndb.UserProperty()
user_handle = ndb.StringProperty()
and in other view/controller code:
def do_something():
try:
User(guser=users.get_current_user(), user_handle='barney').put()
except CapabilityDisabledError:
flash('Oops, database is down, try again later', 'danger')
return redirect(url_for('registration_done'))
Here's a gist of my test code: https://gist.github.com/iandouglas/10441406
In a nutshell, GAE allows us to use capabilities to temporarily disable the stubs for memcache, datastore_v3, etc., and in the main test method:
def test_stuff(self):
# this test ALWAYS passes, making me believe the datastore is temporarily down
self.assertFalse(capabilities.CapabilitySet('datastore_v3').is_enabled())
# but this write to the datastore always SUCCEEDS, so the exception never gets
# thrown, therefore this "assertRaises" always fails
self.assertRaises(CapabilityDisabledError,
lambda: User(guser=self.guser, pilot_handle='foo').put())
I read some other post recommending calling the User.put() as a lambda which results in this traceback:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 28, in test_stuff
self.assertRaises(CapabilityDisabledError, lambda: User(
AssertionError: CapabilityDisabledError not raised
If I remove the lambda: portion, I get this traceback instead:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 31, in test_stuff
pilot_handle_lower='foo'
File "/usr/lib/python2.7/unittest/case.py", line 475, in assertRaises
callableObj(*args, **kwargs)
TypeError: 'Key' object is not callable
Google's tutorials show you how to turn these capabilities on and off for unit testing, and in other tutorials they show you which exceptions could get thrown if their services are offline or experiencing intermittent issues, but they have no tutorials showing how they might work together in a unit test.
Thanks for any ideas.

The datastore stub does not support returning a CapabilityDisabledError, so enabled the error in the capabilities stub will not affect calls to datastore.
As a separate note, if you are using the High Replication Datastore, you'll never experience the CapabilityDisabledError because it does not have scheduled downtime.