I have successfully setup plpyton3u extension in Postgresql 10 (64 bit) on my windows 10 (64 bit) machine. However, when i try to make a http request by calling requests module I am getting attribute error AttributeError: 'module' object has no attribute 'get' . Here is the code that I am using
CREATE OR REPLACE FUNCTION from_url(
-- The URL to download.
IN url text,
-- Should any errors (like HTTP transport errors)
-- throw an exception or simply return default_response
IN should_throw boolean DEFAULT true,
-- The default response if any errors are found.
-- Only used when should_throw is set to true
IN default_response text DEFAULT E''
)
RETURNS text
AS $$
# We will use traceback so we get decent error reporting.
import requests
import traceback
# Either throws an error or returns the defeault response
# depending on the should_throw parameter of the from_url() function
def on_error():
if should_throw:
# plpy.error() throws an exception which stops the current transaction
plpy.error("Error downloading '{0}'\n {1}".format(url, traceback.format_exc()))
else:
return default_response
try:
response = requests.get(url)
return response.data
except:
# Log and re-throw the error or return the default response
return on_error()
$$ LANGUAGE plpython3u VOLATILE;
select from_url(<SOME_DATA_FETCHING_URL>);
Where SOME_DATA_FETCHING_URL is the url of the server serving the data. When I run this code it throws following error
ERROR: plpy.Error: Error downloading SOME_DATA_FETCHING_URL
Traceback (most recent call last):
File "", line 16, in __plpython_procedure_from_url_24640
AttributeError: 'module' object has no attribute 'get'
CONTEXT: Traceback (most recent call last):
PL/Python function "from_url", line 19, in
return on_error()
PL/Python function "from_url", line 11, in on_error
plpy.error("Error downloading '{0}'\n {1}".format(url, traceback.format_exc()))
PL/Python function "from_url"
SQL state: XX000
my PYTHONPATH is set to C:\Python34\;C:\Python34\Scripts;C:\Python34\Lib;
I checked on python command prompt, I am able to import requests and run the get command successfully.
Am I missing any settings in PostGRESQL? which will allow me to run this code successfully?
(In case anyone else stumbles on this question) I had a similar problem, in my case it turned out to be caused by inadequate access permissions for user 'postgres' to the directory containing my Python modules. One thing I have learnt is that to check a PL/Python problem using the Python command prompt, it's always necessary to run it as the same user as the Postgres server (in my case, 'postgres')
Related
I am trying to run ElasticSearch python tests from official elasticsearch-py realease here : https://github.com/elastic/elasticsearch-py/tree/5.x
Actually, I have done all the steps mentionned into elasticsearch-py/test_elasticsearch/README.rst and it turns out that I can run Elasticsearch in a Container with success.
But so far running the tests is failing everytime.
I have got 12 errors out of 124 runned tests and it looks like test_elasticsearch.test_connection.TestRequestsConnection is always failing due to :
AttributeError: 'Session' object has no attribute
'merge_environment_settings'.
What I have done so far is downloading the official Repo, export ES_VERSION=5.4 and then running ElasticSearch with ./start_elasticsearch.sh. After that I am able to navigate to localhost:9200 into my Web Browser.
Then I've changed TEST_ES_SERVER into 'localhost:9200' into elasticsearch5/helpers/test.py file.
I am running the tests using python setup.py test or python run_tests.py but here's what the console returns in both case :
No elasticsearch repo found...
........................./home/user/bin/git/elasticsearch-py/elasticsearch5/client/utils.py:47:
UnicodeWarning: Unicode equal comparison failed to convert both
arguments to Unicode - interpreting them as being unequal quote_plus(_escape(p), b',*') for p in parts if p not in SKIP_IN_PATH)
.EE.EEE.E..EE.EE.EE................................................................................
ERROR: test_body_attached (test_elasticsearch.test_connection.TestRequestsConnection)
Traceback (most recent call last) :
File
"/home/user/bin/git/elasticsearch-py/test_elasticsearch/test_connection.py",
line 213, in test_body_attached
request = self._get_request(con, 'GET', '/', body='{"answer": 42}')
File "/home/user/bin/git/elasticsearch-py/test_elasticsearch/test_connection.py", line 75, in _get_request status, headers, data =
connection.perform_request(*args, **kwargs)
File
"/home/user/bin/git/elasticsearch-py/elasticsearch5/connection/http_requests.py",
line 72, in perform_request
settings =
self.session.merge_environment_settings(prepared_request.url, {},
None, None, None)
AttributeError: 'Session' object has no attribute
merge_environment_settings'
Do you have any idea how I can make this work ? Locally at first.
Thanks and regards.
I'm running a flask app, upgraded everything from Python 2.7 to 3 about 5 months ago.
Most things have gone smooth enough, other than this one that's consistently bugging me locally. I'm on a MacBook on OSX 10.12.6, and a brew install of Python 3.6.3 under a virtualenv.
When a request comes in from a page that seemingly has multiple static requests (.css, .js, and image files mainly), I seem to be able to get this error just about anywhere that's using generators anywhere in my code.
Some examples (request is a flask.request object):
A place that checks to see if a path starts with '/static' of '/admin/static' (my code),
any(request.path.startswith(k) for k in self._static_paths)
.
Exception ignored in: <generator object CustomPrincipal._is_static_route.<locals>.<genexpr> at 0x11450f3b8>
Traceback (most recent call last):
File "/Developer/repos/git/betapilibs/lbbsports/flask_monkeypatches.py", line 22, in <genexpr>
any(_checker(request.path, k) for k in self._static_paths)
SystemError: error return without exception set
If a url path is restricted, check if the logged in user has the proper permissions / role,
return role in (role.name for role in self.roles)
.
Exception ignored in: <generator object UserMixin.has_role.<locals>.<genexpr> at 0x1155a7e08>
Traceback (most recent call last):
File "/Developer/virtualenvs/lbb3/lib/python3.6/site-packages/flask_security/core.py", line 386, in <genexpr>
SystemError: error return without exception set
A custom bit of code to ensure their "sub" account id is valid,
(not any(ident == account_id for ident in account_ids))
.
Exception ignored in: <generator object CustomSession.get_set_accounts.<locals>.<genexpr> at 0x115ff4fc0>
Traceback (most recent call last):
File "/Developer/customflask/flasklogin.py", line 168, in <genexpr>
SystemError: error return without exception set
Now, nothing seems to break in the system, I just get these error messages, and not consistently, only sometimes. If I set a breakpoint anywhere these errors are being reported to be happening, they don't error any more.
If I do something like, in the first example, break it into request.path.startswith('/static') or request.path.startswith('/admin/static'), I no longer get the error message, and in general, I never have a problem using request all over the place in the rest of the app.
A thing that was wrong in my local development setup was that I was serving all the /static and /admin/static through the flask app, instead of serving them through the web-server (in my case, nginx). So for some of the urls I was hitting, there might have been 10 requests come in basically at the same time, with Flask in debug mode, and a debugger connected as well (via PyCharm).
When I went through the trouble to ensure that all '/static' and '/admin/static' get served from there, instead of via flask, and flask was only getting 1 request per url, this problem went away.
I won't mark this as the answer, because there is still an underlying issue, but in case others have the same problem as me, this was a solution for my situation.
I have a Flask project on GAE and I'd like to start adding try/except blocks around database writes in case the datastore has problems, which will definitely fire when there's a real error, but I'd like to mimic that error in a unittest so I can have confidence of what will really happen during an outage.
For example, my User model:
class User(ndb.Model):
guser = ndb.UserProperty()
user_handle = ndb.StringProperty()
and in other view/controller code:
def do_something():
try:
User(guser=users.get_current_user(), user_handle='barney').put()
except CapabilityDisabledError:
flash('Oops, database is down, try again later', 'danger')
return redirect(url_for('registration_done'))
Here's a gist of my test code: https://gist.github.com/iandouglas/10441406
In a nutshell, GAE allows us to use capabilities to temporarily disable the stubs for memcache, datastore_v3, etc., and in the main test method:
def test_stuff(self):
# this test ALWAYS passes, making me believe the datastore is temporarily down
self.assertFalse(capabilities.CapabilitySet('datastore_v3').is_enabled())
# but this write to the datastore always SUCCEEDS, so the exception never gets
# thrown, therefore this "assertRaises" always fails
self.assertRaises(CapabilityDisabledError,
lambda: User(guser=self.guser, pilot_handle='foo').put())
I read some other post recommending calling the User.put() as a lambda which results in this traceback:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 28, in test_stuff
self.assertRaises(CapabilityDisabledError, lambda: User(
AssertionError: CapabilityDisabledError not raised
If I remove the lambda: portion, I get this traceback instead:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 31, in test_stuff
pilot_handle_lower='foo'
File "/usr/lib/python2.7/unittest/case.py", line 475, in assertRaises
callableObj(*args, **kwargs)
TypeError: 'Key' object is not callable
Google's tutorials show you how to turn these capabilities on and off for unit testing, and in other tutorials they show you which exceptions could get thrown if their services are offline or experiencing intermittent issues, but they have no tutorials showing how they might work together in a unit test.
Thanks for any ideas.
The datastore stub does not support returning a CapabilityDisabledError, so enabled the error in the capabilities stub will not affect calls to datastore.
As a separate note, if you are using the High Replication Datastore, you'll never experience the CapabilityDisabledError because it does not have scheduled downtime.
Thanks for your help in advance.
I've got the following class method that I'm trying to test:
def _get_ldap_connection(self):
"""
Instantiate and return simpleldap.Connection object.
Raises:
ldap.SERVER_DOWN: When ldap_url is invalid or server is
not reachable.
"""
try:
ldap_connection = simpleldap.Connection(
self.ldap_url, encryption='ssl', require_cert=False,
debug=False, dn=self.ldap_login_dn,
password=self.ldap_login_password)
except ldap.SERVER_DOWN:
raise ldap.SERVER_DOWN(
"The LDAP server specified, {}, did not respond to the "
"connection attempt.".format(self.ldap_url))
And here's the unittest:
def test__get_ldap_connection(self):
"""
VERY IMPORTANT: This test refers to your actual config.json file.
If it is correctly populated, you can expect this test to fail.
"""
# Instantiate Class
test_extractor = SakaiLdapExtractor('config_files/config.json')
# Monkey with ldap server url to ensure error.
test_extractor.ldap_url = "invalid_ldap_url"
self.assertRaises(
ldap.SERVER_DOWN, test_extractor._get_ldap_connection())
So far, so good. But when I execute the unit tests (via nose) test_extractor._get_ldap_connection() is called from the assertRaises statement, but the exception is not caught and the test fails.
Here is the output:
vagrant#precise64:/vagrant/sakai-directory-integration$ nosetests
...E..
======================================================================
ERROR: VERY IMPORTANT: This test refers to your actual config.json file.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/vagrant/sakai-directory-integration/test_sakaiLdapExtractor.py", line 77, in test__get_ldap_connection
ldap.SERVER_DOWN, test_extractor._get_ldap_connection())
File "/vagrant/sakai-directory-integration/sakai_ldap_integration.py", line 197, in _get_ldap_connection
"connection attempt.".format(self.ldap_url))
SERVER_DOWN: The LDAP server specified, invalid_ldap_url, did not respond to the connection attempt.
----------------------------------------------------------------------
Ran 6 tests in 0.159s
Help me!
Don't call, just pass function (method) itself; drop ():
self.assertRaises(
ldap.SERVER_DOWN, test_extractor._get_ldap_connection)
Alternatively, you can use with self.assertRaises(..) form if you are using recent version of python (Python 2.7+ / Python 3.1+):
with self.assertRaises(ldap.SERVER_DOWN):
test_extractor._get_ldap_connection()
You are not using assertRaises correctly.
You can use it as a context manager:
with self.assertRaises(ldap.SERVER_DOWN):
test_extractor._get_ldap_connection()
or the usual way (self.assertRaises(exception, function, args):
self.assertRaises(ldap.SERVER_DOWN, test_extractor._get_ldap_connection)
Also see:
How to properly use unit-testing's assertRaises() with NoneType objects?
Testing in Python - how to use assertRaises in testing using unittest?
documentation
I'm running into a few problems with adding gae-sessions to a relatively mature GAE app. I followed the readme carefully and also looked at the demo.
First, just adding the gaesesions directory to my app causes the following error when running tests with nose and nose-gae:
Exception ImportError: 'No module named threading' in <bound method local.__del__ of <_threading_local.local object at 0x103e10628>> ignored
All the tests run fine so not a big problem but suggests that something isn't right.
Next, if I add the following two lines of code:
from gaesessions import get_current_session
session = get_current_session()
And run my tests, then I get the following error:
Traceback (most recent call last):
File "/Users/.../unit_tests.py", line 1421, in testParseFBRequest
data = tasks.parse_fb_request(sr)
File "/Users/.../tasks.py", line 220, in parse_fb_request
session = get_current_session()
File "/Users/.../gaesessions/__init__.py", line 36, in get_current_session
return _tls.current_session
File "/Library/.../python2.7/_threading_local.py", line 193, in __getattribute__
return object.__getattribute__(self, name)
AttributeError: 'local' object has no attribute 'current_session'
This error does not happen on the dev server.
Any suggestions on fixing the above would be greatly appreciated.
I ran into the same problem. The problem seems to be that the gae testbed behaves differently than the development server. I don't know the specifics but ended up solving it by adding
def setUp(self):
testbed.Testbed().activate()
# after activating the testbed:
from gaesessions import Session, set_current_session
set_current_session(Session())