Develop Reference

Python 3.6.3: Multiple "Exception ignored in: <generator object..." in flask app

I'm running a flask app, upgraded everything from Python 2.7 to 3 about 5 months ago.
Most things have gone smooth enough, other than this one that's consistently bugging me locally. I'm on a MacBook on OSX 10.12.6, and a brew install of Python 3.6.3 under a virtualenv.
When a request comes in from a page that seemingly has multiple static requests (.css, .js, and image files mainly), I seem to be able to get this error just about anywhere that's using generators anywhere in my code.
Some examples (request is a flask.request object):
A place that checks to see if a path starts with '/static' of '/admin/static' (my code),
any(request.path.startswith(k) for k in self._static_paths)
.
Exception ignored in: <generator object CustomPrincipal._is_static_route.<locals>.<genexpr> at 0x11450f3b8>
Traceback (most recent call last):
File "/Developer/repos/git/betapilibs/lbbsports/flask_monkeypatches.py", line 22, in <genexpr>
any(_checker(request.path, k) for k in self._static_paths)
SystemError: error return without exception set
If a url path is restricted, check if the logged in user has the proper permissions / role,
return role in (role.name for role in self.roles)
.
Exception ignored in: <generator object UserMixin.has_role.<locals>.<genexpr> at 0x1155a7e08>
Traceback (most recent call last):
File "/Developer/virtualenvs/lbb3/lib/python3.6/site-packages/flask_security/core.py", line 386, in <genexpr>
SystemError: error return without exception set
A custom bit of code to ensure their "sub" account id is valid,
(not any(ident == account_id for ident in account_ids))
.
Exception ignored in: <generator object CustomSession.get_set_accounts.<locals>.<genexpr> at 0x115ff4fc0>
Traceback (most recent call last):
File "/Developer/customflask/flasklogin.py", line 168, in <genexpr>
SystemError: error return without exception set
Now, nothing seems to break in the system, I just get these error messages, and not consistently, only sometimes. If I set a breakpoint anywhere these errors are being reported to be happening, they don't error any more.
If I do something like, in the first example, break it into request.path.startswith('/static') or request.path.startswith('/admin/static'), I no longer get the error message, and in general, I never have a problem using request all over the place in the rest of the app.

A thing that was wrong in my local development setup was that I was serving all the /static and /admin/static through the flask app, instead of serving them through the web-server (in my case, nginx). So for some of the urls I was hitting, there might have been 10 requests come in basically at the same time, with Flask in debug mode, and a debugger connected as well (via PyCharm).
When I went through the trouble to ensure that all '/static' and '/admin/static' get served from there, instead of via flask, and flask was only getting 1 request per url, this problem went away.
I won't mark this as the answer, because there is still an underlying issue, but in case others have the same problem as me, this was a solution for my situation.

Python: assertRaises( ) not catching ldap.SERVER_DOWN error when raised

Thanks for your help in advance.
I've got the following class method that I'm trying to test:
def _get_ldap_connection(self):
"""
Instantiate and return simpleldap.Connection object.
Raises:
ldap.SERVER_DOWN: When ldap_url is invalid or server is
not reachable.
"""
try:
ldap_connection = simpleldap.Connection(
self.ldap_url, encryption='ssl', require_cert=False,
debug=False, dn=self.ldap_login_dn,
password=self.ldap_login_password)
except ldap.SERVER_DOWN:
raise ldap.SERVER_DOWN(
"The LDAP server specified, {}, did not respond to the "
"connection attempt.".format(self.ldap_url))
And here's the unittest:
def test__get_ldap_connection(self):
"""
VERY IMPORTANT: This test refers to your actual config.json file.
If it is correctly populated, you can expect this test to fail.
"""
# Instantiate Class
test_extractor = SakaiLdapExtractor('config_files/config.json')
# Monkey with ldap server url to ensure error.
test_extractor.ldap_url = "invalid_ldap_url"
self.assertRaises(
ldap.SERVER_DOWN, test_extractor._get_ldap_connection())
So far, so good. But when I execute the unit tests (via nose) test_extractor._get_ldap_connection() is called from the assertRaises statement, but the exception is not caught and the test fails.
Here is the output:
vagrant#precise64:/vagrant/sakai-directory-integration$ nosetests
...E..
======================================================================
ERROR: VERY IMPORTANT: This test refers to your actual config.json file.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/vagrant/sakai-directory-integration/test_sakaiLdapExtractor.py", line 77, in test__get_ldap_connection
ldap.SERVER_DOWN, test_extractor._get_ldap_connection())
File "/vagrant/sakai-directory-integration/sakai_ldap_integration.py", line 197, in _get_ldap_connection
"connection attempt.".format(self.ldap_url))
SERVER_DOWN: The LDAP server specified, invalid_ldap_url, did not respond to the connection attempt.
----------------------------------------------------------------------
Ran 6 tests in 0.159s
Help me!

Don't call, just pass function (method) itself; drop ():
self.assertRaises(
ldap.SERVER_DOWN, test_extractor._get_ldap_connection)
Alternatively, you can use with self.assertRaises(..) form if you are using recent version of python (Python 2.7+ / Python 3.1+):
with self.assertRaises(ldap.SERVER_DOWN):
test_extractor._get_ldap_connection()

You are not using assertRaises correctly.
You can use it as a context manager:
with self.assertRaises(ldap.SERVER_DOWN):
test_extractor._get_ldap_connection()
or the usual way (self.assertRaises(exception, function, args):
self.assertRaises(ldap.SERVER_DOWN, test_extractor._get_ldap_connection)
Also see:
How to properly use unit-testing's assertRaises() with NoneType objects?
Testing in Python - how to use assertRaises in testing using unittest?
documentation

Database errors in Django when using threading

I am working in a Django web application which needs to query a PostgreSQL database. When implementing concurrency using Python threading interface, I am getting DoesNotExist errors for the queried items. Of course, these errors do not occur when performing the queries sequentially.
Let me show a unit test which I wrote to demonstrate the unexpected behavior:
class ThreadingTest(TestCase):
fixtures = ['demo_city',]
def test_sequential_requests(self):
"""
A very simple request to database, made sequentially.
A fixture for the cities has been loaded above. It is supposed to be
six cities in the testing database now. We will made a request for
each one of the cities sequentially.
"""
for number in range(1, 7):
c = City.objects.get(pk=number)
self.assertEqual(c.pk, number)
def test_threaded_requests(self):
"""
Now, to test the threaded behavior, we will spawn a thread for
retrieving each city from the database.
"""
threads = []
cities = []
def do_requests(number):
cities.append(City.objects.get(pk=number))
[threads.append(threading.Thread(target=do_requests, args=(n,))) for n in range(1, 7)]
[t.start() for t in threads]
[t.join() for t in threads]
self.assertNotEqual(cities, [])
As you can see, the first test performs some database requests sequentially, which are indeed working with no problem. The second test, however, performs exactly the same requests but each request is spawned in a thread. This is actually failing, returning a DoesNotExist exception.
The output of the execution of this unit tests is like this:
test_sequential_requests (cesta.core.tests.threadbase.ThreadingTest) ... ok
test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest) ...
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests
cities.append(City.objects.get(pk=number))
File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get
% self.model._meta.object_name)
DoesNotExist: City matching query does not exist.
... other threads returns a similar output ...
Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests
cities.append(City.objects.get(pk=number))
File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get
% self.model._meta.object_name)
DoesNotExist: City matching query does not exist.
FAIL
======================================================================
FAIL: test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 52, in test_threaded_requests
self.assertNotEqual(cities, [])
AssertionError: [] == []
----------------------------------------------------------------------
Ran 2 tests in 0.278s
FAILED (failures=1)
Destroying test database for alias 'default' ('test_cesta')...
Remember that all this is happening in a PostgreSQL database, which is supposed to be thread safe, not with the SQLite or similars. Test was ran using PostgreSQL also.
At this point, I am totally lost about what can be failing. Any idea or suggestion?
Thanks!
EDIT: I wrote a little view just to check up if it works out of the tests. Here is the code of the view:
def get_cities(request):
queue = Queue.Queue()
def get_async_cities(q, n):
city = City.objects.get(pk=n)
q.put(city)
threads = [threading.Thread(target=get_async_cities, args=(queue, number)) for number in range(1, 5)]
[t.start() for t in threads]
[t.join() for t in threads]
cities = list()
while not queue.empty():
cities.append(queue.get())
return render_to_response('async/cities.html', {'cities': cities},
context_instance=RequestContext(request))
(Please, do not take into account the folly of writing the application logic inside the view code. Remember that this is only a proof of concept and would not be never in the real app.)
The result is that code is working nice, the requests are made successfully in threads and the view finally shows the cities after calling its URL.
So, I think making queries using threads will only be a problem when you need to test the code. In production, it will work without any problem.
Any useful suggestions to test this kind of code successfully?

Try using TransactionTestCase:
class ThreadingTest(TransactionTestCase):
TestCase keeps data in memory and doesn't issue a COMMIT to database. Probably the threads are trying to connect directly to DB, while the data is not commited there yet. Seedescription here:
https://docs.djangoproject.com/en/dev/topics/testing/?from=olddocs#django.test.TransactionTestCase
TransactionTestCase and TestCase are identical except for the manner
in which the database is reset to a known state and the ability for
test code to test the effects of commit and rollback. A
TransactionTestCase resets the database before the test runs by
truncating all tables and reloading initial data. A
TransactionTestCase may call commit and rollback and observe the
effects of these calls on the database.

Becomes more clear from this part of the documentation
class LiveServerTestCase(TransactionTestCase):
"""
...
Note that it inherits from TransactionTestCase instead of TestCase because
the threads do not share the same transactions (unless if using in-memory
sqlite) and each thread needs to commit all their transactions so that the
other thread can see the changes.
"""
Now, the transaction has not been committed inside a TestCase, hence the changes are not visible to the other thread.

This sounds like it's an issue with transactions. If you're creating elements within the current request (or test), they're almost certainly in an uncommitted transaction that isn't accessible from the separate connection in the other thread. You probably need to manage your transctions manually to get this to work.

Is it possible to serialize tasklet code (not just exec state) using SPickle without doing a RPC?

Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.
Been referencing following presentation:
https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle
Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery:
Client code:
from stackless import run, schedule, tasklet
from sPickle import SPickleTools
def test_method():
print "hello from test method"
tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)
pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)
Server code:
pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)
Results in:
[2012-03-09 14:24:59,104: ERROR/MainProcess] Task
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'
Any suggestions on what I am doing incorrect or if this is even possible?
Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). I have experimented with doing:
py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)
but the dynamically loaded module does not seem to persist through different calls to celeryd, i.e. different remote calls.

You must define test_method within its own module. Currently sPickle detects whether test_method is defined in a module that can be imported. An alternative way is to set the __module__ attribute of the function to None.
def test_method():
pass
test_method.__module__ = None

PermanentTaskFailure: 'module' object has no attribute 'Migrate'

I'm using Nick Johnson's Bulk Update library on google appengine (http://blog.notdot.net/2010/03/Announcing-a-robust-datastore-bulk-update-utility-for-App-Engine). It works wonderfully for other tasks, but for some reason with the following code:
from google.appengine.ext import db
from myapp.main.models import Story, Comment
import bulkupdate
class Migrate(bulkupdate.BulkUpdater):
DELETE_COMPLETED_JOBS_DELAY = 0
DELETE_FAILED_JOBS = False
PUT_BATCH_SIZE = 1
DELETE_BATCH_SIZE = 1
MAX_EXECUTION_TIME = 10
def get_query(self):
return Story.all().filter("hidden", False).filter("visible", True)
def handle_entity(self, entity):
comments = entity.comment_set
for comment in comments:
s = Story()
s.parent_story = comment.story
s.user = comment.user
s.text = comment.text
s.submitted = comment.submitted
self.put(s)
job = Migrate()
job.start()
I get the following error in my logs:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 258, in post
run(self.request.body)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 122, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: 'module' object has no attribute 'Migrate'
It seems quite bizarre to me. Clearly that class is right above the job, they're in the same file and clearly the job.start is being called. Why can't it see my Migrate class?
EDIT: I added this update job in a newer version of the code, which isn't the default. I invoke the job with the correct URL (http://version.myapp.appspot.com/migrate). Is it possible this is related to the fact that it isn't the 'default' version served by App Engine?

It seems likely that your declaration of the 'Migrate' class is in the handler script (Eg, the one directly invoked by app.yaml). A limitation of deferred is that you can't use it to call functions defined in the handler module.
Incidentally, my bulk update library is deprecated in favor of App Engine's mapreduce support; you should probably use that instead.