Develop Reference

Python 3.6.3: Multiple "Exception ignored in: <generator object..." in flask app

I'm running a flask app, upgraded everything from Python 2.7 to 3 about 5 months ago.
Most things have gone smooth enough, other than this one that's consistently bugging me locally. I'm on a MacBook on OSX 10.12.6, and a brew install of Python 3.6.3 under a virtualenv.
When a request comes in from a page that seemingly has multiple static requests (.css, .js, and image files mainly), I seem to be able to get this error just about anywhere that's using generators anywhere in my code.
Some examples (request is a flask.request object):
A place that checks to see if a path starts with '/static' of '/admin/static' (my code),
any(request.path.startswith(k) for k in self._static_paths)
.
Exception ignored in: <generator object CustomPrincipal._is_static_route.<locals>.<genexpr> at 0x11450f3b8>
Traceback (most recent call last):
File "/Developer/repos/git/betapilibs/lbbsports/flask_monkeypatches.py", line 22, in <genexpr>
any(_checker(request.path, k) for k in self._static_paths)
SystemError: error return without exception set
If a url path is restricted, check if the logged in user has the proper permissions / role,
return role in (role.name for role in self.roles)
.
Exception ignored in: <generator object UserMixin.has_role.<locals>.<genexpr> at 0x1155a7e08>
Traceback (most recent call last):
File "/Developer/virtualenvs/lbb3/lib/python3.6/site-packages/flask_security/core.py", line 386, in <genexpr>
SystemError: error return without exception set
A custom bit of code to ensure their "sub" account id is valid,
(not any(ident == account_id for ident in account_ids))
.
Exception ignored in: <generator object CustomSession.get_set_accounts.<locals>.<genexpr> at 0x115ff4fc0>
Traceback (most recent call last):
File "/Developer/customflask/flasklogin.py", line 168, in <genexpr>
SystemError: error return without exception set
Now, nothing seems to break in the system, I just get these error messages, and not consistently, only sometimes. If I set a breakpoint anywhere these errors are being reported to be happening, they don't error any more.
If I do something like, in the first example, break it into request.path.startswith('/static') or request.path.startswith('/admin/static'), I no longer get the error message, and in general, I never have a problem using request all over the place in the rest of the app.

A thing that was wrong in my local development setup was that I was serving all the /static and /admin/static through the flask app, instead of serving them through the web-server (in my case, nginx). So for some of the urls I was hitting, there might have been 10 requests come in basically at the same time, with Flask in debug mode, and a debugger connected as well (via PyCharm).
When I went through the trouble to ensure that all '/static' and '/admin/static' get served from there, instead of via flask, and flask was only getting 1 request per url, this problem went away.
I won't mark this as the answer, because there is still an underlying issue, but in case others have the same problem as me, this was a solution for my situation.

Python can't import WMI under special circumstance

I've created a standalone exe Windows service written in Python and built with pyInstaller. When I try to import wmi, an exception is thrown.
What's really baffling is that I can do it without a problem if running the code in a foreground exe, or a foreground python script, or a python script running as a background service via pythonservice.exe!
Why does it fail under this special circumstance of running as a service exe?
import wmi
Produces this error for me:
com_error: (-2147221020, 'Invalid syntax', None, None)
Here's the traceback:
Traceback (most recent call last):
File "<string>", line 43, in onRequest
File "C:\XXX\XXX\XXX.pyz", line 98, in XXX
File "C:\XXX\XXX\XXX.pyz", line 31, in XXX
File "C:\XXX\XXX\XXX.pyz", line 24, in XXX
File "C:\XXX\XXX\XXX.pyz", line 34, in XXX
File "C:\Program Files (x86)\PyInstaller-2.1\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\XXX\XXX\out00-PYZ.pyz\wmi", line 157, in <module>
File "C:\XXX\XXX\out00-PYZ.pyz\win32com.client", line 72, in GetObject
File "C:\XXX\XXX\out00-PYZ.pyz\win32com.client", line 87, in Moniker
wmi.py line 157 has a global call to GetObject:
obj = GetObject ("winmgmts:")
win32com\client__init.py__ contains GetObject(), which ends up calling Moniker():
def GetObject(Pathname = None, Class = None, clsctx = None):
"""
Mimic VB's GetObject() function.
ob = GetObject(Class = "ProgID") or GetObject(Class = clsid) will
connect to an already running instance of the COM object.
ob = GetObject(r"c:\blah\blah\foo.xls") (aka the COM moniker syntax)
will return a ready to use Python wrapping of the required COM object.
Note: You must specifiy one or the other of these arguments. I know
this isn't pretty, but it is what VB does. Blech. If you don't
I'll throw ValueError at you. :)
This will most likely throw pythoncom.com_error if anything fails.
"""
if clsctx is None:
clsctx = pythoncom.CLSCTX_ALL
if (Pathname is None and Class is None) or \
(Pathname is not None and Class is not None):
raise ValueError("You must specify a value for Pathname or Class, but not both.")
if Class is not None:
return GetActiveObject(Class, clsctx)
else:
return Moniker(Pathname, clsctx)
The first line in Moniker(), i.e. MkParseDisplayName() is where the exception is encountered:
def Moniker(Pathname, clsctx = pythoncom.CLSCTX_ALL):
"""
Python friendly version of GetObject's moniker functionality.
"""
moniker, i, bindCtx = pythoncom.MkParseDisplayName(Pathname)
dispatch = moniker.BindToObject(bindCtx, None, pythoncom.IID_IDispatch)
return __WrapDispatch(dispatch, Pathname, clsctx=clsctx)
Note: I tried using
pythoncom.CoInitialize()
which apparently solves this import problem within a thread, but that didn't work...

I also face the same issue and I figure out this issue finally,
import pythoncom and CoInitialize pythoncom.CoInitialize (). They import wmi
import pythoncom
pythoncom.CoInitialize ()
import wmi

I tried solving this countless ways. In the end, I threw in the towel and had to just find a different means of achieving the same goals I had with wmi.
Apparently that invalid syntax error is thrown when trying to create an object with an invalid "moniker name", which can simply mean the service, application, etc. doesn't exist on the system. Under this circumstance "winmgmts" just can't be found at all it seems! And yes, I tried numerous variations on that moniker with additional specs, and I tried running the service under a different user account, etc.

Honestly I didn't dig in order to understand why this occurs.
Anyway, the below imports solved my problem - which was occurring only when ran from a Flask instance:
import os
import pythoncom
pythoncom.CoInitialize()
from win32com.client import GetObject
import wmi

The error "com_error: (-2147221020, 'Invalid syntax', None, None)" is exactly what popped up in my case so I came here after a long time of searching the web and voila:
Under this circumstance "winmgmts" just can't be found at all it
seems!
This was the correct hint for because i had just a typo , used "winmgmt:" without trailing 's'. So invalid sythax refers to the first methods parameter, not the python code itself. o_0 Unfortunately I can't find any reference which objects we can get with win32com.client.GetObject()... So if anybody has a hint to which params are "allowed" / should work, please port it here. :-)
kind regards
ChrisPHL

How to integrate APScheduler and Imp?

I have built a plugin-based application where "plugins" (python modules) can be loaded by imp and then scheduled for later execution by APScheduler, I was able to successfully integrate them but I want to implement persistence in case of crashes or application reestarts, so I changed the default memory job store to the SqlAlchemyJobStore, it works quite well the first time you execute the program: tasks are loaded, scheduled, saved at the database and executed at the right time.
Problem is when I try to load the application again I get this traceback:
ERROR:apscheduler.jobstores.default:Unable to restore job "d3e0f0068df54d15986e9b7b6757f665" -- removing it
Traceback (most recent call last):
File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py", line 126, in _get_jobs
jobs.append(self._reconstitute_job(row.job_state))
File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py", line 114, in _reconstitute_job
job.__setstate__(job_state)
File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/job.py", line 228, in __setstate__
self.func = ref_to_obj(self.func_ref)
File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/util.py", line 257, in ref_to_obj
raise LookupError('Error resolving reference %s: could not import module' % ref)
LookupError: Error resolving reference __init__:run: could not import module
So it is obvious that there is a problem when attempting to import the function again
Here is my scheduler initialization:
executors = {'default': ThreadPoolExecutor(5)}
jobstores = {'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')}
self.scheduler = BackgroundScheduler(executors = executors,jobstores=jobstores)
I have a "tests" dictionary containing the "plugins" that should be loaded and some parameters, "load_plugin" uses imp to load a plugin by it's name.
for test,parameters in tests.items():
if test in pluggins:
module=load_plugin(pluggins[test])
self.jobs[test]=self.scheduler.add_job(module.run,"interval",seconds=parameters["interval"],name=test)
Any idea about how can I handle reconstituting jobs?

Something in the automatic detection of the module name is going wrong. Hard to say what, but the alternative is to manually give it the proper lookup path as a string (e.g. "package.module:function"). If you can do this, you can avoid this problem.

Ambari server setup: 'NoneType' object has no attribute 'lower'

I'm trying to setup Hadoop on my EC2 instance using this tutorial. I'm trying to setup the ambari server when I get this error:
[root#ip-xxx-xxx-xxx-xxx ec2-user]# ambari-server setup
Using python /usr/bin/python2.6
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Ambari-server daemon is configured to run under user 'root'. Change this setting [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking iptables...
Checking JDK...
JCE Policy archive already exists, using /var/lib/ambari-server/resources/jce_policy-6.zip
Completing setup...
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 4236, in <module>
main()
File "/usr/sbin/ambari-server.py", line 4055, in main
setup(options)
File "/usr/sbin/ambari-server.py", line 2089, in setup
retcode = configure_os_settings()
File "/usr/sbin/ambari-server.py", line 1909, in configure_os_settings
os_name = os_info[0].lower()
AttributeError: 'NoneType' object has no attribute 'lower'
I'm really unsure how this is happening and don't know what to do fix this. Does anyone know what I'm doing wrong?
Edit: I looked at the code at found this:
os_info = platform.linux_distribution(
None, None, None, ['SuSE', 'redhat' ], 0
)
os_name = os_info[0].lower()
It appears that platform.linux_distribution is creating an array with it's parameters and doing something else to it. I can't find the implementation of the function in the file and there are several files that are in its same directory, still not sure what should I do.

I solved the problem, I removed the 3 None objects and removed the 2-size array and made them their own parameters. However now Ambari is causing another problem.

Is it possible to serialize tasklet code (not just exec state) using SPickle without doing a RPC?

Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.
Been referencing following presentation:
https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle
Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery:
Client code:
from stackless import run, schedule, tasklet
from sPickle import SPickleTools
def test_method():
print "hello from test method"
tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)
pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)
Server code:
pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)
Results in:
[2012-03-09 14:24:59,104: ERROR/MainProcess] Task
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'
Any suggestions on what I am doing incorrect or if this is even possible?
Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). I have experimented with doing:
py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)
but the dynamically loaded module does not seem to persist through different calls to celeryd, i.e. different remote calls.

You must define test_method within its own module. Currently sPickle detects whether test_method is defined in a module that can be imported. An alternative way is to set the __module__ attribute of the function to None.
def test_method():
pass
test_method.__module__ = None