I'm trying to extend StaticFileHandler in such a way that I can process file requests but call self.render(filename, **kwargs) on the file to actually serve it to the client. (Yes, I realize that at that point it's no longer a static file per se).
Here's the code I'm trying to run:
class MustacheFileHandler(tornado.web.StaticFileHandler):
def get(self, filename):
self.render(_STATIC_ROOT_ + '/' + path.split('/')[len(path.split('/'))-2], userLoginStatus='you are logged out!')
# ...
class Application(tornado.web.Application):
def __init__(self, **overrides):
handlers = [(r'/(.*)', MustacheFileHandler, {'path' : _STATIC_ROOT_})]
# ...
... Where _STATIC_ROOT_ is a variable included in my server's config file loaded on startup.
The issue I've got is, whenever I try to do a GET on a file I know exists on the server, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1332, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "myfile.py", line 173, in get
self.render(_STATIC_ROOT_ + '/' + path.split('/')[len(path.split('/'))-2], userLoginStatus='you are logged out!')
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 747, in render
self.finish(html)
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 877, in finish
self.set_etag_header()
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1257, in set_etag_header
etag = self.compute_etag()
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 2185, in compute_etag
version_hash = self._get_cached_version(self.absolute_path)
AttributeError: 'MustacheFileHandler' object has no attribute 'absolute_path'
I'm not sure what's causing this error or how I can handle it.
Why are you using StaticFileHandler if the response is not static? This is likely to break assumptions built into the class.
StaticFileHandler is designed to be subclassed in limited ways as described in its documentation. In particular, subclasses should not override get(), and attempting to do so results in the error you're seeing.
If you want to use the Tornado template engine as a kind of preprocessor of the files on disk, you could try to do this by overriding both get_content and get_content_size, and making them call self.render_string() (also consider that if your templates are not individually self-contained you'll need to change get_content_version to take all dependencies into account). However, this requires messy caching to avoid rendering the template multiple times. It's probably better to either A) Render the templates on the fly with an ordinary RequestHandler. or B) Write a little script to render all your templates, write them to disk, and serve the results as actual static files.
Related
We use the async version of sqlalchemy and we need to eager load every relationship (lazy loading does not work for async). Accordingly for every relationship in our models we have set lazy='raise'. The problem is that it seems overly aggressive on raising errors. Consider the following unit test:
async def test_user_group_self_allowed(self):
privilege = await self.db.get(Privilege, 1, [joinedload(Privilege.role)])
options = [joinedload(Item.privileges).joinedload(Privilege.role), joinedload(Item.item_group)]
item = await self.db.get(Item, 1, options)
item.privileges.append(privilege)
await (self.db.commit())
options = [joinedload(User.user_groups).joinedload(UserGroup.privileges), joinedload(User.privileges)]
user = await self.db.get(User, 1, options)
user.privileges = []
item = await self.db.get(Item, 1, [joinedload(Item.privileges).joinedload(Privilege.role), joinedload(Item.item_group)])
user_group = await self.db.get(UserGroup, 1, [joinedload(UserGroup.organization)])
print('why?????', user_group.organization)
self.assertTrue(await self.helper.is_authorized(self.db, user, 'edit', user_group))
Notice the print, it results in the following error:
Traceback (most recent call last):
File "/usr/lib/python3.9/unittest/async_case.py", line 65, in _callTestMethod
self._callMaybeAsync(method)
File "/usr/lib/python3.9/unittest/async_case.py", line 88, in _callMaybeAsync
return self._asyncioTestLoop.run_until_complete(fut)
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/usr/lib/python3.9/unittest/async_case.py", line 102, in _asyncioLoopRunner
ret = await awaitable
File "/src/backend-core/backend_core/tests/authorization.py", line 206, in test_user_group_self_allowed
print('why?????', user_group.organization)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/orm/attributes.py", line 481, in __get__
return self.impl.get(state, dict_)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/orm/attributes.py", line 926, in get
value = self._fire_loader_callables(state, key, passive)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/orm/attributes.py", line 962, in _fire_loader_callables
return self.callable_(state, passive)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/orm/strategies.py", line 836, in _load_for_state
self._invoke_raise_load(state, passive, "raise")
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/orm/strategies.py", line 795, in _invoke_raise_load
raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: 'UserGroup.organization' is not available due to lazy='raise'
As you can see it complains about the organization not being eager loaded while I clearly include it in the options with a joinedload. Now we can make this error go away by changing the options for the user query to:
options = [joinedload(User.user_groups).joinedload(UserGroup.privileges), joinedload(User.privileges), joinedload(User.user_groups).joinedload(UserGroup.organization)]
(the same options as before, only we add a joinedload for User -> UserGroups -> Organization)
This makes the error go away and everything is fine again. Now my question is, why does it complain about this to start with? I access user_group.organization not user.user_groups[x].organization.. I don't know how these queries work under the hood exactly, but not only do I have to write way too many joinedloads this way, I think it also results in needless querying.
As it turns out, .get caches more than I would expect. Not only the main entity (in this case usergroup) but also stuff loaded through a joinedload (user.user_group.organization). So that means a direct get on the organization does not overwrite the cached one from user.user_group.organization or usergroup.organization. However it is possible to do e.g. db.get(Organization, 1, populate_existing=True) which will retrieve the entity again and update the cache.
From the docs:
If the given primary key identifier is present in the local identity map, the object is returned directly from this collection and no SQL is emitted, unless the object has been marked fully expired....
...
populate_existing – causes the method to unconditionally emit a SQL query and refresh the object with the newly loaded data, regardless of whether or not the object is already present.
As it tells, another way is to expire an object, read more about that here
I'm running into the following error using get_serving_url in order to serve images from my bucket.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1087, in synctasklet_wrapper
return taskletfunc(*args, **kwds).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 1057, in tasklet_wrapper
result = func(*args, **kwds)
File "/base/data/home/apps/e~tileserve20171207t210519.406056180994857717/blob_upload.py", line 70, in post
bf.put_async()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3473, in _put_async
self._pre_put_hook()
File "/base/data/home/apps/e~tileserve/20171207t210519.406056180994857717/blob_files.py", line 124, in _pre_put_hook
print images.get_serving_url(None, filename='/gs' + '/tileserve.appspot.com/user2test4test4RGB20170927.tif')
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 1868, in get_serving_url
return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 1972, in get_serving_url_hook
raise _ToImagesError(e, readable_blob_key)
TransformationError
When I upload an image to my bucket then it works but when I create an image through processing which should be exposed through get_serving_url I get TransformationError.
I tried two variants for serving images:
test = blobstore.create_gs_key('/gs' + '/path2object')
images.get_serving_url(test, secure_url=True)
images.get_serving_url(None, filename='/gs' + '/' + <bucket name>+ '/'+ <object name>)
I also set the bucket object ACM permissions and the IAM App Engine app default service account permissions (storage admin). Both variants changed nothing but are important in order to access objects of a bucket.
Did somebody experience this issue ? What could be the error? I do not understand why it works when I upload an image but not for images which are generated through processing.
The traceback suggests you may be trying to call images.get_serving_url() while an async operation may still be in progress.
If that op is actually saving the transformed image in GCS then it could explain the failure: get_serving_url() includes a check of the file being a valid image, which would fail with TransformationError if the file is not yet saved.
If so - you can fix the problem by either:
switching to the sync version of saving the image
waiting until saving the image completes (in case you have some other stuff to do in the meantime) - get the result of the sync call before invoking get_serving_url()
repeatedly trying get_serving_url() while catching TransformationError until it no longer fails that way. This is a bit risky as it can end up in an infinite loop if TransformationError is raised for reasons other than simply saving the image being incomplete.
The issue is not with the get_serving_url() method. It is how you are accessing the objects from google cloud storage bucket.
I switched the access control on my bucket from uniform to fine grained which fixed the problem for me.
I am trying to setup an application based on the Google App Engine using the Managed VM feature.
I am using a shared library written in C++ using ctypes
cdll.LoadLibrary('./mylib.so')
which registers a callback function
CB_FUNC_TYPE = CFUNCTYPE(None, eSubscriptionType)
cbFuncType = CB_FUNC_TYPE(scrptCallbackHandler)
in which i want to save data to the ndb datastore
def scrptCallbackHandler(arg):
model = Model(name=str(arg.data))
model.put()
I am registering a callback function in which i want to take the Data from the C++ program and put it in the ndb datastore. This results in an error. On the devserver it behaves slightly different, so from a production server:
suspended generator _put_tasklet(context.py:343) raised BadRequestError(Application Id (app) format is invalid: '_')LOG 2 1429698464071045 suspended generator put(context.py:810) raised BadRequestError(Application Id (app) format is invalid: '_')
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function' File "/home/vmagent/app/isw_cloud_client.py", line 343, in scrptCallbackHandler node.put()
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/model.py", line 3380, in _put return self._put_async(**ctx_options).get_result()
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/tasklets.py", line 325, in get_result self.check_success()
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along value = gen.throw(exc.__class__, exc, tb)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/context.py", line 810, in put key = yield self._put_batcher.add(entity, options)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along value = gen.throw(exc.__class__, exc, tb)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/context.py", line 343, in _put_tasklet keys = yield self._conn.async_put(options, datastore_entities)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion result = rpc.get_result()
File "/home/vmagent/python_vm_runtime/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self)
File "/home/vmagent/python_vm_runtime/google/appengine/datastore/datastore_rpc.py", line 1827, in __put_hook self.check_rpc_success(rpc)
File "/home/vmagent/python_vm_runtime/google/appengine/datastore/datastore_rpc.py", line 1342, in check_rpc_success raise _ToDatastoreError(err)google.appengine.api.datastore_errors.BadRequestError: Application Id (app) format is invalid: '_'
The start of the C++ program is triggered by a call to a Request handler but runs in the background and accepts incoming data which should be processed in the callback.
Update: As Tim pointed out already it seems that the context of the wsgi handler is lost. Most likely the solution here would be to create the application context somehow.
I am only guessing what is my problem and i want to tell what i did to solve it.
The execution context of the callback functions is somewhat different than the rest of the python application. Any asynchronous operation in the callback fails. I tried doing an http call or saving it to the datastore. The operations never finish and after 60s the application shows an error that they crashed. I guess this is because how the python manages the execution and the corresponding memory allocation.
I was able to execute the callback in an object's context by wrapping it in a closure within a class. This wasnt really the problem but the solution can be found in this answer: How can I get methods to work as callbacks with python ctypes?
For my solution i am now using a combination of cloud-endpoints on another module and background threads on the ctypes-module.
Within the C-Callback i start a background thread, which is able to do asynchronous work
# Start a background thread using the background thread service from GAE
background_thread.start_new_background_thread(putData, [name, value])
And here the simple task it executes:
# Here i call my cloud-endpoints
def putData(name, value):
body = {
'name' : 'name',
'value' : int(value)
}
res = service.objects().create(body=body).execute()
Of course i need to do error handling and additional stuff, but for me this is a good solution.
Note: Adding models to the datastore in the bg thread failed because the environment in the bg thread is different from the application and the app id was not set.
I've been trying to start profiling my CherryPy webserver, but the documentation is lacking in detail in how this should be set up. I understand that I should be able to use cherrypy.lib.profiler as middleware to mount my initial server. Right now, I have code like the following:
server_app = ServerClass()
cherrypy.tree.mount(server_app, '/', '/path/to/config/file.cfg')
cherrypy.engine.start()
cherrypy.engine.block()
I want to mount the profiling middleware, and it seems that something like the following is required:
from cherrypy.lib import profiler
server_app = ServerClass()
server_cpapp = cherrypy.Application(server_app, '/', '/path/to/config/file.cfg')
server_profile_cpapp = profiler.make_app(server_cpapp, '/home/ken/tmp/cprofile', True)
#cherrypy.tree.mount(server_profile_cpapp)
cherrypy.tree.graft(server_profile_cpapp)
cherrypy.engine.start()
cherrypy.engine.block()
For some reason cherrypy.tree.mount doesn't work, but if I use cherrypy.tree.graft all seems to operate fine (I can make requests to the server as normal)
However, the above code generates a cp_0001.prof file under /home/ken/tmp/cprofile and I am not sure how to interpret it. I have tried using pyprof2calltree to read the data into KCacheGrind, but I get a parsing error. Does what I'm doing seem correct, and if so how do I interpret the output file?
It turns out that the profile files generated by CherryPy can be interpreted using the profiler.py script shipped as part of CherryPy. Simply run profiler.py in the <site-packages>/cherrypy/lib directory as follows:
python profiler.py /directory/containing/prof/files 8080
Then navigate to localhost:8080 in your browser and the profiling results for all .prof files in the target directory will be displayed in a simple text interface.
I would still prefer to be able to export the results into a calltree to profile using KCacheGrind, but this seems to do for basic profiling.
This is documented in the change log for v2.1 of CherryPy when the profiler was introduced (although the other details on that page describing how to set up the profiler has since become deprecated)
I am also trying to get profiling up and running for a cherrypy instance. I used the same code you have in your initial question, which seems to work in that it generates a cp_0001.prof file in the folder.
To answer your question, I am able to open this file in runsnakerun to see the profiling output in a tree view.
The problem I have is that every request I do to the server now fails, with the following output in the log:
[29/May/2013:16:39:32] ENGINE AssertionError('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
Traceback (most recent call last):
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "<path>\packages\cherrypy\_cptree.py", line 290, in __call__
return app(environ, start_response)
File "<path>\packages\cherrypy\lib\profiler.py", line 188, in __call__
return self.profiler.run(gather)
File "<path>\packages\cherrypy\lib\profiler.py", line 147, in run
result = self.profiler.runcall(func, *args)
File "<path>\python\lib\profile.py", line 472, in runcall
return func(*args, **kw)
File "<path>\packages\cherrypy\lib\profiler.py", line 183, in gather
def gather():
File "<path>\python\lib\profile.py", line 246, in trace_dispatch_i
if self.dispatch[event](self, frame, t):
File "<path>\python\lib\profile.py", line 301, in trace_dispatch_call
frame, frame.f_back)
AssertionError: ('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
I am using python 2.6.6 and cherrypy 3.2.2
Any suggestions?
I have a program that uses the win32com library to control iTunes, but have been having some issues getting it to compile into an executable. The problem seems to revolve around using DispatchWithEvents instead of Dispatch. I've created a very simple program to illustrate my problem:
import win32com.client
win32com.client.gencache.is_readonly = False #From py2exe wiki
class ITunesEvents(object):
def __init__(self): self.comEnabled = True
def OnCOMCallsDisabledEvent(self, reason): self.comEnabled = False
def OnCOMCallsEnabledEvent(self): self.comEnabled = True
# The first line works in the exe, the second doesn't.
itunes = win32com.client.Dispatch("iTunes.Application")
#itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
lib = getattr(itunes, "LibraryPlaylist")
src = getattr(lib, "Source")
playlists = getattr(src, "Playlists")
print "Found %i playlists." % getattr(playlists, "Count")
Using Dispatch, the program compiles and runs correctly. Using DispatchWithEvents, the program runs fine when called from the command line, but produces the following error when running the exe:
Traceback (most recent call last):
File "sandbox.py", line 16, in <module>
itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
File "win32com\client\__init__.pyc", line 252, in DispatchWithEvents
File "win32com\client\gencache.pyc", line 520, in EnsureModule
File "win32com\client\gencache.pyc", line 287, in MakeModuleForTypelib
File "win32com\client\makepy.pyc", line 259, in GenerateFromTypeLibSpec
File "win32com\client\gencache.pyc", line 141, in GetGeneratePath
IOError: [Errno 2] No such file or directory: '[distDir]\\library.zip\\win32com\\gen_py\\__init__.py'
I've also tried using PyInstaller, which gives a similar error:
File "<string>", line 16, in <module>
File "[outDir]/win32com.client", line 252, in DispatchWithEvents
File "[outDir]/win32com.client.gencache", line 520, in EnsureModule
File "[outDir]/win32com.client.gencache", line 287, in MakeModuleForTypelib
File "[outDir]/win32com.client.makepy", line 286, in GenerateFromTypeLibSpec
File "[outDir]/win32com.client.gencache", line 550, in AddModuleToCache
File "[outDir]/win32com.client.gencache", line 629, in _GetModule
File "[pyinstallerDir]\iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named win32com.gen_py.9E93C96F-CF0D-43F6-8BA8-B807A3370712x0x1x13
I know I can manually add the typelib in my setup.py file, but I'd like to run the code on computers with different versions of iTunes without recompiling so I'd prefer to dynamically create it. If there's no way to do this with the setup/spec, maybe there is another way to load the events? Thanks.
Addition:
Thanks to Ryan, I found I could take the generated py file and after a little digging, was able to come up with the following.
Take the generated py file (from makepy.py) and rename it somewhere like cominterface.py. Then you'll need to do the following to actually create the COM object with event handler.
import cominterface
from types import ClassType
from win32com.client import EventsProxy, _event_setattr_
class ItunesEvents:
'''iTunes events class. See cominterface for details.'''
def OnPlayerPlayEvent(self, t):print "Playing..."
def OnPlayerStopEvent(self, t): print "Stopping..."
itunes = cominterface.iTunesApp()
rClass = ClassType("COMEventClass", (itunes.__class__, itunes.default_source, ItunesEvents), {'__setattr__': _event_setattr_})
instance = rClass(itunes._oleobj_)
itunes.default_source.__init__(instance, instance)
#ItunesEvents.__init__(instance) #Uncomment this line if your events class has __init__.
itunes = EventsProxy(instance)
Then you can go about your business.
I was experiencing the exact same error.
This link put me in the right direction -->
http://www.py2exe.org/index.cgi/UsingEnsureDispatch
however it mentions that :
NB You must ensure that the python...\win32com.client.gen_py dir does not exist
to allow creation of the cache in %temp%
Which was a bit confusing.
What solved it for me was renaming "C:\Python26\Lib\site-packages\win32com\gen_py" to "C:\Python26\Lib\site-packages\win32com\gen_pybak" (when running py2exe)
This is the official way to do it.
(Edit: copied verbatim example from that link above)
import win32com.client
if win32com.client.gencache.is_readonly == True:
#allow gencache to create the cached wrapper objects
win32com.client.gencache.is_readonly = False
# under p2exe the call in gencache to __init__() does not happen
# so we use Rebuild() to force the creation of the gen_py folder
win32com.client.gencache.Rebuild()
# NB You must ensure that the python...\win32com.client.gen_py dir does not exist
# to allow creation of the cache in %temp%
# Use SAPI speech through IDispatch
from win32com.client.gencache import EnsureDispatch
from win32com.client import constants
voice = EnsureDispatch("Sapi.SpVoice", bForDemand=0)
voice.Speak( "Hello World.", constants.SVSFlagsAsync )
Instead of depending on the cache, I'd recommend going into the local cache directory, copying the generated file into your local project file, and naming it something like ITunesInterface.py, and calling to that explicitly. This will make py2exe pull it into your compiled app.