I'm trying to reload module that's already loaded within sys.modules, but somehow, reload() doesn't seem to work.
(compile()+load would work although, still I can't use that since it's code reload through twisted)
for loader, module_name, is_pkg in pkgutil.walk_packages(main_module.__path__):
if(not module_name in sys.modules):
module = loader.find_module(module_name).load_module(module_name)
else:
module = sys.modules.get(module_name)
#this is unhelpful
#sys.path.append(module.__path__)
#this works
#imp.find_module(module.__name__, module.__path__)
#this doesn't
reload(module)
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/celery/execute/trace.py", line 36, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/usr/lib64/python2.7/site-packages/celery/app/task/__init__.py", line 232, in __call__
return self.run(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/celery/app/__init__.py", line 172, in run
return fun(*args, **kwargs)
File "/home/myuser/Dropbox/job-pipe/job_pipe/apps/tasks/crawl.py", line 71, in crawl
crawler = CrawlerScript()
File "/home/myuser/Dropbox/job-pipe/job_pipe/apps/tasks/crawl.py", line 37, in __init__
reload(module)
ImportError: No module named example
What's the proper way to reload? I thought it was caused by find_module, though it seems to work properly, and error is not helpful.
Thanks.
Once you have loaded a module, to reload it you must do
module = reload(module)
Related
We are trying to add a z sql method to access the database. We have already done this and tested it. However, when following instructions about how to access the method in a script (Setting a variable for a Z SQL method and http://docs.zope.org/zope2/zope2book/RelationalDatabases.html), the script fails with this error in the instance log:
AttributeError: MethodName
This is when calling : context.MethodName
The function is intended for use in a customized 'portal_skins/plone_scripts'
Using: Plone 4.2.6
Anyone have a clue?
2014-01-24T12:48:16 ERROR Zope.SiteErrorLog 1390585696.260.938992561119 http://foo.com/mail_password
Traceback (innermost last):
Module ZPublisher.Publish, line 138, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 48, in call_object
Module Shared.DC.Scripts.Bindings, line 322, in __call__
Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 344, in _exec
Module script, line 5, in mail_password
- <CustomizedPythonScript at /mail_password>
- Line 5
Module AccessControl.ImplPython, line 675, in guarded_getattr
AttributeError: InsertMessage
The insert message is called : context.InsertMessage(variables) in 'portal_skins/plone_scripts/mail_password' script
I'm trying to debug an application which makes use of the pynetdicom library. I'm not sure how relevant that specific detail is, however what IS relevant is that it makes heavy use of multithreading to run background socket listener tasks without blocking the main thread. The storescp.py example can be used to reproduce this.
Whenever I place a breakpoint that gets encountered (regardless of what thread, main or child, it gets encountered in), I get the following traceback:
Traceback (most recent call last):
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 1397, in <module>
debugger.run(setup['file'], None, None)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 1090, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/Users/alexw/Development/Python/kreport2/KReport2/dicomdatascraper.py", line 183, in <module>
oldDicomList = copy.copy(newData)
File "/Users/alexw/Development/Python/kreport2/KReport2/dicomdatascraper.py", line 183, in <module>
oldDicomList = copy.copy(newData)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_frame.py", line 135, in trace_dispatch
self.doWaitSuspend(thread, frame, event, arg)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_frame.py", line 25, in doWaitSuspend
self._args[0].doWaitSuspend(*args, **kwargs)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 832, in doWaitSuspend
self.processInternalCommands()
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 360, in processInternalCommands
thread_id = GetThreadId(t)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_constants.py", line 140, in GetThreadId
return thread.__pydevd_id__
File "/Users/alexw/.virtualenvs/kreport2dev/devlibs/pynetdicom/source/netdicom/applicationentity.py", line 73, in __getattr__
obj = eval(attr)()
File "<string>", line 1, in <module>
NameError: name '__pydevd_id__' is not defined
My thought is that, perhaps, in order to make things work, PyDev monkey-patches a __pydevd_id__ into spawned threads, however fails to patch those into these threads because they are, in fact, subclasses and not direct instances of threading.Thread (in this case, the worker is an instance of class Association(threading.Thread):).
Of course, I don't know PyDev well enough to confirm this theory, or else fix it. And it seems neither does the internet.
Is subclassing Thread so rarely used a pattern that it's simply not considered in the PyDev architecture? Without re-architecting the library, how could this issue be remedied?
I simply needed to look harder at that traceback.
The pynetdicom library, in its subclassing of threading.Thread, overrode __getattr__ and somewhat broke it. The problem was:
def __getattr__(self, attr):
#while not self.AssociationEstablished:
# time.sleep(0.001)
obj = eval(attr)
# do some stuff
return obj
when a nonexistent attribute is passed, a NameError is raised. This isn't caught by pydev's monkeypatching routine (if thread.__pydevd_id__ raises AttributeError, thread.__pydevd_id__ = stuff)
The solution was to update that section thusly:
def __getattr__(self, attr):
#while not self.AssociationEstablished:
# time.sleep(0.001)
try:
obj = eval(attr)
except NameError:
raise AttributeError
# do some stuff
return obj
This intercepts the NameError and raises an AttributeError instead, as __getattr__ should if the queried attribute doesn't exist.
I am trying to run Pyramid with Jinja2 using new Python 2.7 runtime in threadsafe mode and GAE 1.6.0 pre-release SDK. I've made modifications to my app as outlined here, i.e. I've set runtime: python27, threadsafe: true in app.yaml and got rid of main() function. When I generate response by myself it works fine, but when I try to bring jinja2 into the equation, I get the following exception:
ERROR 2011-11-07 00:10:34,356 wsgi.py:170]
Traceback (most recent call last):
File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
[...]
File "/myapp/source/myapp-tip/main.py", line 29, in <module>
config.include('pyramid_jinja2')
File "/myapp/source/myapp-tip/lib/dist/pyramid/config/__init__.py", line 616, in include
c(configurator)
File "lib/dist/pyramid_jinja2/__init__.py", line 390, in includeme
_get_or_build_default_environment(config.registry)
File "/lib/dist/pyramid_jinja2/__init__.py", line 217, in _get_or_build_default_environment
_setup_environment(registry)
File "/lib/dist/pyramid_jinja2/__init__.py", line 253, in _setup_environment
package = _caller_package(('pyramid_jinja2', 'jinja2', 'pyramid.config'))
File "/lib/dist/pyramid_jinja2/__init__.py", line 136, in caller_package
for t in self.inspect.stack():
File "/usr/lib/python2.7/inspect.py", line 1056, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/lib/python2.7/inspect.py", line 1034, in getouterframes
framelist.append((frame,) + getframeinfo(frame, context))
File "/usr/lib/python2.7/inspect.py", line 1009, in getframeinfo
lines, lnum = findsource(frame)
File "/usr/lib/python2.7/inspect.py", line 534, in findsource
module = getmodule(object, file)
File "/usr/lib/python2.7/inspect.py", line 506, in getmodule
main = sys.modules['__main__']
KeyError: '__main__'
I tried to mess around a bit with pyramid_jinja2 code to work around this issue, only to be left with another exception:
ERROR 2011-11-04 12:06:38,720 wsgi.py:170]
Traceback (most recent call last):
File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
[...]
File "/myapp/source/myapp-tip/main.py", line 29, in <module>
config.add_jinja2_search_path("templates")
File "/myapp/source/myapp-tip/lib/dist/pyramid/config/util.py", line 28, in wrapper
result = wrapped(self, *arg, **kw)
File "/lib/dist/pyramid_jinja2/__init__.py", line 311, in add_jinja2_search_path
env.loader.searchpath.append(abspath_from_resource_spec(d))
File "/myapp/source/myapp-tip/lib/dist/pyramid/asset.py", line 38, in abspath_from_asset_spec
return pkg_resources.resource_filename(pname, filename)
File "/myapp/source/myapp-tip/pkg_resources.py", line 840, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/myapp/source/myapp-tip/pkg_resources.py", line 160, in get_provider
__import__(moduleOrReq)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1756, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1628, in FindAndLoadModule
description)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1571, in LoadModuleRestricted
description)
ImportError: Cannot re-init internal module __main__
I'd be happy if anybody could shed some light on what pyramid is trying to do under the hood. Judging by the latter stack trace it seems it's trying to resolve an asset, but why is it trying to reload __main__? I'm not even sure my problem is caused by pyramid or GAE.
Thanks for any insight on this issue.
I'm not familiar with pyramid, but the problem really does seem to be with this line:
config.include('pyramid_jinja2')
Whatever that config thing is, it seems to be doing some dynamic import magic.
Don't do that.
The app engine environment doesn't handle imports the way you would in normal python. Step through that line with a debugger and you'll wind up in the replacement version of the import system, which you'll soon see, only implements a small part of what real python does.
If possible, just use a normal import statement... Otherwise, you're going to have to dig into config.include and get it to play nice with the restricted importing features on GAE.
I managed to make it work using Pyramid 1.3's AssetResolver. First attempt is here. Just not sure what the lifetime/scope of the resolver should be in this case, I will figure it out later.
In pyramid_jinja2/__init__.py add the following code before _get_or_build_default_environment()
class VirtualModule(object):
def __init__(self,name):
import sys
sys.modules[name]=self
def __getattr__(self,name):
return globals()[name]
VirtualModule("__main__")
def _get_or_build_default_environment(registry):
(http://www.inductiveautomation.com/forum/viewtopic.php?f=70&p=36917)
I am trying to get started with using nosegae, however I run into the issue that I can't seem to get it to pass even the simplest of cases when using django.
when running without the --without-sandbox flag both the following tests fail
def test_import_django ():
import django
def test_import_django_http ():
import django.http
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 2, in test_import_d
jango
import django
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
207, in find_module
return super(HookMixin, self).find_module(fullname, path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1998, in find_module
search_path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2119, in FindModuleRestricted
result = self.FindPathHook(submodule, submodule_fullname, path_entry)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2219, in FindPathHook
return self._imp.find_module(submodule, [path_entry])
Howevere if I do use --without-sandbox at least the first test passes
myproj.tests.test_import_django ... ok
myproj.tests.test_import_django_http ... ERROR
======================================================================
ERROR: myproj.tests.test_import_django_http
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 5, in test_import_d
jango_http
import django.http
File "C:\Program Files (x86)\Google\google_appengine\lib\django_1_2\django\htt
p\__init__.py", line 9, in <module>
from mod_python.util import parse_qsl
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
199, in find_module
mod_path = self.find_mod_path(fullname)
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
251, in find_mod_path
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
Has anyone encountered and know how I can go about past this?
Edit
It seems that the issue is recursive imports
def test_import_pdb ():
import pdb
pdb.set_trace ()
part of the stack trace is
File "C:\Python27\lib\pdb.py", line 72, in __init__
import readline
notice that an import in __init__ of django.http is also part of the stack trace
Read https://docs.djangoproject.com/en/dev/topics/testing/ about Django testing.
As I know it's better to use unittest or doctest shipped with django as it have several improvements for django-specific testing like form field output testing and some database features. Hovewer it's not essential and if you want to continue using nose - think you missed django environment setup:
from django.test.utils import setup_test_environment
setup_test_environment()
This lines needed to run your tests outside of ./manage.py --test
UPD
Yeah my previous thought's were wrong. So I just digged into sources of nose and nose-gae, and what I think - check HardenedModulesHook definition in your nose version, cause in trunk of nose I've found following:
class HardenedModulesHook(object):
...
def __init__(self,
module_dict,
imp_module=imp,
os_module=os,
dummy_thread_module=dummy_thread,
pickle_module=pickle):
...
That gives following - when noseGAE plugin begin() method is executed -> there self._install_hook(dev_appserver.HardenedModulesHook) is called which declares mixed-hook class and creates it's instance like self.hook = Hook(sys.modules, self._path). <- There is HardenedModulesHook.__init__ called with second argument as mystic '_path' however in NOSE this argument should be 'imp' module by default -> That makes an exception you've got:
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
So I think it might be a problem with nose-gae :(
I was trying to install a module for opencv and added an opencv.pth file to the folder beyond my sites.py file. I have since deleted it and no change.
When I try to run help('modules'), I get the following error:
Please wait a moment while I gather a
list of all available modules...
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/twisted/words/im/init.py:8:
UserWarning: twisted.im will be
undergoing a rewrite at some point in
the future.
warnings.warn("twisted.im will be
undergoing a rewrite at some point in
the future.")
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pkgutil.py:110:
DeprecationWarning: The wxPython
compatibility package is no longer
automatically generated or actively
maintained. Please switch to the wx
package as soon as possible.
import(name) Traceback (most recent call last): File "",
line 1, in File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.py",
line 348, in call
return pydoc.help(*args, **kwds) File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pydoc.py",
line 1644, in call
self.help(request) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pydoc.py",
line 1681, in help
elif request == 'modules': self.listmodules() File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pydoc.py",
line 1802, in listmodules
ModuleScanner().run(callback) File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pydoc.py",
line 1853, in run
for importer, modname, ispkg in pkgutil.walk_packages(): File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pkgutil.py",
line 110, in walk_packages
import(name) File "/BinaryCache/wxWidgets/wxWidgets-11~262/Root/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/wxaddons/init.py",
line 180, in import_hook File
"/Library/Python/2.5/site-packages/ctypes_opencv/init.py",
line 19, in
from ctypes_opencv.cv import * File
"/BinaryCache/wxWidgets/wxWidgets-11~262/Root/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/wxaddons/init.py",
line 180, in import_hook File
"/Library/Python/2.5/site-packages/ctypes_opencv/cv.py",
line 2567, in ('desc', CvMat_r, 1), # CvMat* desc File
"/Library/Python/2.5/site-packages/ctypes_opencv/cxcore.py",
line 114, in cfunc
return CFUNCTYPE(result, *atypes)((name, dll), tuple(aflags)) AttributeError: dlsym(0x2674d10, cvCreateFeatureTree): symbol not found
What gives?!
This happens because help('modules') imports all modules, which can result in a lot of unsentineled code being executed. There's nothing you can do short of reporting bugs in every single package that causes this (opencv in this case) and wait for them to fix it.