I am using the Parallel Python module (pp), and want to submit a job to a worker. However, the function that I want to execute is in another module (written with Cython), and I don't know how to import the function name to the new worker. The method suggested here, i.e importing the module "walkerc" inside the function cannot work since walk itself is defined in walkerc, from the filename "walkerc.so"
import pp
from walkerc import walk
# Other stuff here
ser = pp.Server()
# Some more definitions
ser.submit(walk, (it, params))
ser.submit(walk, (1000, params), modules = ("walkerc",), globals = globals())
Both the statements above fail, I get the following error:
Traceback (most recent call last):
File "", line 1, in
ser.submit(walk, (1000, params), modules = ("walkerc",), globals = globals())
File "/usr/lib/python2.7/site-packages/pp.py", line 458, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/usr/lib/python2.7/site-packages/pp.py", line 629, in
__dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/usr/lib/python2.7/site-packages/pp.py", line 696, in
__get_source
sourcelines = inspect.getsourcelines(func)[0]
File "/usr/lib/python2.7/inspect.py", line 690, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python2.7/inspect.py", line 526, in findsource
file = getfile(object)
File "/usr/lib/python2.7/inspect.py", line 420, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: '<'built-in function walk'>' is not a module, class, method,
function, traceback, frame, or code object
The function 'walk' itself is imported properly within the main program, it is the process of submitting it to a new worker that is problematic.
How can I specify the function name 'walk' properly?
I do not want to define 'walk' in the same file as which I have called it because I have modified it in Cython and want to have better performance. Is there an alternative?
Try renaming your walk function to something else, mywalk for example. As the exception text suggests, your environment seems to have a built-in function that goes by the name walk, so the inspect module gets confused.
I can successfully pass my imported walk function like this on my system, no conflict here and nothing more needed, the function gets executed using the given argument:
import pp
from walkerc import walk
pps = pp.Server()
pps.submit(walk, args=(1,))
But passing dir, which is a built-in function for sure:
pps.submit(dir)
I get the exact same error as you do:
Traceback (most recent call last):
File "parallel.py", line 9, in
pps.submit(dir)
...
File ".../lib/python2.7/inspect.py", line 420, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: is not a module, class, method, function, traceback, frame, or code object
Update after the below discussion:
So the problem here is that Python treats the members that come from C extensions as built-ins. The code above works with the regular Python module, but I was able to replicate the OP's error when importing and passing the function from a C extension.
Therefore I wrapped the C extension function call inside a normal Python function, which does the trick. Note that now the walk function import was moved to the wrapping function, so that it can construct it's own context itself when dispatched.
import pp
def walk(n):
import walkerc
return walkerc.walk(n)
def print_callback(result):
print('callback: ', result)
pps = pp.Server()
job = pps.submit(walk, args=(1,), callback=print_callback)
Related
I've written a bunch of tests for my Python application, but they've suddenly appeared to no longer be working properly!
I've created a bunch of tests inside a tests module:
Inside of tests.__main__.py, I've included the following code to load my test suite:
import unittest
if __name__ == "__main__":
loader = unittest.TestLoader()
start_dir = "."
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
runner = unittest.TextTestRunner()
runner.run(suite)
I know I am going to sound like a complete noob, but these tests were working perfectly for me about an hour ago. I would issue them by typing python3 -m tests from my base directory. However, I'm now receiving a really strange TypeError:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/yuchen/Desktop/myproj/myapp/tests/__main__.py", line 6, in <module>
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 452, in _find_test_path
return self.loadTestsFromModule(module, pattern=pattern), False
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 123, in loadTestsFromModule
tests.append(self.loadTestsFromTestCase(obj))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 92, in loadTestsFromTestCase
loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 24, in __init__
self.addTests(tests)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 57, in addTests
for test in tests:
TypeError: __init__() takes 0 positional arguments but 2 were given
I've followed the stack trace down to what I believe is the appropriate file causing the issue, the suite.py file of unittest:
"""TestSuite"""
import sys
from . import case
from . import util
__unittest = True
def _call_if_exists(parent, attr):
func = getattr(parent, attr, lambda: None)
func()
class BaseTestSuite(object):
"""A simple test suite that doesn't provide class or module shared fixtures.
"""
_cleanup = True
def __init__(self, tests=()):
self._tests = []
self._removed_tests = 0
print("Tests", tests) # <--- this was added by me
self.addTests(tests)
However, the __init__() signature doesn't appear to require 0 positional arguments. Moreover, it's clearly executing, since after I added in print("Tests", tests) in the code above, I saw the following output before the error was thrown:
Tests: []
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: [<unittest.suite.TestSuite tests=[... a list of my tests]]
Tests: <map object at 0x110483978>
Tests: <map object at 0x110483978>
Traceback (most recent call last):
I'm pretty at a loss for what is going on. The error printed in the stack trace doesn't appear to correspond at all with what I'm seeing in the source code.
Edit (Resolution):
The accepted answer was spot on. I had written another test class, and decided to simply add in a stub for the __init__ method:
class MyUnfinishedTest(BaseTest):
def __init__():
pass
Then I had forgotten about it and worked on other stuff, and ran the tests. That's when I started seeing my error. After removing this class from my test module, the tests ran smoothly.
One of the reasons that the traceback seems a little confusing is that the line:
for test in tests
is actually hitting a generator, which is discovering and loading the tests on the fly.
So I'm guessing that this:
TypeError: __init__() takes 0 positional arguments but 2 were given
actually means that the discovery process has found a test class it cannot load, possibly because that class has its own __init__ method with a different number of arguments to what discover expects.
I would double-check your test classes for __init__ methods, and possibly also change this line:
start_dir = "."
to:
start_dir = os.path.dirname(__file__)
That would mean that the discovery process would start looking from the tests folder down, which is safer than "." (current working directory) because the current working directory might change depending upon where you start the tests from (and may end up picking up tests that you weren't expecting).
I'm going to dump the error code I got while try a python script :
Preprocess validation data upfront
Using gpu device 0: Tesla K20c
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "C:\Users\Administrator\Desktop\Galaxy Data\kaggle-galaxies-master\kaggle-galaxies-master\try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py", line 133, in <module>
for data, length in create_valid_gen():
File "load_data.py", line 572, in buffered_gen_mp
process.start()
`File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 258, in init
cmd = get_command_line() + [rhandle]
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 358, in get_command_line`
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
As I understand I have to insert a line
if __name__ == '__main__':
Some where to get it to work
Can anyone tell me in which File I should insert it ? I have included the affected files list in the initial error logs
The affected file :
https://github.com/benanne/kaggle-galaxies/blob/master/try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py
Lines 131-134
and
https://github.com/benanne/kaggle-galaxies/blob/master/load_data.py
line 572
Python documentation is quite clear in this case.
The important part is Safe importing of main module.
Your try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py script is doing lots of things on module level. Without reading it in details I can already say that you should wrap the workflow with a function and use it like this:
if __name__ == '__main__':
freeze_support() # Optional under circumstances described in docs
your_workflow_function()
Besides the problem you have, it's a good habit not to surprise possible user of your script with side effects, if the user just wants to import it and reuse some of it's functionality.
So don't put your code on module level. It's ok to have constants on module level but the workflow should be in functions and classes.
If Python module is intended to be used as a script (like in your case), you simply put the if __name__ == '__main__' in the very end of this module, calling your_workflow_function() only if the module is the entry point for the interpreter - so called main module.
I was coding something at work and it seems that some C API functions provided by python are not working. I tried mainly the function that check types, for example:
import ctypes
python33_dll = ctypes.CDLL('python33.dll')
a_float = python33_dll.PyFloat_FromDouble(ctypes.c_float(2.0))
python33_dll.PyFloat_Check(a_float)
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
python33_dll.PyFloat_Check(a_float)
File "C:\Python33\lib\ctypes\__init__.py", line 366, in __getattr__
func = self.__getitem__(name)
File "C:\Python33\lib\ctypes\__init__.py", line 371, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'PyFloat_Check' not found
Is there anything specific I need to do to use this function, or is it a bug?
docs.python.org/3.3/c-api/float.html?highlight=double#PyFloat_Check
PyFloat_Check() is a macro. You will need to expand it manually and call the correct function instead.
I'm trying to debug an application which makes use of the pynetdicom library. I'm not sure how relevant that specific detail is, however what IS relevant is that it makes heavy use of multithreading to run background socket listener tasks without blocking the main thread. The storescp.py example can be used to reproduce this.
Whenever I place a breakpoint that gets encountered (regardless of what thread, main or child, it gets encountered in), I get the following traceback:
Traceback (most recent call last):
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 1397, in <module>
debugger.run(setup['file'], None, None)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 1090, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/Users/alexw/Development/Python/kreport2/KReport2/dicomdatascraper.py", line 183, in <module>
oldDicomList = copy.copy(newData)
File "/Users/alexw/Development/Python/kreport2/KReport2/dicomdatascraper.py", line 183, in <module>
oldDicomList = copy.copy(newData)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_frame.py", line 135, in trace_dispatch
self.doWaitSuspend(thread, frame, event, arg)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_frame.py", line 25, in doWaitSuspend
self._args[0].doWaitSuspend(*args, **kwargs)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 832, in doWaitSuspend
self.processInternalCommands()
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd.py", line 360, in processInternalCommands
thread_id = GetThreadId(t)
File "/Applications/Aptana Studio 3/plugins/org.python.pydev_2.7.0.2013012902/pysrc/pydevd_constants.py", line 140, in GetThreadId
return thread.__pydevd_id__
File "/Users/alexw/.virtualenvs/kreport2dev/devlibs/pynetdicom/source/netdicom/applicationentity.py", line 73, in __getattr__
obj = eval(attr)()
File "<string>", line 1, in <module>
NameError: name '__pydevd_id__' is not defined
My thought is that, perhaps, in order to make things work, PyDev monkey-patches a __pydevd_id__ into spawned threads, however fails to patch those into these threads because they are, in fact, subclasses and not direct instances of threading.Thread (in this case, the worker is an instance of class Association(threading.Thread):).
Of course, I don't know PyDev well enough to confirm this theory, or else fix it. And it seems neither does the internet.
Is subclassing Thread so rarely used a pattern that it's simply not considered in the PyDev architecture? Without re-architecting the library, how could this issue be remedied?
I simply needed to look harder at that traceback.
The pynetdicom library, in its subclassing of threading.Thread, overrode __getattr__ and somewhat broke it. The problem was:
def __getattr__(self, attr):
#while not self.AssociationEstablished:
# time.sleep(0.001)
obj = eval(attr)
# do some stuff
return obj
when a nonexistent attribute is passed, a NameError is raised. This isn't caught by pydev's monkeypatching routine (if thread.__pydevd_id__ raises AttributeError, thread.__pydevd_id__ = stuff)
The solution was to update that section thusly:
def __getattr__(self, attr):
#while not self.AssociationEstablished:
# time.sleep(0.001)
try:
obj = eval(attr)
except NameError:
raise AttributeError
# do some stuff
return obj
This intercepts the NameError and raises an AttributeError instead, as __getattr__ should if the queried attribute doesn't exist.
I have a program that uses the win32com library to control iTunes, but have been having some issues getting it to compile into an executable. The problem seems to revolve around using DispatchWithEvents instead of Dispatch. I've created a very simple program to illustrate my problem:
import win32com.client
win32com.client.gencache.is_readonly = False #From py2exe wiki
class ITunesEvents(object):
def __init__(self): self.comEnabled = True
def OnCOMCallsDisabledEvent(self, reason): self.comEnabled = False
def OnCOMCallsEnabledEvent(self): self.comEnabled = True
# The first line works in the exe, the second doesn't.
itunes = win32com.client.Dispatch("iTunes.Application")
#itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
lib = getattr(itunes, "LibraryPlaylist")
src = getattr(lib, "Source")
playlists = getattr(src, "Playlists")
print "Found %i playlists." % getattr(playlists, "Count")
Using Dispatch, the program compiles and runs correctly. Using DispatchWithEvents, the program runs fine when called from the command line, but produces the following error when running the exe:
Traceback (most recent call last):
File "sandbox.py", line 16, in <module>
itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
File "win32com\client\__init__.pyc", line 252, in DispatchWithEvents
File "win32com\client\gencache.pyc", line 520, in EnsureModule
File "win32com\client\gencache.pyc", line 287, in MakeModuleForTypelib
File "win32com\client\makepy.pyc", line 259, in GenerateFromTypeLibSpec
File "win32com\client\gencache.pyc", line 141, in GetGeneratePath
IOError: [Errno 2] No such file or directory: '[distDir]\\library.zip\\win32com\\gen_py\\__init__.py'
I've also tried using PyInstaller, which gives a similar error:
File "<string>", line 16, in <module>
File "[outDir]/win32com.client", line 252, in DispatchWithEvents
File "[outDir]/win32com.client.gencache", line 520, in EnsureModule
File "[outDir]/win32com.client.gencache", line 287, in MakeModuleForTypelib
File "[outDir]/win32com.client.makepy", line 286, in GenerateFromTypeLibSpec
File "[outDir]/win32com.client.gencache", line 550, in AddModuleToCache
File "[outDir]/win32com.client.gencache", line 629, in _GetModule
File "[pyinstallerDir]\iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named win32com.gen_py.9E93C96F-CF0D-43F6-8BA8-B807A3370712x0x1x13
I know I can manually add the typelib in my setup.py file, but I'd like to run the code on computers with different versions of iTunes without recompiling so I'd prefer to dynamically create it. If there's no way to do this with the setup/spec, maybe there is another way to load the events? Thanks.
Addition:
Thanks to Ryan, I found I could take the generated py file and after a little digging, was able to come up with the following.
Take the generated py file (from makepy.py) and rename it somewhere like cominterface.py. Then you'll need to do the following to actually create the COM object with event handler.
import cominterface
from types import ClassType
from win32com.client import EventsProxy, _event_setattr_
class ItunesEvents:
'''iTunes events class. See cominterface for details.'''
def OnPlayerPlayEvent(self, t):print "Playing..."
def OnPlayerStopEvent(self, t): print "Stopping..."
itunes = cominterface.iTunesApp()
rClass = ClassType("COMEventClass", (itunes.__class__, itunes.default_source, ItunesEvents), {'__setattr__': _event_setattr_})
instance = rClass(itunes._oleobj_)
itunes.default_source.__init__(instance, instance)
#ItunesEvents.__init__(instance) #Uncomment this line if your events class has __init__.
itunes = EventsProxy(instance)
Then you can go about your business.
I was experiencing the exact same error.
This link put me in the right direction -->
http://www.py2exe.org/index.cgi/UsingEnsureDispatch
however it mentions that :
NB You must ensure that the python...\win32com.client.gen_py dir does not exist
to allow creation of the cache in %temp%
Which was a bit confusing.
What solved it for me was renaming "C:\Python26\Lib\site-packages\win32com\gen_py" to "C:\Python26\Lib\site-packages\win32com\gen_pybak" (when running py2exe)
This is the official way to do it.
(Edit: copied verbatim example from that link above)
import win32com.client
if win32com.client.gencache.is_readonly == True:
#allow gencache to create the cached wrapper objects
win32com.client.gencache.is_readonly = False
# under p2exe the call in gencache to __init__() does not happen
# so we use Rebuild() to force the creation of the gen_py folder
win32com.client.gencache.Rebuild()
# NB You must ensure that the python...\win32com.client.gen_py dir does not exist
# to allow creation of the cache in %temp%
# Use SAPI speech through IDispatch
from win32com.client.gencache import EnsureDispatch
from win32com.client import constants
voice = EnsureDispatch("Sapi.SpVoice", bForDemand=0)
voice.Speak( "Hello World.", constants.SVSFlagsAsync )
Instead of depending on the cache, I'd recommend going into the local cache directory, copying the generated file into your local project file, and naming it something like ITunesInterface.py, and calling to that explicitly. This will make py2exe pull it into your compiled app.