Using a dynamically loaded function as a target with multiprocessing.Pool - python

I am working on a GUI that needs to do some heavy computation in the background and then update the GUI when the calculation is complete. The multiprocessing module seems to be a good solution since I can use the *apply_async* method to specify the target and callback function. The callback function is used to update the GUI with the result. However I am having difficulty when trying to combine multiprocessing with a dynamically loaded module as in the following code. The error message is ImportError: No module named calc.
Is the error due to the fact that multiprocessing just doesn't work with dynamically loaded modules? If not, are there any ideas on a better approach?
from PySide.QtCore import *
from PySide.QtGui import *
import multiprocessing
import time
import sys
import os
import logging
import imp
PluginFolder = "plugins"
plugins = {}
def f(x):
y = x*x
time.sleep(2) #Simulate processing time.
return y
def load_plugin(name):
'''Load the python module 'name'
'''
location = os.path.join('.', PluginFolder)
info = imp.find_module(name, [location])
plugin = {"name": name, "info": info}
plugins[name] = imp.load_module(name, *plugin["info"])
class MainWindow(QMainWindow):
def __init__(self):
super(MainWindow, self).__init__()
self.pool = multiprocessing.Pool()
load_plugin('calc') #load ./plugins/calc.py
button1 = QPushButton('Calculate', self)
button1.clicked.connect(self.calculate)
button2 = QPushButton('Test', self)
button2.clicked.connect(self.run_test)
self.text = QTextEdit()
vbox1 = QVBoxLayout()
vbox1.addWidget(button1)
vbox1.addWidget(button2)
vbox1.addWidget(self.text)
myframe = QFrame()
myframe.setLayout(vbox1)
self.setCentralWidget(myframe)
self.show()
self.raise_()
def calculate(self):
#self.pool.apply_async(f, [10], callback=self.update_gui) #This works
#result = plugins['calc'].f(10) #this works
#self.update_gui(result)
self.pool.apply_async(plugins['calc'].f, [10], callback=self.update_gui) #This doesn't
def update_gui(self, result):
self.text.append('Calculation complete. Result = %d\n' % result)
def run_test(self):
self.text.append('Testing\n')
if __name__ == '__main__':
app = QApplication(sys.argv)
gui = MainWindow()
app.exec_()
In ./plugins/calc.py, the function f is defined as in the above code.

This doesn't work since you're loading your calc module as a top-level module. Since no module calc is present in your sys.path or in current directory, it can't be found by import statement. Replacing import statement with following will do the trick:
plugins[name] = imp.load_module('{}.{}'.format(PluginFolder, name),
*plugin["info"])
For a plugin.calc being importable, plugins has to be a python module, i.e. contain a __init__.py file.
Any import <module> statement in your plugins files, such as in plugins/calc.py, will lead to a warning,
RuntimeWarning: Parent module 'plugins' not found while handling absolute import import <module>
The reason is that import process looks if parent module contains <module>, and while inside calc.py, can't find parent plugins module. You can rid of the error explicetely specifying plugins module location with, for example import plugins statement in main code.

Related

How can I debug a single class?

I'm writing a code on PyCharm, but it is easier for me if I could debugg continually while I'm writing my code. I can't figure out, that how do I get this class debugged? :
from PyQt5.Qt import QGraphicsPixmapItem
from PyQt5.QtGui import QPixmap
class Level():
def __init__(self, scene, level):
self.scene = scene
self.level = open(level)
def add_item_to_scene(self):
for x in range(self.level):
for y in range(self.level[0]):
if y == 'X':
brick = QPixmap('brick.png')
self.scene.addItem(brick)
I have just begun and this is all what I've written. I'm doing a Platformer game with PyQt5, and I'm trying to set the levels now. Thank you for any help.
Use a convention to run code during development and not run the code on import as shown in the following code.
from PyQt5.Qt import QGraphicsPixmapItem
from PyQt5.QtGui import QPixmap
class Level():
def __init__(self, scene, level):
self.scene = scene
self.level = open(level)
def add_item_to_scene(self):
for x in range(self.level):
for y in range(self.level[0]):
if y == 'X':
brick = QPixmap('brick.png')
self.scene.addItem(brick)
# Use the following convention.
# In python the initial module run receives the name __main__.
if __name__ == '__main__':
print(f'Hello my name is {__name__=} because my module was started by '
f'pyhton. ')
# So while developing whenever you run this module commands here will run.
# Later on when you import the module the commands will not run.
# because __main__ will be the name of this module
l = Level('brick', 5)
l.add_item_to_scene()
else:
# Place any commands you want to run on import here.
print(f'Hello my name is {__name__=} because my module was imported by '
f'another module. ')
Demo Results:
If run directly from python
Hello my name is __name__='__main__' because my module was started by pyhton.
Traceback (most recent call last):
File "C:\Users\ctynd\OneDrive\CodeBase\StackOverflowActivity\OldScratches\scratch_2.py", line 27, in <module>
l = Level('brick', 5)
File "C:\Users\ctynd\OneDrive\CodeBase\StackOverflowActivity\OldScratches\scratch_2.py", line 9, in __init__
self.level = open(level)
OSError: [WinError 6] The handle is invalid
If imported:
import scratch_2
Results
Hello my name is __name__='scratch_2' because my module was imported by another module.

Load py files and do something with the class in a for loop from directory without knowing how many files exist?

I am attempting to use a Directory to hold my growing list of frames and when my MAIN.py file is executed I want the code to load each frame into a Tkinter notebook widget.
I can do this manually by importing each file from the sub directory in my MAIN folder however I am attempting to have this load them pragmatically without having to know each file name and not having to import them myself.
I have made some progress but I got stuck at actually loading the class in the files. I can get a list of all files and I believe I am importing them just fine I just cannot get them to load as it keeps telling me that a module does not exist.
I must be misunderstanding something here as I cannot figure out how to properly call a class from the imported file.
Error:
C:\Users\USER\PycharmProjects\TEST\venv\Scripts\python.exe C:/Users/USER/PycharmProjects/TEST/MAIN/MAIN.py
# This below list is a result of my print statement to see if I got all the file names.
['TaskFrame', 'TestFrame1', 'TestFrame2', 'TestFrame3']
Traceback (most recent call last):
File "C:/Users/MCDOMI3/PycharmProjects/TEST/MAIN/MAIN.py", line 46, in <module>
ScriptManager().mainloop()
File "C:/Users/MCDOMI3/PycharmProjects/TEST/MAIN/MAIN.py", line 39, in __init__
self.add_frame_to_book(foo.tab_title)
AttributeError: module 'ScriptFrames.TaskFrame' has no attribute 'tab_title'
MAIN Code:
import tkinter as tk
import tkinter.ttk as ttk
from os.path import dirname, basename, isfile, join
import glob
import importlib.util
modules = glob.glob(join(dirname('.\ScriptFrames\\'), "*.py"))
__all__ = [basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
print(__all__)
class ScriptManager(tk.Tk):
def __init__(self):
super().__init__()
self.book = ttk.Notebook(self)
self.book.grid(row=0, column=0, sticky='nsew')
for fn in __all__:
spec = importlib.util.spec_from_file_location('ScriptFrames.{}'.format(fn),
'.\ScriptFrames\\{}.py'.format(fn))
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
# Here is my problem I do not know if I am attempting to get the class
# attribute correctly here. I am sure that I am calling this wrong.
self.add_frame_to_book(foo.tab_title)
def add_frame_to_book(self, fname):
self.book.add(fname, text='test')
if __name__ == '__main__':
ScriptManager().mainloop()
Each test file is a simple tkinter frame.
Test Frame Code:
import tkinter as tk
import tkinter.ttk as ttk
class TabFrame(ttk.Frame):
def __init__(self):
super().__init__()
self.tab_title = 'Test Frame 1'
self.columnconfigure(0, weight=1)
self.rowconfigure(0, weight=1)
tk.Label(self, text='test').grid(row=0, column=0, sticky='nsew')
If it helps here is my file structure:
You're trying to access tab_title for the module, and not from the (Frame) class inside it.
Once the module is loaded, get its attributes using [Python 3.Docs]: Built-in Functions:
Get module attribute names (using dir)
For each name, get the corresponding attribute (using getattr)
Test whether the attribute is a ttk.Frame (as the module has other attributes as well)
If it is a frame class, instantiate it in order to access tab_title, otherwise just discard it
Example:
# ...
# The rest of your code
spec.loader.exec_module(foo)
# At this point, 'foo' module is loaded. Get its attributes
module_attrs = [getattr(foo, attr_name, None) for attr_name in dir(foo)]
# Filter out the attributes only keeping the ones that we care about
frame_classes = [attr for attr in module_attrs if isinstance(attr, (type,)) and issubclass(attr, (ttk.Frame,))]
# Then handle the module classes (it will probably be only one)
for frame_class in frame_classes:
if frame_class.__name__.startswith("Tab"): # Or any other filtering
# It would have been much more efficient to filter names when computing attr_names, but it's more consistent to be here (especially if other filtering beside name is needed)
frame_class_instance = frame_class() # Instantiate the class, as tab_title is an instance attribute
print(frame_class.__name__, frame_class_instance.tab_title)
Here is what I do.
def load(self, file=None):
file_type = path.splitext(file)[1].lstrip('.').lower()
if file_type == 'py' and path.exists(file):
spec = spec_from_file_location("module.name", file)
module = module_from_spec(spec)
spec.loader.exec_module(module)
self.nodes = module.config

How can I resolve the 'TypeError: 'list' object is not callable' error in this threaded pyside code that seems to work fine in a similar case?

The full traceback for my error is:
> python zthreadtest_tjedit.py
None
Traceback (most recent call last):
File "zthreadtest_tjedit.py", line 17, in run self.function()
TypeError: 'list' object is not callable
I apologize that's this code is a bit system specific and probably won't serve as an executable example for most people. Hopefully the solution is simple enough for someone to see right off. If you're not running zfs with a currently imported zpool, but are on a *nix platform with the weir.zfs module installed, it will return an empty list with the non-threaded code enabled (see comments in code for toggling threading). With the threading module enabled, it throws the error as shown above.
Confusing to me is that the second snip of code from Jo Plaete (https://joplaete.wordpress.com/2010/07/21/threading-with-pyqt4/) runs without error, and I simply modified this code to my needs.
EDIT: This error-causing difference may be related to the list object used in my code and not hers, but I still need to make mine work.
My question is: how do I resolve my error such that my threaded module runs correctly?
This seems simple, but I'm absolutely stumped. So much so that this is the first question I've posted on any help forum ever! I hope I've asked my question properly and I appreciate any help.
My Code, from much larger pyside gui program:
import PySide, sys
from PySide import QtCore, QtGui
from PySide.QtCore import *
from PySide.QtGui import *
import re, subprocess, threading
from weir import zfs
class WorkerThread(QThread):
def __init__(self, function):
QThread.__init__(self)
self.function = function
def __del__(self):
self.wait()
def run(self):
self.function()
return
class MainZ(QMainWindow):
def __init__(self):
super(MainZ, self).__init__()
# print(self)
# imported_pools = self.get_imported() # No threading
imported_pools = self.thread_test() # Use threading module
print(imported_pools)
def thread_test(self):
self.threader = WorkerThread(self.get_imported())
self.threader.start()
def get_imported(self):
pool_string = subprocess.getoutput(
'zpool list |grep -v ^NAME.*SIZE.*ALLOC |grep -o ^[a-Z0-9]*')
imported_pools = re.split(r'\s *', pool_string)
return imported_pools
app = QApplication(sys.argv)
form = MainZ()
app.exec_()
Code I modeled from Jo Plaete that works for me without error:
import sys, time
from PySide import QtCore, QtGui
class GenericThread(QtCore.QThread):
def __init__(self, function, *args, **kwargs):
QtCore.QThread.__init__(self)
self.function = function
self.args = args
self.kwargs = kwargs
def __del__(self):
self.wait()
def run(self):
self.function(*self.args,**self.kwargs)
return
class MyApp(QtGui.QWidget):
def __init__(self, parent=None):
QtGui.QWidget.__init__(self, parent)
self.setGeometry(300, 300, 280, 600)
self.setWindowTitle('threads')
self.layout = QtGui.QVBoxLayout(self)
self.testButton = QtGui.QPushButton("test")
self.connect(self.testButton, QtCore.SIGNAL("released()"), self.test)
self.listwidget = QtGui.QListWidget(self)
self.layout.addWidget(self.testButton)
self.layout.addWidget(self.listwidget)
def add(self, text):
""" Add item to list widget """
print("Add: " + text)
self.listwidget.addItem(text)
self.listwidget.sortItems()
def addBatch(self,text="test",iters= 5,delay=0.2):
""" Add several items to list widget """
for i in range(iters):
time.sleep(delay) # artificial time delay
self.add(text+" "+str(i))
def test(self):
self.listwidget.clear()
self.genericThread = GenericThread(
self.addBatch,"from generic thread ",delay=0.3)
self.genericThread.start()
# run
app = QtGui.QApplication(sys.argv)
test = MyApp()
test.show()
app.exec_()
The
self.threader = WorkerThread(self.get_imported())
should read
self.threader = WorkerThread(self.get_imported)
When creating the thread, you want to pass the function itself and not the result of calling the function (which is a list).

Python passing handle to interface to multiple modules

I have a python program with multiple modules. There is a module that contains code to initialize a UDP Interface to send messages. I would like this Interface to be used across all the modules. The only way I've thought of is passing the object to the other modules through a predefined function.
Main Module:
from mod1 import *
from mod2 import *
from Interfaces import *
class MainClass():
def __init__(self):
# Initialize Modules
self.m1 = Module1()
self.m2 = Module2()
self.intf = UdpInterface()
self._init_module_interfaces()
def _init_module_interfaces(self):
self.m1.InitIntf(self.intf)
self.m2.InitIntf(self.intf)
if __name__ == '__main__':
main = MainClass()
Submodule:
class Module1():
def __init__(self):
...
def InitIntf(self, intf):
self.intf = intf
Is there a better way to do this?

How can I create a status bar item with Cocoa and Python (PyObjC)?

I have created a brand new project in XCode and have the following in my AppDelegate.py file:
from Foundation import *
from AppKit import *
class MyApplicationAppDelegate(NSObject):
def applicationDidFinishLaunching_(self, sender):
NSLog("Application did finish launching.")
statusItem = NSStatusBar.systemStatusBar().statusItemWithLength_(NSVariableStatusItemLength)
statusItem.setTitle_(u"12%")
statusItem.setHighlightMode_(TRUE)
statusItem.setEnabled_(TRUE)
However, when I launch the application no status bar item shows up. All the other code in main.py and main.m is default.
The above usage of .retain() is required because the statusItem is being destroyed upon return from the applicationDidFinishLaunching() method. Bind that variable as a field in instances of MyApplicationAppDelegate using self.statusItem instead.
Here is a modified example that does not require a .xib / etc...
from Foundation import *
from AppKit import *
from PyObjCTools import AppHelper
start_time = NSDate.date()
class MyApplicationAppDelegate(NSObject):
state = 'idle'
def applicationDidFinishLaunching_(self, sender):
NSLog("Application did finish launching.")
self.statusItem = NSStatusBar.systemStatusBar().statusItemWithLength_(NSVariableStatusItemLength)
self.statusItem.setTitle_(u"Hello World")
self.statusItem.setHighlightMode_(TRUE)
self.statusItem.setEnabled_(TRUE)
# Get the timer going
self.timer = NSTimer.alloc().initWithFireDate_interval_target_selector_userInfo_repeats_(start_time, 5.0, self, 'tick:', None, True)
NSRunLoop.currentRunLoop().addTimer_forMode_(self.timer, NSDefaultRunLoopMode)
self.timer.fire()
def sync_(self, notification):
print "sync"
def tick_(self, notification):
print self.state
if __name__ == "__main__":
app = NSApplication.sharedApplication()
delegate = MyApplicationAppDelegate.alloc().init()
app.setDelegate_(delegate)
AppHelper.runEventLoop()
I had to do this to make it work:
Open MainMenu.xib. Make sure the class of the app delegate is MyApplicationAppDelegate. I'm not sure if you will have to do this, but I did. It was wrong and so the app delegate never got called in the first place.
Add statusItem.retain() because it gets autoreleased right away.

Categories

Resources