In a.py)
def pool_init():
global aa
aa = "I'm in a.py"
def pool_func(chunk, job_func):
return job_func(chunk)
def run_pool(mp_iterable, job_func, pool_func = pool_func):
rst = []
partial(pool_func, job_func = job_func)
with multiprocessing.Pool(4, initializer=pool_init,) as p:
for ir in p.imap_unordered(pool_func, mp_iterable):
rst.append(ir)
In main.py)
def job_func():
print(aa)
a.run_pool(range(5), job_func = job_func)
raised (in Cent OS, jupyter)
NameError: name 'aa' is not defined
Questions)
Why did this error occured?
How could i fix this error?
Edit)
Recently, I need multiprocessing module more often, so I wanted to make some functions to run multiprocessing.Pool in my custom module not to write redundant codes.
I want to run multiprocessing.Pool just like:
run_pool(iterable, job_func)
# Acutal processes in job_func, the both arguments are defined in main.py
Related
I have a working code, but I would like to know what is the proper Pythonic approach.
My goal: have a directory of "plugins" (one module per plugin), which is dynamically loaded when the program runs. All of the modules will have a function defined, which will act as an "entrypoint".
The aim is to have a script which is easily extended by some extra functionality.
What I have come up with is the following. reporter = plugin in this case.
import os
import importlib
import reporters # Package, where plugins (reporters) will reside
def find_reporters():
# Find all modules in directory "reporters" which look like "*_reporter.py"
reporters = [rep.rsplit('.py', 1)[0] for rep in os.listdir('reporters') if rep.endswith('_reporter.py')]
functions = []
for reporter in reporters:
module = importlib.import_module('.' + reporter, 'reporters')
try:
func = getattr(module, 'entry_function') # Read the entry_function if present
functions.append(func) # Add the function to the list to be returned
except AttributeError as e:
print(e)
return functions
def main():
funcs = find_reporters()
for func in funcs:
func() # Execute all collected functions
I am not too seasoned in Python, so is this an acceptable solution?
Need a help with the next situation. I want to implement debug mode in my script through printing small completion report in functions with command executed name and ellapsed time like:
def cmd_exec(cmd):
if isDebug:
commandStart = datetime.datetime.now()
print commandStart
print cmd
...
... exucuting commands
...
if isDebug:
print datetime.datetime.now() - command_start
return
def main():
...
if args.debug:
isDebug = True
...
cmd_exec(cmd1)
...
cmd_exec(cmd2)
...
How can isDebug variable be simply passed to functions?
Should I use "global isDebug"?
Because
...
cmd_exec(cmd1, isDebug)
...
cmd_exec(cmd2, isDebug)
...
looks pretty bad. Please help me find more elegant way.
isDebug is state that applies to the application of a function cmd_exec. Sounds like a use-case for a class to me.
class CommandExecutor(object):
def __init__(self, debug):
self.debug = debug
def execute(self, cmd):
if self.debug:
commandStart = datetime.datetime.now()
print commandStart
print cmd
...
... executing commands
...
if self.debug:
print datetime.datetime.now() - command_start
def main(args):
ce = CommandExecutor(args.debug)
ce.execute(cmd1)
ce.execute(cmd2)
Python has a built-in __debug__ variable that could be useful.
if __debug__:
print 'information...'
When you run your program as python test.py, __debug__ is True. If you run it as python -O test.py, it will be False.
Another option which I do in my projects is set a global DEBUG var at the beginning of the file, after importing:
DEBUG = True
You can then reference this DEBUG var in the scope of the function.
You can use a module to create variables that are shared. This is better than a global because it only affects code that is specifically looking for the variable, it doesn't pollute the global namespace. It also lets you define something without your main module needing to know about it.
This works because modules are shared objects in Python. Every import gets back a reference to the same object, and modifications to the contents of that module get shared immediately, just like a global would.
my_debug.py:
isDebug = false
main.py:
import my_debug
def cmd_exec(cmd):
if my_debug.isDebug:
# ...
def main():
# ...
if args.debug:
my_debug.isDebug = True
Specifically for this, I would use partials/currying, basically pre-filling a variable.
import sys
from functools import partial
import datetime
def _cmd_exec(cmd, isDebug=False):
if isDebug:
command_start = datetime.datetime.now()
print command_start
print cmd
else:
print 'isDebug is false' + cmd
if isDebug:
print datetime.datetime.now() - command_start
return
#default, keeping it as is...
cmd_exec = _cmd_exec
#switch to debug
def debug_on():
global cmd_exec
#pre-apply the isDebug optional param
cmd_exec = partial(_cmd_exec, isDebug=True)
def main():
if "-d" in sys.argv:
debug_on()
cmd_exec("cmd1")
cmd_exec("cmd2")
main()
In this case, I check for -d on the command line to turn on debug mode and I do pre-populate isDebug on the function call by creating a new function with isDebug = True.
I think even other modules will see this modified cmd_exec, because I replaced the function at the module level.
output:
jluc#explore$ py test_so64.py
isDebug is falsecmd1
isDebug is falsecmd2
jluc#explore$ py test_so64.py -d
2016-10-13 17:00:33.523016
cmd1
0:00:00.000682
2016-10-13 17:00:33.523715
cmd2
0:00:00.000009
Currently I am working on jython script that can execute an external jython script. The executed (external) jython script must run in the same weblogic session where my script runs in order to be able to cancel the session (in case of error) or activate it. The weblogic connection is being created in my script and the called script has to use the created (with ‘startEdit()’) session.
I found a hybrid solution but perhaps it can be done better.
The executed script in the working solution:
import wlst_operations as wl
print 'Start'
wl.cd('/')
print wl.cmo
wl.cd('/Servers/AdminServer')
print wl.cmo
wl.cd('/JDBCSystemResources/pasDataSource/JDBCResource/pasDataSource/JDBCConnectionPoolPara ms/pasDataSource')
print wl.cmo
wl.cmo.setInitialCapacity(6)
The wlst_operations jython was taken from http://www.javamonamour.org/2013/08/wlst-nameerror-cd.html.
As you can see an object like reference (‘wl.’) must be put in front of each WLST command…
The output is fine:
[MBeanServerInvocationHandler]com.bea:Name=gintdev1,Type=Domain
[MBeanServerInvocationHandler]com.bea:Name=AdminServer,Type=Server
[MBeanServerInvocationHandler]com.bea:Name=pasDataSource,Type=weblogic.j2ee.descriptor.wl.JDBCConnectionPoolParamsBean,Parent=[gintdev1]/JDBCSystemResources[pasDataSource],Path=JDBCResource[pasDataSource]/JDBCConnectionPoolParams
When I don’t use the object reference:
from wlstModule import *
print 'user defined'
cd('/')
print cmo
cd('/Servers/AdminServer')
print cmo
cd('/JDBCSystemResources/pasDataSource/JDBCResource/pasDataSource/JDBCConnectionPoolParams/pasDataSource')
print cmo
cmo.setInitialCapacity(6)
Then the output is:
[MBeanServerInvocationHandler]com.bea:Name=gintdev1,Type=Domain
[MBeanServerInvocationHandler]com.bea:Name=gintdev1,Type=Domain
[MBeanServerInvocationHandler]com.bea:Name=gintdev1,Type=Domain
Problem invoking WLST - Traceback (innermost last):
File "/tmp/lv30083/./orchestrator.py", line 83, in ?
File "/tmp/lv30083/./orchestrator.py", line 66, in main
File "/tmp/lv30083/./utils/orch_wl.py", line 55, in execute
File "user_defined_script_incorrect.py", line 11, in ?
AttributeError: setInitialCapacity
i.e. the cd commands are executed (not getting error) but it just doesn’t jump to the datasource…
My script is
import orch_logging
import sys
from wlstModule import *
class WeblogicManager(object):
def connect_to_server(self, p_ssl, p_domainName, p_userConfigFile, p_userKeyFile):
logger = orch_logging.Logger()
logger.info('Trying to connect to the node manager. domainName='+p_domainName+',userConfigFile='+p_userConfigFile+',ssl='+p_ssl+',p_userKeyFile='+p_userKeyFile)
try:
connect(domainName=p_domainName,userConfigFile=p_userConfigFile,userKeyFile=p_userKeyFile)
return True
except:
logger.error("Error while trying to connect to node manager!")
return False
def startEdit(self):
edit()
startEdit()
def activate(self):
activate()
def undo(self):
cancelEdit('y')
def disconnect(self):
disconnect()
def execute(self, path):
execfile(path)
Is there any way to use the WLST commands without using the 'wl.' reference in front of them?
Thanks,
V.
I had to modify my script. All the operations are now in one context (in the context of my script)
import sys
from wlstModule import *
from weblogic.management.scripting.utils import WLSTUtil
import sys
# needed to execute normal (without object reference) WLST commands in child scripts
origPrompt = sys.ps1
theInterpreter = WLSTUtil.ensureInterpreter();
WLSTUtil.ensureWLCtx(theInterpreter)
execfile(WLSTUtil.getWLSTScriptPath())
execfile(WLSTUtil.getOfflineWLSTScriptPath())
exec(WLSTUtil.getOfflineWLSTScriptForModule())
execfile(WLSTUtil.getWLSTCommonModulePath())
theInterpreter = None
sys.ps1 = origPrompt
modules = WLSTUtil.getWLSTModules()
for mods in modules:
execfile(mods.getAbsolutePath())
wlstPrompt = "false"
class WeblogicManager(object):
...
def execute(self, path):
execfile(path)
I've got a function here for running an external python script in another process. m is the Multiprocessing module
def run(app,WindowOffX,WindowOffY,WindowWidth,WindowHeight):
try:
exec("import Programs."+app+".main as Foo")
Foo.init()
p = m.Process(target=Foo.main(WindowOffX,WindowOffY,WindowWidth,WindowHeight))
except ImportError:
print("That app doesn't exist!!! O.O")
But this generates NameError: global name 'Foo' is not defined. Can someone help?
Ha, the problem is that python doesn't know your exec statement defines Foo, so it tries to look it up as a global. To clue it in, try this:
try:
Foo = None
exec("import Programs."+app+".main as Foo")
Foo.init()
Incidentally, here's how you can do what you're after without using exec:
Foo = __import__("Programs."+app+".main")
Is there a pdb equivalent to disp in gdb?
E.g. when I'm debugging C using gdb I can have variables printed on every 'step' through the code by typing:
disp var
When I'm debugging python using pdb I would like similar functionality, but disp does not seem to be there, the python pdb documentation does not seem to offer an alternative - but it seems like an odd omission?
The code bellow uses Python introspection features to add two new commands to the PDB module 0
just put the given function, and its call in a separate module, and import this module before starting debugging - you should have the 'disp' and 'undisp' commands do add and retract watches to variables.
It works by monkeypatching Python's pdb module, which is written in pure python.
# -*- coding: utf-8 -*-
def patch_pdb():
import pdb
def wrap(func):
def new_postcmd(self, *args, **kw):
result = func(self, *args, **kw)
if hasattr(self, "curframe") and self.curframe and hasattr(self, "watch_list"):
for arg in self.watch_list:
try:
print >> self.stdout, "%s: %s"% (arg, self._getval(arg)) + ", ",
except:
pass
self.stdout.write("\n")
return result #func(self, *args, **kw)
return new_postcmd
pdb.Pdb.postcmd = wrap(pdb.Pdb.postcmd)
def do_disp(self, arg):
if not hasattr(self, "watch_list"):
self.watch_list = []
self.watch_list.append(arg)
pdb.Pdb.do_disp = do_disp
def do_undisp(self, arg):
if hasattr(self, "watch_list"):
try:
self.watch_list.remove(arg)
except:
pass
pdb.Pdb.do_undisp = do_undisp
patch_pdb()
if __name__ == "__main__":
# for testing
import pdb; pdb.set_trace()
a = 0
for i in range(10):
print i
a += 2
Unfortunately I could only make it display the state of the variables as they where previously to the execution of the last command. (I tried a little bit, but monkeypatching the bdb module, which is the base for the Pdb did not seem to work as well). You can try and change the methods in either pdb.Pdb, bdb.Bdb or cmd.Cmd that are decorated by wrap to find one that is called after the debugged frame state has changed.
You can set up some aliases that will do this for you:
alias n next;; p var
alias s step;; p var
Printing a whole list of variable names is left as an exercise to the reader. Unfortunately doing it this way means that when you send the debugger an empty line, the "last command" it executes is p var rather than, for example, n. If you want to fix that, then you can use this somewhat hacky set of Pdb commands instead:
!global __stack; from inspect import stack as __stack
!global __Pdb; from pdb import Pdb as __Pdb
!global __pdb; __pdb = [__framerec[0].f_locals.get("pdb") or __framerec[0].f_locals.get("self") for __framerec in __stack() if (__framerec[0].f_locals.get("pdb") or __framerec[0].f_locals.get("self")).__class__ == __Pdb][-1]
alias s step;; p var;; !__pdb.lastcmd = "!__pdb.cmdqueue.append('s')"
alias n next;; p var;; !__pdb.lastcmd = "!__pdb.cmdqueue.append('n')"
During the pdb debugging you can type normal python code, beyond the one letter commands - so just using print var should work for you.