I am seeking to better understand the following behavior when using dask.delayed to call a function that depends on parameters. The issue seems to arise when parameters are specified in a parameters file read by configparser. Here is a complete example:
parameter file:
#zpar.ini: parameter file for configparser
[my pars]
my_zpar = 2.
parser:
#zippy_parser
import configparser
def read(_rundir):
global rundir
rundir = _rundir
cp = configparser.ConfigParser()
cp.read(rundir + '/zpar.ini')
#[my pars]
global my_zpar
my_zpar = cp['my pars'].getfloat('my_zpar')
and the main python file:
# dask test with configparser
import dask
from dask.distributed import Client
import zippy_parser as zpar
def my_func(x, y):
# print stuff
print("parameter from main is: {}".format(main_par))
print("parameter from configparser is: {}".format(zpar.my_zpar))
# do stuff
return x + y
if __name__ == '__main__':
client = Client(n_workers = 4)
#read parameters from input file
rundir = '/path/to/parameter/file'
zpar.read(rundir)
#test zpar
print("zpar is {}".format(zpar.my_zpar))
#define parameter and call my_func
main_par = 5.
z = dask.delayed(my_func)(1., 2.)
z.compute()
client.close()
The first print statement in my_func() executes just fine, but the second print statement raises an exception. The output is:
zpar is 2.0
parameter from main is: 5.0
distributed.worker - WARNING - Compute Failed
Function: my_func
args: (1.0, 2.0)
kwargs: {}
Exception: AttributeError("module 'zippy_parser' has no attribute 'my_zpar'",)
I am new to dask. I suppose this has something to do with the serialization, which I do not understand. Can someone enlighten me and/or point to relevant documentation? Thanks!
I will try to keep this brief.
When a function is serialised in order to be sent to workers, python also sends local variables and functions needed by the function (its "closure"). However, it stores the modules it references by name, it does not try to serialise your whole runtime.
This means that zippy_parser is imported in the worker, not deserialised. Since the function read has never been called
in the worker, the global variable is never initialised.
So, you could call read in the workers as part of your function or otherwise, but probably the pattern or setting module-global variables from with a function isn't great. Dask's delayed mechanism prefers functional purity, that the result you get should not depend on the current state of the runtime.
(note that if you had created the client after calling read in the main script, the workers might have got the in-memory version, depending on how subprocesses are configured to be created on your system)
I encourage you to pass in all parameters to your dask delayed functions explicitly, rather than relying on the global namespace.
Right now I'm trying to call a CAPL function in CANoe with the CANoe COM API using the python comtypes package.
For this I created the following small short python program:
from comtypes.client import CreateObject
c=CreateObject("CANoe.Application")
squareFunction=c.CAPL.GetFunction("square")
res=squareFunction.Call(5)
print(res==25)
This should call my short CAPL function:
int square(int x) {
return x*x;
}
Unfortunately, the program yields in c.CAPL.GetFunction("square") an exception, in case the simulation is running in CANoe.
COMError: (-2147418113, 'Critical Error', (None, None, None, 0, None))
If the simulation in CANoe is stopped, there is no error, but the call of the function yields None.
Does anyone knows, what is going on here?
First, make sure your function is defined in a CAPL block in the measurement-setup, not in the simulation-setup.
The application note CANalyzer/CANoe as a COM Server by Vector link states on page 15 that
The assignment of a CAPL function to a variable can only be done in
the OnInit event handler of the Measurement object.
I.e. your squareFunction variable has to be initialized during the OnInit event. Similar to this:
def OnInit():
self.squareFunction = c.CAPL.GetFunction("square")
c.Measurement.OnInit += CANoe._IMeasurementEvents_OnInitEventHandler(self.OnInit)
By this OnInit will be executed during measurement init and you can later execute self.squareFunction.Call(5)
I'm trying to figure out how to proceed and if it is feasible or not in general.
I working with external DLL to control my mechanical delay line.
This API has it internal procedure for message output in separate window. I have a strong desire to catch this message flow and present in my python (PyQT5) written application.
In API description there is a function:
int LS_SetProcessMessagesProc(void *pProc);
Function returns either 0 or 1, if there is no error or it is present, respectively.
According to dll description
It enables the replacement of the internal message-dispatching procedure of the LStep API.
The LStep API processes during waiting for confirmation of the LStep in the main-thread messages. If you want to switch of the Message-Dispatching or replace with your onw Code, you can use SetProcessMessagesProc for using a callback-procedure.
pProc must be a pointer to a stdcall-procedure without a parameter:
void MyProcessMessages() {...}
Example:
LS.SetProcessMessagesProc(&MyProcessMessages);
As example if we take python stdout, how I can send the message to it?
I'm going to illustrate everything on the:
[MSDN]: EnumWindows function - which enumerates all windows on the screen, and for each of them calls a callback function - check next bullet
[MSDN]: EnumWindowsProc callback function - which is used to handle every enumerated window
which is (a slightly more complicated example of) what you need: a function defined in an external .dll which needs to call another custom function (written by you in Python), via [Python]: ctypes module (on Win).
The code:
import ctypes
from ctypes import wintypes
try:
from win32gui import GetWindowText
pywin32_present = True
except ImportError:
pywin32_present = False
def enum_windows_proc(hwnd, l_param):
print("HWND: {}\n".format(hwnd))
if pywin32_present:
txt = GetWindowText(hwnd)
if txt and "MSCTFIME UI" not in txt and "Default IME" not in txt:
print(" Window text: {}\n".format(txt))
return 1
def main():
user32_dll = ctypes.windll.LoadLibrary("user32.dll")
enum_windows = user32_dll.EnumWindows
WND_ENUM_PROC_TYPE = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM)
enum_windows.argtypes = (WND_ENUM_PROC_TYPE, wintypes.LPARAM)
enum_windows.restype = wintypes.BOOL
enum_windows(WND_ENUM_PROC_TYPE(enum_windows_proc), wintypes.LPARAM(0))
if __name__ == "__main__":
main()
Notes:
imports:
wintypes is a ctypes sub-module that defines a bunch of Ms specific data (constants, structs, enums, ...)
[Python]: pywin32 is a Python wrapper over C Win functions, it's basically a more advanced (and pythonic) approach of what ctypes does. It doesn't come with Python by default, it must be manually installed; in our example, it's optional
def enum_windows_proc(hwnd, l_param)::
It's the Python form of BOOL CALLBACK EnumWindowsProc(_In_ HWND hwnd, _In_ LPARAM lParam);
Prints the hwnd (window handle) for each window (note that there will be lots of such windows, since most of them are "invisible" to the user)
If pywin32 module is installed, it will be used to extract each window's title(caption). Of course, that can also be done with ctypes but it's a little bit more complicated
The title filtering is to avoid printing useless text (for most of the windows). If you want more details, check: [SO]:
Get the title of a window of another program using the process name
The main function:
First, the .dll that contains the function (in our case user32.dll) needs to be loaded. This is done using [MSDN]: LoadLibrary function (Ux: [man]: dlopen). Also, the internal structure (needed for next line to look so simple) of the returned object (user32_dll) is initialized
The function (or better: a pointer to it) is being retrieved (enum_windows) using [MSDN]:
GetProcAddress function (Ux: [man]: dlsym)
The next 3 lines of code are used to let Python know the details about the loaded function pointer (return type and argument types). Note that (in some cases) there is a simpler way (codewise) to do all that, but for learning purposes, it's OK to go through the whole thing
Finally, call the external function, with our custom function as an argument
Since there will be lots of output (and will be mostly memory addresses), I won't paste it here.
Going to your problem, based on the function headers you pasted in the question, we can take the same approach (note that the code will not work copy/pasted OOTB):
import ctypes
from ctypes import wintypes
def my_process_messages():
# Your code here (delete the next (`pass`) line)
pass
dll_name = "your dll path (full or relative)"
dll_object = ctypes.windll.LoadLibrary(dll_name)
ls_set_process_messages_proc = dll_object.LS_SetProcessMessagesProc
PROCESS_MESSAGES_TYPE = ctypes.WINFUNCTYPE(None)
ls_set_process_messages_proc.argtypes = (PROCESS_MESSAGES_TYPE,)
ls_set_process_messages_proc.restype = ctypes.c_int
print("ls_set_process_messages_proc returned: {}\n".format(ls_set_process_messages_proc(PROCESS_MESSAGES_TYPE(my_process_messages))))
Note: The example is based on the fact that the external .dll:
Is Win style (uses stdcall calling convention). If that's not true (it uses cdecl), you need to change (for rigorousity's sake I'm going to say 2 things):
ctypes.windll to ctypes.cdll
ctypes.WINFUNCTYPE to ctypes.CFUNCTYPE
Exports C style functions (not C++ which mangles function names), which I'm almost 100% sure. But, if this is not the case then, sorry, nothing to do here. For more details on this topic, check: [SO]: Excel VBA, Can't Find DLL Entry Point from a DLL file.
Read about ctypes tutorial
Loading-dynamic-link-libraries
Accessing functions from loaded dlls
Calling functions
Linux example there, which uses the standard C library's qsort function:
Load the libc.so.6 dll.
from ctypes import *
libc = CDLL("libc.so.6")
Get a function pointer to qsort.
qsort = libc.qsort
qsort.restype = None
Create the type for the callback function
and implement the Python callback function.
CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
def py_cmp_func(a, b):
return a[0] - b[0]
cmp_func = CMPFUNC(py_cmp_func)
Define a C-Type Integer Array with values
and use qsort to sort the Array using cmp_func.
IntArray5 = c_int * 5
ia = IntArray5(5, 1, 7, 33, 99)
qsort(ia, len(ia), sizeof(c_int), cmp_func)
for i in ia:
print(i)
Output
1 5 7 33 99
I am designing a program that simulates a Turing machine, with the rules in a separate file for easy editing. Unfortunately, it throws up a NameError when it first tries to compute from the ruleTable.
The offending snippet:
import TheRules
def turIt():
global ruleTable #Global has been used here.
tapeSegment = tape[tapePos]
for x in range(0,len(ruleTable)): #Error here.
if ruleTable[x][2] == machineState and ruleTable[x][3] == tape[tapePos]:
machineState = ruleTable[x][4]
tape[tapePos] = ruleTable[x][5]
move(ruleTable[x][6])
TheRules:
ruleTable = [1]
ruleTable[0] = ("startRule","anyVal","anyVal","1","1",1)
#New rules go down here:
To be precise: "NameError: Global name 'ruleTable' is not defined"
What would be the easiest way to deal with this? I am thoroughly confused by it.
import TheRules only loads TheRules module into local namesapce, not its contents (varaibles, functions, ...).
Use from TheRules import ruleTable to load ruleTable to local namespace.
Or access the variable using TheRules.ruleTable
I have post the similar question before,however,I think I may have misinterpreted my question,so may I just post my origin code here,and looking for someone can help me,I am really stuck now..thanks alot.
from numpy import *
import math as M
#initial condition All in SI unit
G=6.673*10**-11 #Gravitational constant
ms=1.9889*10**30 #mass of the sun
me=5.9742*10**24 #mass of the earth
dt=10 #time step
#Creat arrays
vs=array([[0,0,0]]) #1st element stand for x component of V of earth
ve=array([[29770,0,0]])
rs=array([[0,0,0]])
re=array([[0,1.4960*10**11,0]])
#First update velocity in order to start leapfrog approximation
fs=-G*ms*me*((rs-re)/(M.sqrt((rs-re)[0][0]**2+(rs-re)[0][1]**2+(rs-re)[0][2]**2))**3)
fe=-fs
vs=vs+fs*dt/ms
ve=ve+fe*dt/me
n=input('please enter the number of timestep you want it evolve:')
#update force
def force(n,ms,me,rs,re,G):
rs,re=update_r(rs,re,n,dt)
fs=-G*ms*me*((rs-re)/(M.sqrt((rs-re)[0][0]**2+(rs-re)[0][1]**2+(rs-re)[0][2]**2))**3)
fe=-fs
return fs,fe
#update velocities
def update_v(n,vs,ve,ms,me,dt,fs,fe):
fs,fe=force(n,ms,me,rs,re,G)
i=arange(n)
vs=vs+fs[:]*i[:,newaxis]*dt/ms
ve=ve+fe[:]*i[:,newaxis]*dt/me
return vs,ve
#update position
def update_r(rs,re,n,dt):
vs,ve=update_v(n,vs,ve,ms,me,dt,fs,fe)
i=arange(n)
rs=rs+vs[:]*i[:,newaxis]*dt
re=re+ve[:]*i[:,newaxis]*dt
return rs,re
#there is start position,v,r,f all have initial arrays(when n=0).
#then it should calculate f(n=1) then use this to update v(n=0)
#to v(n=1),then use v(n=1) update r(n=0) to r(n=1),then use r(n=1)
#update f(n=1) to f(n=2)....and so on until finish n.but this code seems doesnt do this,,how can I make it? –
when i call force python gives:
please enter the number of timestep you want it evolve:4Traceback (most recent call last):
File "<pyshell#391>", line 1, in <module>
force(n,ms,me,rs,re,G)
File "/Users/Code.py", line 24, in force
rs,re=update_r(rs,re,n,dt)
File "/Users/Code.py", line 39, in update_r
vs,ve=update_v(n,vs,ve,ms,me,dt,fs,fe)
UnboundLocalError: local variable 'vs' referenced before assignment
can anyone give me some tips?thanks......
where do you call force in this code?
In any event, the problem is in update_r. You reference vs in the first line of update_r even though vs is not defined in this function. Python is not looking at the vs defined above. Try adding
global vs
as the first line of update_r or adding vs to the parameter list for update_r
In the first line of update_r, you have vs,ve=update_v(n,vs,ve,ms,me,dt,fs,fe). Look at the function that you are calling. You are calling update_v with a bunch of parameters. One of these parameters is vs. However, that is the first time in that function that vs appears. The variable vs does not have a value associated with it yet. Try initializing it first, and your error should disappear
Put an additional global statement containing all your globals after each def statement. Otherwise, all globals are transformed into locals within your def without it.
def update_v(n,vs,ve,ms,me,dt,fs,fe):
global vs, ve, ...
On line 39 you do
vs,ve=update_v(n,vs,ve,ms,me,dt,fs,fe)
while you are inside a function.
Since you defined a global variable called vs, you would expect this to work.
It would have worked if you had:
vs_new,ve_new = update_v(n,vs,ve,ms,me,dt,fs,fe)
because then the interpreter knows vs in the function arguments is the global one. But since you had vs in the left hand side, you created an uninitialized local variable.
But dude, you have a much bigger problem in your code:
update_r calls update_v, update_v calls force, and force calls update_r - you will get a stack overflow :)
I got that error when my class name was assigned to a variable that is called exactly like its name. example ClassName = ClassName. You may do this if you come from .Net