I'm programming an optimizer that has to run through several possible variations. The team wants to implement multithreading to get through those variants faster. This means I've had to put all my functions inside a thread-class. My problem is with my call of the wrapper function
class variant_thread(threading.Thread):
def __init__(self, name, variant, frequencies, fit_vals):
threading.Thread.__init__(self)
self.name = name
self.elementCount = variant
self.frequencies = frequencies
self.fit_vals = fit_vals
def run(self):
print("Running Variant:", self.elementCount) # display thread currently running
fitFunction = self.Wrapper_Function(self.elementCount)
self.popt, pcov, self.infoRes = curve_fit_my(fitFunction, self.frequencies, self.fit_vals)
def Optimize_Wrapper(self, frequencies, *params): # wrapper which returns values in manner which optimizer can work with
cut = int(len(frequencies)/2) <---- ERROR OCCURS HERE
freq = frequencies[:cut]
vals = (stuff happens here)
return (stuff in proper form for optimizer)
I've cut out as much as I could to simplify the example, and I hope you can understand what's going on. Essentially, after the thread is created it calls the optimizer. The optimizer sends the list of frequencies and the parameters it wants to change to the Optimize_Wrapper function.
The problem is that Optimize-Wrapper takes the frequencies-list and saves them to "self". This means that the "frequencies" variable becomes a single float value, as opposed to the list of floats it should be. Of course this throws an errorswhen I try to take len(frequencies). Keep in mind I also need to use self later in the function, so I can't just create a static method.
I've never had the problem that a class method saved any values to "self". I know it has to be declared explicitly in Python, but anything I've ever passed to the class method always skips "self" and saves to my declared variables. What's going on here?
Don't pass instance variables to methods. They are already accessible through self. And be careful about which variable is which. The first parameter to Wrapper_function is called "frequency", but you call it as self.Wrapper_Function(self.elementCount) - so you have a self.frequency and a frequency ... and they are different things. Very confusing!
class variant_thread(threading.Thread):
def __init__(self, name, variant, frequencies, fit_vals):
threading.Thread.__init__(self)
self.name = name
self.elementCount = variant
self.frequencies = frequencies
self.fit_vals = fit_vals
def run(self):
print("Running Variant:", self.elementCount) # display thread currently running
fitFunction = self.Wrapper_Function()
self.popt, pcov, self.infoRes = curve_fit_my(fitFunction, self.frequencies, self.fit_vals)
def Optimize_Wrapper(self): # wrapper which returns values in manner which optimizer can work with
cut = int(len(self.frequencies)/2) <---- ERROR OCCURS HERE
freq = self.frequencies[:cut]
vals = (stuff happens here)
return (stuff in proper form for optimizer)
You don't have to subclass Thread to run a thread. Its frequently easier to define a function and have Thread call that function. In your case, you may be able to put the variant processing in a function and use a thread pool to run them. This would save all the tedious handling of the thread object itself.
def run_variant(name, variant, frequencies, fit_vals):
cut = int(len(self.frequencies)/2) <---- ERROR OCCURS HERE
freq = self.frequencies[:cut]
vals = (stuff happens here)
proper_form = (stuff in proper form for optimizer)
return curve_fit_my(fitFunction, self.frequencies, self.fit_vals)
if __name__ == "__main__":
variants = (make the variants)
name = "name"
frequencies = (make the frequencies)
fit_vals = (make the fit_vals)
from multiprocessing.pool import ThreadPool
with ThreadPool() as pool:
for popt, pcov, infoRes in pool.starmap(run_variant,
((name, variant, frequencies, fit_vals) for variant in variants)):
# do the other work here
Related
In Python, is there a simple way for an invoked function to get a value from the calling function/class ? I'm not sure if I'm phrasing that right, but I'm trying to do something like this:
class MainSection(object):
def function(self):
self.var = 47 # arbitrary variable
self.secondaryObject = secondClass() # Create object of second class
self.secondaryObject.secondFunction(3) # call function in that object
and
class secondClass(object):
def secondFunction(self, input)
output = input + self.var # calculate value based on function parameter AND variable from calling function
return output
#Access self.var from MainSection
This might be my lack of knowledge about Python, but I'm having a hard time finding a clear answer here. Is the best way to do that just passing the variable I want in as another second parameter to the second class?
These are in separate files, if that makes a difference.
Is the best way to do that just passing the variable I want in as another second parameter to the second class?
Yes, especially if there's only a transient relationship between the objects:
class secondClass(object):
def secondFunction(self, input, var_from_caller)
output = input + var_from_caller # calculate value based on function parameter AND variable from calling function
return output
You can even pass around the whole object if you like:
class secondClass(object):
def secondFunction(self, input, calling_object)
output = input + calling_object.var # calculate value based on function parameter AND variable from calling function
return output
If the relationship is more permanent, you could consider storing references to the related objects in instance variables:
class MainSection(object):
def function(self):
self.var = 47 # arbitrary variable
self.secondaryObject = secondClass(self) # Create object of second class
self.secondaryObject.secondFunction(3) # call function in that object
...
class secondClass(object):
def __init__(self, my_friend):
self.related_object = my_friend
def secondFunction(self, input)
output = input + self.related_object.var # calculate value based on function parameter AND variable from calling function
return output
#Access self.var from MainSection
I am writing some code that i have threaded, and am using various different functions at once. I have a variable called ref that is different for each thread.
ref is defined within a function within the threaded function, so when I use global ref, all the threads use the same value for ref (which I don't want). However when I don't use global ref, other functions can't use ref as it is not defined.
E.g.:
def threadedfunction():
def getref():
ref = [get some value of ref]
getref()
def useref():
print(ref)
useref()
threadedfunction()
If defining ref as global doesn't fit your needs then you don't have many other options...
Edit your function's parameters and returns. Possible solution:
def threadedfunction():
def getref():
ref = "Hello, World!"
return ref # Return the value of ref, so the rest of the world can know it
def useref(ref):
print(ref) # Print the parameter ref, whatever it is.
ref = getref() # Put in variable ref whatever function getref() returns
useref(ref) # Call function useref() with ref's value as parameter
threadedfunction()
I am writing a simple class to retrieve a signal x digitized at constant sampling rate Fs. Digitization begins at time t0. Given the signal length N = len(x), the sampling rate, and the initial time, the signal's time base is uniquely determined. I rarely need to access the time base, but I would like an easy means of doing so when needed. Below, I've implemented a minimal working example of my desired time-base functionality using a property() decorator:
import numpy as np
class Signal(object):
def __init__(self, x, Fs, t0):
self.x = x
self.Fs = Fs
self.t0 = t0
return
#property
def t(self):
return self.t0 + (np.arange(len(self.x)) / self.Fs)
I'd like to know about the creation and "persistence" of the time-base property Signal.t. Take the example use case below:
x = np.arange(10)
Fs = 1.
t0 = 0.
sig = Signal(x, Fs, t0)
print sig.t
When is the time-base array t generated? During initialization or dynamically when print sig.t is called? If the sig.t attribute is dynamically calculated, will it persist beyond the print command? (i.e. has memory been allocated to store the time base as an object attribute?).
While the above is a trivial example, my typical signals are very large, and I do not want the memory overhead of creating and storing the time base for every signal. I'd like an easy means of dynamically generating the time base on an as-needed basis, however; the time base should not persist as an object attribute after its one-off use (e.g. for creating a plot of the raw signal).
If the property() decorator does not provide this desired functionality (i.e. minimal memory overhead, ease of use on an as-needed basis), what should I be using? Simply a class method? Or is there a different, more optimal solution? Thanks!
Every time you access sig.t, the t function you decorated with #property is (re)run, and the result is used as the value of sig.t. That means the array is created on demand and never stored on the t object.
You seem to want this, but I'd be wary about it. Property accesses are generally expected to be cheap, and this property isn't. Consider making an ordinary method instead.
When is the time-base array t generated?
When it is used. i.e. when you write print sig.t
If the sig.t attribute is dynamically calculated, will it persist beyond the print command? (i.e. has memory been allocated to store the time base as an object attribute?).
Nope. The next time your code references sig.t, a new object will be created.
If the property() decorator does not provide this desired functionality (i.e. minimal memory overhead, ease of use on an as-needed basis), what should I be using? Simply a class method? Or is there a different, more optimal solution? Thanks!
There are differing opinions here I suspect... you can modify the code so that you cache the value and return the same thing each call:
class Signal(object):
def __init__(self, x, Fs, t0):
self.x = x
self.Fs = Fs
self.t0 = t0
self._t = None
return
#property
def t(self):
if self._t is not None:
return self._t
self._t = self.t0 + (np.arange(len(self.x)) / self.Fs)
return self._t
But here you have no way of telling the class that t should be re-computed unless you make a setter...
If t isn't going to change after initialization, then why not just make it a public property?
class Signal(object):
def __init__(self, x, Fs, t0):
self.x = x
self.Fs = Fs
self.t0 = t0
self.t = self.t0 + (np.arange(len(self.x)) / self.Fs)
In your example, you are dynamically generating the Signal.t value every time you access the attribute t because you are essentially calling Signal.t() to access it.So, you are not storing the value just returning it.
Whenever there is a #property decorator being used around a class, it is quite common for that function to act as a "getter" for a "private"(not really) variable, and sometimes there is a "setter" for that "private" variable.
When I mean "private" I really mean attributes that are named to reflect that these attributes should not be accessed directly. However, in python you can access any attribute and so there isn't any private variables since python objects can be changed quite easy.
If you want to store your values, then you should do something like this.
import numpy as np
class Signal(object):
def __init__(self, x, Fs, t0):
self.x = x
self.Fs = Fs
self.t0 = t0
self._t = None
return
#property
def t(self):
if self._t is None:
self._t = self.t0 + (np.arange(len(self.x)) / self.Fs)
return self._t
#t.setter
def t(self,value):
self._t = value
The above example, will only calculate it once and store it inside _t , but you get the point. When the #property decorator is being used, usually there is an underlying variable that is used to retrieve and store a value.Hope that helps
So i have a relatively convoluted setup for something I'm working on explained as follows:
This is is python. and more of a rough outline, but it covers everything I need. Though the process next function is the same so feel free to clean that up if you want.
#timer event that runs every .1 second and processes events in a queue
some_event_timer():
events.process_next()
class Event_queue:
def __init__(self):
self.events = []
def push(self, event, parameters):
self.events.insert(len(self.events), event, parameters)
def process_next(self):
event = self.pop(0)
event[0](event[1])
class Foo:
def __init__(self, start_value = 1):
self.value = start_value
def update_value(self, multiple):
self.value *= multiple
def return_bah(self)
return self.value + 3
class Bar:
def __init__(self, number1, number2):
self.init = number1
self.add = number2
def print_alt_value(self, in_value):
print in_value * (self.init + self.add)
That is a barebones of what I have, but it illustrates my problem:
Doing the below
events2 = Event_queue2()
foo1 = Foo(4) ----> foo1.value = 4 here
bar1 = Bar(4, 2)
events2.push(foo1.update_value,1.5)
events2.push(bar1.print_alt_value,foo1.value)
events2.push(bar.print_alt_value,foo1.return_bah())
events2.process_next() ----> should process update_value to change foo.value to 6
events2.process_next() ----> should process print_alt_value in bar class - expected 36
events2.process_next() ----> should process print_alt_value - expected 54
I initially expected my output to be 36 6 * (4 + 2)
I know why its not, foo1.value and foo1.return_bah() gets passed as an evaluated parameter (correct term?).
What I really want is to pass the reference to the variable or the reference to the method, rather than having it evaluate when I put it in my event queue.
Can anyone help me.
I tried searching, but I couldn't piece together what I wanted exactly.
TO get what I have now I initially looked at these threads:
Calling a function of a module from a string with the function's name in Python
Use a string to call function in Python
But I don't see how to support parameters from that properly or how to support passing another function or reference to a variable from those.
I suppose at least for the method call, I could perhaps pass the parameter as foo1.return.bah and evaluate in the process_next method, but I was hoping for a general way that would accept both standard variables and method calls, as the event_queue will take both.
Thank you for the help
Update edit:
So I following the suggestion below, and got really close, but:
Ok, so I followed your queue suggestion and got really close to what I want, but I don't completely understand the first part about multiple functions.
I want to be able to call a dictionary of objects with this as well.
for example:
names = ["test1", "test2"]
for name in names:
names_objs[name] = Foo(4)
Then when attempting to push via lambda
for name in names_list:
events2.push(lambda: names_objs[name].update_value(2))
doesn't work. When teh event actually gets processed it only runs on whatever name_objs[name] references, and if the name variable is no longer valid or has been modified outside the function, it is wrong.
This actually wasn't surprising, but adding a:
name_obj_hold = name_objs[name]
then pushing that didn't either. it again only operates on whatever name_obj_hold last referenced.
Can someone clarify the multiple funcs thing. I'm afraid I'm having trouble wrapping my head around it.
basically I need the initial method call evaluated, so something like:
names_objs[name].some_func(#something in here#)
gets the proper method and associated with the right class object instance, but the #something in here# doesn't get evaluated (whether it is a variable or another function) until it actually gets called from the event queue.
Instead of passing in the function to call func1 and the arguments that should be passed to the function, pass in a function func2 that calls func1 with the arguments that should be passed in.
d = {"a":1}
def p(val):
print val
def func1():
p(d["a"])
def call_it(func):
func()
call_it(func1)
d["a"] = 111
call_it(func1)
Within func1, d["a"] is not evaluated until func1 actually executes.
For your purposes, your queue would change to:
class EventQueue(object):
def __init__(self):
self.events = deque()
def push(self, callable):
self.events.append(callable)
def process_next(self):
self.events.popleft()()
collections.deque will be faster at popping from the front of the queue than a list.
And to use the EventQueue, you can use lambdas for quick anonymous function.
events2 = EventQueue()
foo1 = Foo(4)
bar1 = Bar(4, 2)
events2.push(lambda: foo1.update_value(1.5))
events2.push(lambda: bar1.print_alt_value(foo1.value))
events2.push(lambda: bar1.print_alt_value(foo1.return_bah()))
events2.process_next()
events2.process_next() # 36.0
events2.process_next() # 54.0
For Edit:
In this case you need to "capture" the value in a variable that is more tightly scoped than the loop. You can use a normal function and partial() to achieve this.
for name in names_list:
def update(name):
names_objs[name].update_value(2)
events2.push(partial(update, name))
This is a design principle question for classes dealing with mathematical/physical equations where the user is allowed to set any parameter upon which the remaining are being calculated.
In this example I would like to be able to let the frequency be set as well while avoiding circular dependencies.
For example:
from traits.api import HasTraits, Float, Property
from scipy.constants import c, h
class Photon(HasTraits):
wavelength = Float # would like to do Property, but that would be circular?
frequency = Property(depends_on = 'wavelength')
energy = Property(depends_on = ['wavelength, frequency'])
def _get_frequency(self):
return c/self.wavelength
def _get_energy(self):
return h*self.frequency
I'm also aware of an update trigger timing problem here, because I don't know the sequence the updates will be triggered:
Wavelength is being changed
That triggers an updated of both dependent entities: frequency and energy
But energy needs frequency to be updated so that energy has the value fitting to the new wavelength!
(The answer to be accepted should also address this potential timing problem.)
So, what' the best design pattern to get around these inter-dependent problems?
At the end I want the user to be able to update either wavelength or frequency and frequency/wavelength and energy shall be updated accordingly.
This kind of problems of course do arise in basically all classes that try to deal with equations.
Let the competition begin! ;)
Thanks to Adam Hughes and Warren Weckesser from the Enthought mailing list I realized what I was missing in my understanding.
Properties do not really exist as an attribute. I now look at them as something like a 'virtual' attribute that completely depends on what the writer of the class does at the time a _getter or _setter is called.
So when I would like to be able to set wavelength AND frequency by the user, I only need to understand that frequency itself does not exist as an attribute and that instead at _setting time of the frequency I need to update the 'fundamental' attribute wavelength, so that the next time the frequency is required, it is calculated again with the new wavelength!
I also need to thank the user sr2222 who made me think about the missing caching. I realized that the dependencies I set up by using the keyword 'depends_on' are only required when using the 'cached_property' Trait. If the cost of calculation is not that high or it's not executed that often, the _getters and _setters take care of everything that one needs and one does not need to use the 'depends_on' keyword.
Here now the streamlined solution I was looking for, that allows me to set either wavelength or frequency without circular loops:
class Photon(HasTraits):
wavelength = Float
frequency = Property
energy = Property
def _wavelength_default(self):
return 1.0
def _get_frequency(self):
return c/self.wavelength
def _set_frequency(self, freq):
self.wavelength = c/freq
def _get_energy(self):
return h*self.frequency
One would use this class like this:
photon = Photon(wavelength = 1064)
or
photon = Photon(frequency = 300e6)
to set the initial values and to get the energy now, one just uses it directly:
print(photon.energy)
Please note that the _wavelength_default method takes care of the case when the user initializes the Photon instance without providing an initial value. Only for the first access of wavelength this method will be used to determine it. If I would not do this, the first access of frequency would result in a 1/0 calculation.
I would recommend to teach your application what can be derived from what. For example, a typical case is that you have a set of n variables, and any one of them can be derived from the rest. (You can model more complicated cases as well, of course, but I wouldn't do it until you actually run into such cases).
This can be modeled like this:
# variable_derivations is a dictionary: variable_id -> function
# each function produces this variable's value given all the other variables as kwargs
class SimpleDependency:
_registry = {}
def __init__(self, variable_derivations):
unknown_variable_ids = variable_derivations.keys() - self._registry.keys():
raise UnknownVariable(next(iter(unknown_variable_ids)))
self.variable_derivations = variable_derivations
def register_variable(self, variable, variable_id):
if variable_id in self._registry:
raise DuplicateVariable(variable_id)
self._registry[variable_id] = variable
def update(self, updated_variable_id, new_value):
if updated_variable_id not in self.variable_ids:
raise UnknownVariable(updated_variable_id)
self._registry[updated_variable_id].assign(new_value)
other_variable_ids = self.variable_ids.keys() - {updated_variable_id}
for variable_id in other_variable_ids:
function = self.variable_derivations[variable_id]
arguments = {var_id : self._registry[var_id] for var_id in other_variable_ids}
self._registry[variable_id].assign(function(**arguments))
class FloatVariable(numbers.Real):
def __init__(self, variable_id, variable_value = 0):
self.variable_id = variable_id
self.value = variable_value
def assign(self, value):
self.value = value
def __float__(self):
return self.value
This is just a sketch, I didn't test or think through every possible issue.