I have a dataset, which I call with batch['data'] and get my image output MxM. After I get my image I want to process it with some numpy operations. In this process I want my dataset to give me the image with GPU and changing the outputs device to CPU after that.
My question is, is concetanation of functions in Python being executed in an order? And can I make this process with
base = batch['data'].cuda().function().cpu()
And is this the same as:
base = batch['data'].cuda().function()
base.cpu()
Thanks in advance!
Well, the CPU(s) will do the same work, but the result is not the same.
base = batch['data'].cuda().cpu()
After that line, you have the output of cpu() stored in the variable called base.
base = batch['data'].cuda()
base.cpu()
After these two lines, you have the output of cuda() stored in the variable called base and you have forgotten the result of cpu().
is concatenation of functions in Python being executed in an order?
Yes, of course: the first method returns some object, and the next one is called on that returned object.
No, these pieces of code are not the same:
The first one assigns the return value of cpu to base
The second one throws this value away
Also, if you need the object returned by batch['data'].cuda(), then the first code will call cpu on it and potentially throw it away afterwards. The second one saves that object but gets rid of the result of calling cpu, which may not be desirable
Same thing is with writing batch['data'].cuda() or tmp = batch['data']; base = tmp.cuda(): batch['data'] returns some object, and then .cuda can be called on that object.
As long as functions return object that have the methods you want to call, you can chain as many methods as you want to: thing().a().b().c().d()
Related
I am trying to write a testing program for a python program that takes data, does calculations on it, then puts the output in a class instance object. This object contains several other objects, each with their own attributes. I'm trying to access all the attributes and sub-attributes dynamically with a one size fits all solution, corresponding to elements in a dictionary I wrote to cycle through and get all those attributes for printing onto a test output file.
Edit: this may not be clear from the above but I have a list of the attributes I want, so using something to actually get those attributes is not a problem, although I'm aware python has methods that accomplish this. What I need to do is to be able to get all of those attributes with the same function call, regardless of whether they are top level object attributes or attributes of object attributes.
Python is having some trouble with this - first I tried doing something like this:
for string in attr_dictionary:
...
outputFile.print(outputclass.string)
...
But Python did not like this, and returned an AttributeError
After checking SE, I learned that this is a supposed solution:
for string in attr_dictionary:
...
outputFile.print(getattr(outputclass, string))
...
The only problem is - I want to dynamically access the attributes of objects that are attributes of outputclass. So ideally it would be something like outputclass.objectAttribute.attribute, but this does not work in python. When I use getattr(outputclass, objectAttribute.string), python returns an AttributeError
Any good solution here?
One thing I have thought of trying is creating methods to return those sub-attributes, something like:
class outputObject:
...
def attributeIWant(self,...):
return self.subObject.attributeIWant
...
Even then, it seems like getattr() will return an error because attributeIWant() is supposed to be a function call, it's not actually an attribute. I'm not certain that this is even within the capabilities of Python to make this happen.
Thank you in advance for reading and/or responding, if anyone is familiar with a way to do this it would save me a bunch of refactoring or additional code.
edit: Additional Clarification
The class for example is outputData, and inside that class you could have and instance of the class furtherData, which has the attribute dataIWant:
class outputData:
example: furtherData
example = furtherData()
example.dataIWant = someData
...
with the python getattr I can't access both attributes directly in outputData and attributes of example unless I use separate calls, the attribute of example needs two calls to getattr.
Edit2: I have found a solution I think works for this, see below
I was able to figure this out - I just wrote a quick function that splits the attribute string (for example outputObj.subObj.propertyIWant) then proceeds down the resultant array, calling getattr on each subobject until it reaches the end of the array and returns the actual attribute.
Code:
def obtainAttribute(sample, attributeString: str):
baseObj = sample
attrArray = attributeString.split(".")
for string in attrArray:
if(attrArray.index(string) == (len(attrArray) - 1)):
return getattr(baseObj,string)
else:
baseObj = getattr(baseObj,string)
return "failed"
sample is the object and attributeString is, for example object.subObject.attributeYouWant
I am writing a program in Python that communicates with a spectrometer from Avantes. There are some proprietary dlls available whose code I don't access to, but they have some decent documentation. I am having some trouble to find a good way to store the data received via callbacks.
The proprietary shared library
Basically, the dll contains a function that I have to call to start measuring and that receives a callback function that will be called whenever the spectrometer has finished a measurement. The function is the following:
int AVS_MeasureCallback(AvsHandle a_hDevice,void (*__Done)(AvsHandle*, int*),short a_Nmsr)
The first argument is a handle object that identifies the spectrometer, the second is the actual callback function and the third is the amount of measurements to be made.
The callback function will receive then receive another type of handle identifying the spetrometer and information about the amount of data available after a measurement.
Python library
I am using a library that has Python wrappers for many equipments, including my spectrometer.
def measure_callback(self, num_measurements, callback=None):
self.sdk.AVS_MeasureCallback(self._handle, callback, num_measurements)
And they also have defined the following decorator:
MeasureCallback = FUNCTYPE(None, POINTER(c_int32), POINTER(c_int32))
The idea is that when the callback function is finally called, this will trigger the get_data() function that will retrieve data from the equipment.
The recommended example is
#MeasureCallback
def callback_fcn(handle, info):
print('The DLL handle is:', handle.contents.value)
if info.contents.value == 0: # equals 0 if everything is okay (see manual)
print(' callback data:', ava.get_data())
ava.measure_callback(-1, callback_fcn)
My problem
I have to store the received data in a 2D numpy array that I have created somewhere else in my main code, but I can't figure out what is the best way to update this array with the new data available inside the callback function.
I wondered if I could pass this numpy array as an argument for the callback function, but even in this case I cannot find a good way to do this since it is expected that the callback function will have only those 2 arguments.
Edit 1
I found a possible solution here but I am not sure it is the best way to do it. I'd rather not create a new class just to hold a single numpy array inside.
Edit 2
I actually changed my mind about my approach, because inside my callback I'd like to do many operations with the received data and save the results in many different variables. So, I went back to the class approach mentioned here, where I would basically have a class with all the variables that will somehow be used in the callback function and that would also inherit or have an object of the class ava.
However, as shown in this other question, the self parameter is a problem in this case.
If you don't want to create a new class, you can use a function closure:
# Initialize it however you want
numpy_array = ...
def callback_fcn(handle, info):
# Do what you want with the value of the variable
store_data(numpy_array, ...)
# After the callback is called, you can access the changes made to the object
print(get_data(numpy_array))
How this works is that when the callback_fcn is defined, it keeps a reference to the value of the variable numpy_array, so when it's called, it can manipulate it, as if it were passed as an argument to the function. So you get the effect of passing it in, without the callback caller having to worry about it.
I finally managed to solve my problem with a solution envolving a new class and also a closure function to deal with the self parameter that is described here. Besides that, another problem would appear by garbage collection of the new created method.
My final solution is:
class spectrometer():
def measurement_callback(self,handle,info):
if info.contents.value >= 0:
timestamp,spectrum = self.ava.get_data()
self.spectral_data[self.spectrum_index,:] = np.ctypeslib.as_array(spectrum[0:pixel_amount])
self.timestamps[self.spectrum_index] = timestamp
self.spectrum_index += 1
def __init__(self,ava):
self.ava = ava
self.measurement_callback = MeasureCallback(self.measurement_callback)
def register_callback(self,scans,pattern_amount,pixel_amount):
self.spectrum_index = 0
self.timestamps = np.empty((pattern_amount),dtype=np.uint32)
self.spectral_data = np.empty((pattern_amount,pixel_amount),dtype=np.float64)
self.ava.measure_callback(scans, self.measurement_callback)
I'm writing a python script that takes user input through a GUI and then passes that along to a function that basically reads through text files and checks that what the user requested is there. This function returns an array with True or False for each check.
I want to use this array in a different function (def markup()), but if I call it without giving the function the user input, I get an error.
Is there a way for me to store the results of this function and pass it without needing the user input each time?
Pseudo code:
def clickButton():
userInput = [A,B,C,D,E]
textCheck(userInput)
def textCheck(userInput):
*code for checking text creates an array named allResults*
return allResults
def markup():
results = textCheck()
print(results)
You need to manage allResults as a persistent object. One way is to pass the results everywhere as a parameter, such that whatever thread is executing always has a handle to the list. Another way is to (shudder) make it a global variable -- this is somewhat dangerous as a habit, but may be the easiest for you to implement and maintain. You can also create a Results class and instantiate an object that persists as long as you need it.
I can't recommend one over another without having the flow of the main program.
I ended up calling markup(allResults) at the end of textChecker(). It worked.
I have a whole series of arrays with similar names mcmcdata.rho0, mcmcdata.rho1, ... and I want to be able to loop through them while updating their values. I can't figure out how this might be done or even what such a thing might be called.
I read my data in from file like this:
names1='l b rho0 rho1 rho2 rho3 rho4 rho5 rho6 rho7 rho8 rho9 rho10 rho11 rho12 rho13 rho14 rho15 rho16 rho17 rho18 rho19 rho20 rho21 rho22 rho23'.split()
mcmcdata=np.genfromtxt(filename,names=names1,dtype=None).view(np.recarray)
and I want to update the "rho" arrays later on after I do some calculations.
for jj in range(dbins):
mcmc_x, mcmc_y, mcmc_z = wf.lbd_to_xyz(mcmcdata.l,mcmcdata.b,d[jj],R_sun)
rho, thindisk, thickdisk, halo = wf.total_density_fithRthinhRthickhzthinhzthickhrfRiA( mcmc_x, mcmc_y, mcmc_z, R_sun,params)
eval("mcmcdata."+names1[2+jj]) = copy.deepcopy(rho)
eval("mcmcthin."+names1[2+jj]) = copy.deepcopy(thindisk)
eval("mcmcthick."+names1[2+jj]) = copy.deepcopy(thickdisk)
eval("mcmchalo."+names1[2+jj]) = copy.deepcopy(halo)
But the eval command is giving an error:
File "<ipython-input-133-30322c5e633d>", line 13
eval("mcmcdata."+names1[2+jj]) = copy.deepcopy(rho)
SyntaxError: can't assign to function call
How can I loop through my existing arrays and update their values?
or
How can identify the arrays by name so I can update them?
The eval command doesn't work the way you seem to think it does. You appear to be using it like a text-replacement macro, hoping that Python will read the given string and then pretend you wrote that text in the original source code. Instead, it receives a string, and then it executes that code. You're giving it an expression that refers to an attribute of an object, which is fine, but the result of evaluating that expression does not yield a thing you can assign to. It yields the value of that attribute.
Although Python provides eval, it also provides many other things that often obviate the need for eval. In the case of your code, Python provides setattr. You give it an object, the name of an attribute on that object, and a value, and it assigns that object's attribute to refer to the given value.
setattr(mcmcdata, names1[2+jj], copy.deepcopy(rho))
It might make the code more readable to get rid of the names1 portion, too. I might write the code like this:
setattr(mcmcdata, 'rho' + str(jj), copy.deepcopy(rho))
That way, it's clear that I'm assigning the rho-related attributes of the object without having to go look at what's held in the names1 list; the name names1 doesn't offer much information about what's in it.
Suppose I have the following code:
def my_func(input_line):
is_skip_line = self.is_skip_line(input_line) # parse input line check if skip line
if is_skip_line:
# do something...
# do more ...
if is_skip_line:
# do one last thing
So we have a check for is_skip_line (if is_skip_line:) that appears twice. Does it mean that due to lazy evaluation the method self.is_skip_line(input_line) will be called twice?
If so, what is the best work around, given that self.is_skip_line(input_line) is time consuming? Do I have to "immediately invoke" it, like below?
is_skip_line = (lambda x: self.is_skip_line(x))(input_line)
Thanks.
The misconception here is that this statement is not being immediately invoked:
is_skip_line = self.is_skip_line(input_line)
...when in fact, it is.
The method self.is_skip_line will only ever be invoked once. Since you assign it to a variable, you can use that variable as many times as you like in any context you like.
If you're concerned about the performance of it, then you could use cProfile to really test the performance of the method it's called in with respect to the method it's calling.