I have programmend a framework that concatenate different ( quite complicated ) linear operators in an abstract manner. It overrides the operators, "+,*,#,-" and chooses a path through the graph of compositions of functions. It isn't easy to debug to say the least, however the control flow isn't depending on the data itself and of course any operation is done with tensorflow. I was hoping to use tf.function to compile it and get an ( hopefully much faster) tf.function by XLA. However I get the following error:
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
#tf.function
def has_init_scope():
my_constant = tf.constant(1.)
with tf.init_scope():
added = my_constant * 2
The graph tensor has name: Reshape_2:0
I do not use the tf.init_scope anywhere and there are 8 (!) google results regarding to this error - while none of them provides me any clue how to debug it.
# initilize linear operators, these are python objects that override __matmul__ etc.
P = ...
A = ...
# initilize vectors, these are compatible python objects to P and A
x = ...
y = ...
# This function recreates the python object from its raw tensorflow data.
# Since it might be dependend on the spaces and
# they also need to be set up for deserializaton the method is returned by a function of x.
# But since many vectors share the same spaces I was hoping to reuse it.
deserialize = x.deserialize()
# We want to compile the action on x to a function now
def meth( data ):
result = P # ( A.T # A # deserialize( data ) )
# we return only the raw data
return result.serialize()
meth = tf.function( meth,
#experimental_compile = True ,
input_signature = (x.serialize_signature,),
).get_concrete_function()
# we want to use meth now for many vectors
# executing this line throws the error
meth(x1)
meth(x2)
meth(x3)
Needless to say that works without the tf.function.
Did anyone stumble across the error and can help me to understand it better ? Or is the hole setup I'm trying not suitable for tensorflow ?
Edit:
The Error was caused by implicitly capturing a constant tensor in the linear operator class by a local lambda. To be honest, the error message suggest something like that, however it was difficult to understand which line in the code caused it and it wasn't easy to find the bug in the end.
Related
I am working on some paper replication, but I am having trouble with it.
According to the log, it says that RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation. However, when I check the line where the error is referring to, it was just a simple property setter inside the class:
#pdfvec.setter
def pdfvec(self, value):
self.param.pdfvec[self.key] = value # where the error message is referring to
Isn't in-place operations are something like += or *= etc.? I don't see why this error message appeared in this line.
I am really confused about this message, and I will be glad if any one knows any possible reason this can happen.
For additional information, this is the part where the setter function was called:
def _update_params(params, pdfvecs):
idx = 0
for param in params:
totdim = param.stats.numel()
shape = param.stats.shape
param.pdfvec = pdfvecs[idx: idx + totdim].reshape(shape) # where the setter function was called
idx += totdim
I know this can still lack information for solving the problem, but if you know any possiblity why the error message appeared I would be really glad to hear.
In-place operation means the assignment you've done is modifiying the underlying storage of your Tensor, of which requires_grad is set to True, according to your error message.
That said, your param.pdfvec[self.key] is not a leaf Tensor, because they will be updated during back-propagation. And you tried to assign a value to it , that will interference with autograd, so this action is prohibited by default. You can do this by directly modifying its underlying storage(f.e., with .data).
I am using this customized function to reshape my tensors in the customized loss function.
def reshape_fortran(x, shape):
if len(x.shape) > 0:
x = x.permute(*reversed(range(len(x.shape))))
return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape))))
Though, I receive this error:
RuntimeError: _unsafe_view does not support automatic differentiation for outputs with complex dtype.
for reshape_fortran output.
Do you know what might be the problem? which function is not supported in Pytorch autograd for complex numbers?
Complex Autograd was in Beta as of version 1.8, but is now stable and should fully support such operations as of 1.9.
I was able to fix the problem by using this function instead, but I have no idea why it was not working and why it is working now:
def convert_output84_torch(input):
shape84 = (8,4)
T1 = torch.transpose(input,0,2).permute(0,1,2).contiguous()
T2 = T1.view(shape84[::-1])
output = torch.transpose(T2,0,1)
return output
I replaced reshape with view, and it works fine (for now). But as my code was working fine before, I am not sure if it is a long-term solution as I am not aware of the main source.
I am encountering some problems when translating the following code from MATLAB to Python:
Matlab code snippet:
x=M_test %M_test is a 1x3 array that holds the adjustment points for the function
y=R_test %R_test is also a 1x3 array
>> M_test=[0.513,7.521,13.781]
>> R_test=[2.39,3.77,6.86]
expo3= #(b,x) b(1).*(exp(-(b(2)./x).^b(3)));
NRCF_expo3= #(b) norm(y-expo3(b,x));
B0_expo3=[fcm28;1;1];
B_expo3=fminsearch(NRCF_expo3,B0_expo3);
Data_raw.fcm_expo3=(expo3(B_expo3,Data_raw.M));
The translated (python) code:
expo3=lambda x,M_test: x[0]*(1-exp(-1*(x[1]/M_test)**x[2]))
NRCF_expo3=lambda R_test,x,M_test: np.linalg.norm(R_test-expo3,ax=1)
B_expo3=scipy.optimize.fmin(func=NRCF_expo3,x0=[fcm28,1,1],args=(x,))
For clarity, the object function 'expo3' wants to go through the adjustment points defined by M_test.
'NRCF_expo3' is the function that wants to be minimised, which is basically the error between R_test and the drawn exponential function.
When I run the code, I obtain the following error message:
B_expo3=scipy.optimize.fmin(func=NRCF_expo3,x0=[fcm28,1,1]),args=(x,))
NameError: name 'x' is not defined
There are a lot of similar questions that I have perused.
If I delete the 'args' from the optimization function, as numpy/scipy analog of matlab's fminsearch
seems to indicate it is not necessary, I obtain the error:
line 327, in function_wrapper
return function(*(wrapper_args+args))
TypeError: <lambda>() missing 2 required positional arguments: 'x' and 'M_test'
There are a lot of other modifications that I have tried, following examples like Using scipy to minimize a function that also takes non variational parameters or those found in Open source examples, but nothing works for me.
I expect this is probably quite obvious, but I am very new to Python and I feel like I am looking for a needle in a haystack. What am I not seeing?
Any help would be really appreciated. I can also provide more code, if that is necessary.
I think you shouldn't use lambdas in your code, make instead a single target function with your three parameters (see PEP8). There is a lot of missing information in you post, but for what I can infer, you want something like this:
from scipy.optimize import fmin
# Define parameters
M_TEST = np.array([0.513, 7.521, 13.781])
X_ARR = np.array([2.39,3.77,6.86])
X0 = np.array([10, 1, 1]) # whatever your variable fcm28 is
def nrcf_exp3(r_test, m_test, x):
expo3 = x[0] * (1 - np.exp(-(x[1] / m_test) ** x[2]))
return np.linalg.norm(r_test - expo3)
fmin(nrcf_exp3, X0, args=(M_TEST, X_ARR))
Numba & CUDA noob here. I'd like to be able to have one numba.cuda function programmatically call another one from the device, without having to pass any data back to the host. For example, given the setup
from numba import cuda
#cuda.jit('int32(int32)', device=True)
def a(x):
return x+1
#cuda.jit('int32(int32)', device=True)
def b(x):
return 2*x
I'd like to be able to define a composition kernel function like
#cuda.jit('void(int32, __device__, int32)')
def b_comp(x, inner, result):
y = inner(x)
result = b(y)
and successfully obtain
b_comp(1, a, result)
assert result == 4
Ideally I'd like b_comp to accept varying function arguments after it compiles [e.g. after the above call, to still accept b_comp(1, b, result)] -- but a solution where the function arguments become fixed at compile time will still work for me.
From what I've read, it seems that CUDA supports passing function pointers. This post suggests that numba.cuda has no such support, but the post isn't convincing, and is also a year old. The page for supported Python in numba.cuda doesn't mention function pointer support. But it links to the supported Python in numba page, which makes it clear that numba.jit() does support functions as arguments, although they get fixed at compile time. If numba.cuda.jit() does the same, like I said above, that'll work. In that case, when specifying the signature for comp, how should I state the variable type? Or could I use numba.cuda.autojit()?
If numba doesn't support any such direct approach, is metaprogramming a reasonable option? E.g. once I know the inner function, my script could create a new script containing a python function that composes those specific functions, and then apply numba.cuda.jit(), and then import the result. It seems convoluted, but it's the only other numba-based option I could think of.
If numba won't do the trick at all, or at least not without serious cludgery, I'd be happy with an answer that gave a few details, plus a rec like "switch to PyCuda".
Here's what worked for me:
Not decorating my functions with cuda.jit initially, so that they still possess the __name__ attribute
Getting the __name__ attribute
Now applying cuda.jit to my functions by directly calling the decorator
Creating the python for the composition function in a string, and passing it to exec
The exact code:
from numba import cuda
import numpy as np
def a(x):
return x+1
def b(x):
return 2*x
# Here, pretend we've been passed the inner function and the outer function as arguments
inner_fun = a
outer_fun = b
# And pretend we have noooooo idea what functions these guys actually point to
inner_name = inner_fun.__name__
outer_name = outer_fun.__name__
# Now manually apply the decorator
a = cuda.jit('int32(int32)', device=True)(a)
b = cuda.jit('int32(int32)', device=True)(b)
# Now construct the definition string for the composition function, and exec it.
exec_string = '#cuda.jit(\'void(int32, int32[:])\')\n' \
'def custom_comp(x, out_array):\n' \
' out_array[0]=' + outer_name + '(' + inner_name + '(x))\n'
exec(exec_string)
out_array = np.array([-1])
custom_comp(1, out_array)
print(out_array)
As expected, the output is
[4]
I have a Pyomo model which has the form:
from pyomo.environ import *
from pyomo.dae import *
m = AbstractModel()
m.t = ContinuousSet(bounds=(0,120))
m.T = Param(default=120)
m.S = Var(m.t, bounds=(0,None))
m.Sdot = DerivativeVar(m.S)
m.obj = Objective(expr=m.S[120],sense=maximize)
Note that the objective m.obj relies on the parameter m.T. Attempting to run this gives the error:
TypeError: unhashable type: 'SimpleParam'
Using a value, such as expr=m.S[120] gives the error:
ValueError: Error retrieving component S[120]: The component has not been constructed.
In both cases, my goal is the same: to optimize for the largest possible value of S at the horizon.
How can I create an abstract model which expresses this?
You are hitting on two somewhat separate issues:
TypeError: unhashable type: 'SimpleParam'
Is due to a bug in Pyomo 4.3 where you cannot directly use simple Params as indexes into other components. That said, the fix for this particular problem will not fix your example model.
The trick to fixing your Objective declaration is to encapsulate the Objective expression within a rule:
def obj_rule(m):
return m.S[120]
# or better yet:
# return m.S[m.T]
# or
# return m.S[m.t.last()]
m.obj = Objective(rule=obj_rule,sense=maximize)
The problem is that when you are writing an Abstract model, each component is only being declared, but not defined. So, the Var S is declared to exist, but has not been defined (it is an empty shell with no members). This causes a problem because Python (not Pyomo) attempts to resolve the m.S[120] to a specific variable immediately before calling the Objective constructor. The use of rules (functions) in Abstract models allows you to defer the resolution of the expression until Pyomo is actually constructing the model instance. Pyomo constructs the instance components in the same order that you declared them on the Abstract model, so when it fires the obj_rule, the previous components (S, T, and t) are all constructed and S has valid members at the known points of the ContinuousSet (in this case, the bounds).