I get the following UserWarning when trying to cache results using joblib:
import numpy
from tempfile import mkdtemp
cachedir = mkdtemp()
from joblib import Memory
memory = Memory(cachedir=cachedir, verbose=0)
#memory.cache
def get_nc_var3d(path_nc, var, year):
"""
Get value from netcdf for variable var for year
:param path_nc:
:param var:
:param year:
:return:
"""
try:
hndl_nc = open_or_die(path_nc)
val = hndl_nc.variables[var][int(year), :, :]
except:
val = numpy.nan
logger.info('Error in getting var ' + var + ' for year ' + str(year) + ' from netcdf ')
hndl_nc.close()
return val
I get the following warning when calling this function using parameters:
UserWarning: Persisting input arguments took 0.58s to run.
If this happens often in your code, it can cause performance problems
(results will be correct in all cases).
The reason for this is probably some large input arguments for a wrapped function (e.g. large strings).
THIS IS A JOBLIB ISSUE. If you can, kindly provide the joblib's team with an example so that they can fix the problem.
Input parameters: C:/Users/rit/Documents/PhD/Projects/\GLA/Input/LUWH/\LUWLAN_v1.0h\transit_model.nc range_to_large 1150
How do I get rid of the warning? And why is it happening, since the input parameters are not too long?
I don't have an answer to the "why doesn't this work?" portion of the question. However to simply ignore the warning you can use warnings.catch_warnings with warnings.simplefilter as seen here.
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore")
your_code()
Obviously, I don't recommend ignoring the warning unless you're sure its harmless, but if you're going to do it this way will only suppress warnings inside the context manager and is straight out of the python docs
UserWarning: Persisting input arguments took 0.58s to run.
If this happens often in your code, it can cause performance problems
(results will be correct in all cases).
The reason for this is probably some large input arguments for a wrapped function (e.g. large strings).
THIS IS A JOBLIB ISSUE. If you can, kindly provide the joblib's team with an example so that they can fix the problem.
the warning itself is self explanatory in my humble opinion. it might be in your code issue you can try to decrease the input size,or you can share your report with joblib team so that they can either help to improve joblib or suggest your better approach of usage to avoid this type of performance warnings.
The warning occurs because joblib is trying to persist the input arguments to disk for caching purposes, but it's taking too long for this operation. The cause of the issue could be due to some large input arguments, such as a long string, which is taking time to serialize.
To resolve the issue, you can either disable the persist argument of the cache method, which would result in no caching, or you can try to preprocess the input arguments to reduce their size before calling the cache method.
#memory.cache(persist=False)
Related
I do not see a way to have the scheduler forget keys that were previously used in an executed dask graph. Minimal example:
client = Client("127.0.0.1:8786")
def f():
raise KeyError()
dsk = {'A': (f,)}
client.get(dsk, 'A') # raises KeyError
If I go back to fix a bug and resubmit the graph:
def f():
return True
dsk = {'A': (f,)}
client.get(dsk, 'A') # still raises KeyError, but:
dsk = {'A1': (f,)}
client.get(dsk, 'A1') # returns True
I understand that this is the correct behavior, since f is already pickled and sent to the scheduler as is with the initial get call. Is there a way that I can have the scheduler forget 'A' before resubmission (without full restart)?
It looks like there are two issues here:
A future exists
The scheduler does clear out keys pretty aggressively once no future points to them. In this case it looks like the KeyError and traceback raised in the get call maintain a reference to the temporary future, which is unfortunate, given that it also maintains a circular reference back to them (I'd consider this a bug). If you were able to clean those up then things would probably be fine.
Cancellation policy doesn't forget exceptions
I was going to suggest that you can explicitly clear out state creating and cancelling a future
from dask.distributed import Future
Future('A').cancel()
Unfortunately for you it looks like current policy is to remember exceptions even past cancellation. You might consider raising an issue for this
Or just use delayed
It's rare to see people use custom graphs unless they are integrating with some other graph scheduling system. Most users use dask.delayed which handles all of these issue for you. In this case dask.delayed would construct a new key for you, as you have done in your example with A1, but it would all be hidden behind a comfortable usability layer.
I am trying to speed up a python function using numba, however I cannot seem to make it compile.
The input for my function is a 27x4 array of type np.int32.
My function is:
#nb.jit(nopython=True)
def edge_profile(input):
pos = input[:,:3]
val = input[:,3]
centre = np.mean(pos,axis=0).astype(np.int32)
diff = np.absolute(pos-centre).sum(axis=1)
cell_edge = np.zeros(3)
for i in range(3):
idx = np.where(diff==i+1)[0]
idy = np.where(val[idx]==1)[0]
cell_edge[i] = len(idy)
return cell_edge.astype(np.int32)
However this produces an extremely large error message which I have unable to use to diagnose the problem. I have tried specifying the input types as follows:
#nb.jit(nb.int32[:](nb.int32[:,:]))
def ...
however this produces an equally large error message.
I fell that I am probably using some function/feature that is not supported in numba, but I do not know enough about it to identify the problem. Any help would be greatly appreciated.
Numba should work ok so long as you stick to basic lists and arrays in the function you want to speed up. It appears that you are already using functions from numpy that are probably already well optimized. So its unlikely you will see a speed up even if you did get it to work. You haven't mentioned what your OS is. Under ubuntu 14.04 you can get it to work through some steps outlined here.
I run "ipython debugf.py" and it gave me error message as below
IndexError Traceback (most recent call last)
/home/ml/debugf.py in <module>()
8 fff = theano.function(inputs=[index],
9 outputs=cost,
---> 10 givens={x: train_set_x[index: index+1]})
IndexError: failed to coerce slice entry of type TensorVariable to integer"
I search the forum and no luck, is there someone can help ?
thanks!
debugf.py :
import theano.tensor as T
import theano
import numpy
index =T.lscalar()
x=T.dmatrix()
cost=x +index
train_set_x=numpy.arange(100).reshape([20,5])
fff=theano.function(inputs=[index],
outputs=cost,
givens={x:train_set_x[index: index+1]}) #<--- Error here
Change train_set_x variable to theano.shared variable, and the code is OK.
I dont know the reason, but it works! Hope this post can help others.
The correct code is as below
import theano.tensor as T
import theano
import numpy
index =T.lscalar()
x=T.dmatrix()
cost=x +index
train_set_x=numpy.arange(100.).reshape([20,5]) #<--- change to float,
#because shared must be floatX type
#change to shared variable
shared_x = theano.shared(train_set_x)
fff=theano.function(inputs=[index],
outputs=cost,
givens={x:shared_x[index: index+1]}) #<----change to shared_x
The reason this occurs is because index is a tensor symbolic variable (a long scalar, as you can see on line 4). So when python tries to build the dictionary that theano needs for its 'given' input, it tries to slice the numpy array using the symbolic variable – which it obviously can't do because it doesn't have a value yet (it is only set when you input something to the function).
As you've realised passing the data through theano.shared is the best approach. This means all the training data can be offloaded to the GPU, and then sliced/indexed on the fly to run each example.
However you might find that you have too much training data to fit in your GPU's memory, or for some other reason don't want to use a shared variable. Then you could just change your function definition
data = T.matrix()
fff=theano.function(inputs=[data],
outputs=cost,
givens={x: data}
)
Then instead of writing
fff(index)
You write
fff(train_set_x[index: index+1])
Be warned the process of moving data onto the GPU is slow, so it's much better to minimise the number of transfers if possible.
I have a piece of code that is supposed to calculate a simple
matrix product, in python (using theano). The matrix that I intend to multiply with is a shared variable.
The example is the smallest example that demonstrates my problem.
I have made use of two helper-functions. floatX converts its input to something of type theano.config.floatX
init_weights generates a random matrix (in type floatX), of given dimensions.
The last line causes the code to crash. In fact, this forces so much output on the commandline that I can't even scroll to the top of it anymore.
So, can anyone tell me what I'm doing wrong?
def floatX(x):
return numpy.asarray(x,dtype=theano.config.floatX)
def init_weights(shape):
return floatX(numpy.random.randn(*shape))
a = init_weights([3,3])
b = theano.shared(value=a,name="b")
x = T.matrix()
y = T.dot(x,b)
f = theano.function([x],y)
This work for me. So my guess is that you have a problem with your blas installation. Make sure to use Theano development version:
http://deeplearning.net/software/theano/install.html#bleeding-edge-install-instructions
It have better default for some configuration. If that do not fix the problem, look at the error message. There is main part that is after the code dump. After the stack trace. This is what is the most useful normally.
You can disable direct linking by Theano to blas with this Theano flag: blas.ldflags=
This can cause slowdown. But it is a quick check to confirm the problem is blas.
If you want more help, dump the error message to a text file and put it on the web and link to it from here.
Alright, so a couple days ago I decided to try and write a primitive wrapper for the PARI library. Ever since then I've been playing with ctypes library in loading the dll and accessing the functions contained using code similar to the following:
from ctypes import *
libcyg=CDLL("<path/cygwin1.dll") #It needs cygwin to be loaded. Not sure why.
pari=CDLL("<path>/libpari-gmp-2.4.dll")
print pari.fibo #fibonacci function
#prints something like "<_FuncPtr object at 0x00BA5828>"
So the functions are there and they can potentially be accessed, but I always receive an access violation no matter what I try. For example:
pari.fibo(5) #access violation
pari.fibo(c_int(5)) #access violation
pari.fibo.argtypes = [c_long] #setting arguments manually
pari.fibo.restype = long #set the return type
pari.fibo(byref(c_int(5))) #access violation reading 0x04 consistently
and any variation on that, including setting argtypes to receive pointers.
The Pari .dll is written in C and the fibonacci function's syntax within the library is GEN fibo(long x).
Could it be the return type that's causing these errors, as it is not a standard int or long but a GEN type, which is unique to the PARI library? Any help would be appreciated. If anyone is able to successfully load the library and use ANY function from within python, please tell; I've been at this for hours now.
EDIT: Seems as though I was simply forgetting to initialize the library. After a quick pari.pari_init(4000000,500000) it stopped erroring. Now my problem lies in the in the fact that it returns a GEN object; which is fine, but whenever I try to reference the address to which it points, it's always 33554435, which I presume is still an address. I'm trying further commands and I'll update if I succeed in getting the correct value of something.
You have two problems here, one give fibo the correct return type and two convert the GEN return type to the value you are looking for.
Poking around the source code a bit, you'll find that GEN is defined as a pointer to a long. Also, at looks like the library provides some converting/printing GENs. I focused in on GENtostr since it would probably be safer for all the pari functions.
import cytpes
pari = ctypes.CDLL("./libpari.so.2.3.5") #I did this under linux
pari.fibo.restype = ctypes.POINTER(ctypes.c_long)
pari.GENtostr.restype = ctypes.POINTER(ctypes.c_char)
pari.pari_init(4000000,500000)
x = pari.fibo(100)
y = pari.GENtostr(x)
ctypes.string_at(y)
Results in:
'354224848179261915075'