Develop Reference

AttributeError: Can't get attribute 'tokenizer' on <module '__main__'>

I trained a logistic regression model on textual data and saved the model using pickle. But for testing when I try to load the model I got the error mentioned in the title while executing the following line:
model = pickle.load(open("sentiment.model", "rb"))
Following is the code used for saving the model:
import pickle
print("[INFO] saving Model...")
f = open('sentiment.model', "wb")
# first I saved the best_estimator_
f.write(pickle.dumps(gs_lr_tfidf.best_estimator_))
# but again I saved the model completely without mentioning any attribute i.e:
# f.write(pickle.dumps(gs_lr_tfidf))
# but none of them helped and I got the same error
f.close()
print("[INFO] Model saved!")
This error doesn't show up when I load the model in the same notebook just after finishing the training process (in the same runtime). But this error occurs when I try to load the model separately in different runtime even if the model loader code is the same. Why this is happening?

I think the problems is from the behaviour of pickle, as what #hafiz031 said, it's normal when run the same code in the file. So short answer is you need to import tokenizer(from whatever lib you use) before you load the model
For people who know chinese, you can go to this CSDN link for more info.
For people who don't know chinese, sorry for my bad English and I'll try my best to explain.
The documentation says:
pickle.loads(data, /, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)
Return the reconstituted object hierarchy of the pickled representation data of an object. data must be a bytes-like object.
There is an implicit requirement if you use pickle.loads, the object hierarchy must be declared before you load it. Intuitively you can think as you bring USD to north pole and you want to exchange USD to fish with a penguin. As they don't have the concept what is money, they won't make the deal. Same as pickle, if you haven't import tokenizer before, after pickle loads the bytes back to tokenizer, they don't know what is 'tokenizer' and return error to you. Thats why your code works in training file but fail when you loads the model in a different file.
in my case, I just import an extra lib.
# import your own lib
import pickle
import nltk.tokenizer
import genism
import sklearn
#...
model = pickle.load(open("sentiment.model", "rb"))
#model.predict()

Problem with loading language_model_learner fastai

I have problem with fastai library. My code below:
import fastai
from fastai.text import *
import os
import pandas as pd
import fastai
from fastai import *
lab = df.columns[0]
data_lm = TextLMDataBunch.from_csv(r'/AWD', 'data.csv', label_cols = lab, text_cols = ['text'])
data_clas = TextClasDataBunch.from_csv(r'/AWD', 'data.csv', vocab = data_lm.train_ds.vocab, bs = 256,label_cols = lab, text_cols=['text'])
data_lm.save('data_lm_export.pkl')
data_clas.save('data_clas.pkl')
learn = language_model_learner(data_lm,AWD_LSTM,drop_mult = 0.3)
learn.lr_find()
learn.recorder.plot(skip_end=10)
learn.fit_one_cycle(10,1e-2,moms=(0.8,0.7))
learn.save('fit_head')
learn.load('fit_head')
My data is quite big, so each epoch in fit_one_cycle lasts about 6h. My resources enables me only to train model in SLURM JOB 70h, so my whole script will be cancelled. I wanted to divide my script into pieces and the first longest part has to learn and save fit_head. Everything was ok, and after that I wanted to load my model to train it again, but i got this error:
**RuntimeError: Error(s) in loading state_dict for SequentialRNN:
size mismatch for 0.encoder.weight: copying a param with shape torch.Size([54376, 400]) from checkpoint, the shape in current model is torch.Size([54720, 400]).
**
I have checked similar problems on github/stack posts and I tried those solutions like this below, but i cannot find anything usefull.
data_clas.vocab.stoi = data_lm.vocab.stoi
data_clas.vocab.itos = data_lm.vocab.itos
Is there any possibility to load trained model without having this issue ?

When you do learner.save() only the model weights are saved on your disk and not the model state dict which contains the model architecture information.
To train the model in a different session you must first define the model itself. Remember to use the same code to define your new model. Since your data is quite heavy as you mentioned you can use a very small subset (~16 records) of your data to create this new model and then do learn.load(model_path) and you should be able to resume training.
you can modify the training data with learn.data.train_dl = new_dl

Why CNN running in python is extremely slow in comparison to Matlab?

I have trained a CNN in Matlab 2019b that classifies images between three classes. When this CNN was tested in Matlab it was functioning fine and only took 10-15 seconds to classify an image. I used the exportONNXNetwork function in Maltab so that I can implement my CNN in Tensorflow. This is the code I am using to use the ONNX file in python:
import onnx
from onnx_tf.backend import prepare
import numpy as np
from PIL import Image
onnx_model = onnx.load('trainednet.onnx')
tf_rep = prepare(onnx_model)
filepath = 'filepath.png'
img = Image.open(filepath).resize((224,224)).convert("RGB")
img = array(img).transpose((2,0,1))
img = np.expand_dims(img, 0)
img = img.astype(np.uint8)
probabilities = tf_rep.run(img)
print(probabilities)
When trying to use this code to classify the same test set, it seems to be classifying the images correctly but it is very slow and freezes my computer as it reaches high memory usages of up to 95+% at some points.
I also noticed in the command prompt while classifying it prints this:
2020-04-18 18:26:39.214286: W tensorflow/core/grappler/optimizers/meta_optimizer.cc:530] constant_folding failed: Deadline exceeded: constant_folding exceeded deadline., time = 486776.938ms.
Is there any way I can make this python code classify faster?

Maybe you could try to understand what part of the code takes a long time this way:
import onnx
from onnx_tf.backend import prepare
import numpy as np
from PIL import Image
import datetime
now = datetime.datetime.now()
onnx_model = onnx.load('trainednet.onnx')
tf_rep = prepare(onnx_model)
filepath = 'filepath.png'
later = datetime.datetime.now()
difference = later - now
print("Loading time : %f ms" % (difference.microseconds / 1000))
img = Image.open(filepath).resize((224,224)).convert("RGB")
img = array(img).transpose((2,0,1))
img = np.expand_dims(img, 0)
img = img.astype(np.uint8)
now = datetime.datetime.now()
probabilities = tf_rep.run(img)
later = datetime.datetime.now()
difference = later - now
print("Prediction time : %f ms" % (difference.microseconds / 1000))
print(probabilities)
Let me know what the output looks like :)

In this case, it appears that the Grapper optimization suite has encountered some kind of infinite loop or memory leak. I would recommend filing an issue against the Github repo.
It's challenging to debug why constant folding is taking so long, but you may have better performance using the ONNX TensorRT backend as compared to the TensorFlow backend. It achieves better performance as compared to the TensorFlow backend on Nvidia GPUs while compiling typical graphs more quickly. Constant folding usually doesn't provide large speedups for well optimized models.
import onnx
import onnx_tensorrt.backend as backend
import numpy as np
model = onnx.load("trainednet.onnx'")
engine = backend.prepare(model, device='CUDA:1')
filepath = 'filepath.png'
img = Image.open(filepath).resize((224,224)).convert("RGB")
img = array(img).transpose((2,0,1))
img = np.expand_dims(img, 0)
img = img.astype(np.uint8)
output_data = engine.run(img)[0]
print(output_data)

You should consider some points while working on TensorFlow with Python. A GPU will be better for work as it fastens the whole processing. For that, you have to install CUDA support. Apart from this, the compiler also sometimes matters. I can tell VSCode is better than Spyder from my experience.
I hope it helps.

Since the command prompt states that your program takes a long time to perform constant folding, it might be worthwhile to turn this off. Based on this documentation, you could try running:
import numpy as np
import timeit
import traceback
import contextlib
import onnx
from onnx_tf.backend import prepare
from PIL import Image
import tensorflow as tf
#contextlib.contextmanager
def options(options):
old_opts = tf.config.optimizer.get_experimental_options()
tf.config.optimizer.set_experimental_options(options)
try:
yield
finally:
tf.config.optimizer.set_experimental_options(old_opts)
with options({'constant_folding': False}):
onnx_model = onnx.load('trainednet.onnx')
tf_rep - prepare(onnx_model)
filepath = 'filepath.png'
img = Image.open(filepath).resize((224,224)).convert("RGB")
img = array(img).transpose((2,0,1))
img = np.expand_dims(img, 0)
img = img.astype(np.uint8)
probabilities = tf_rep.run(img)
print(probabilities)
This disables the constant folding performed in the TensorFlow Graph optimization. This can work both ways: on the one hand it will not reach the constant folding deadline, but on the other hand disabling constant folding can result in significant runtime increases. Anyway it is worth trying, good luck!

How to deploy my .pkl machine learnt model made with fastai on my other machine with python on the desktop

I have followed the fastai documentation and the videos on how to create a ML model that can detect the different home care products like soaps and deodorants and so on.
I have now come to where I have the model that supposedly works with an error rate of 0.03...
to my understanding its about a 97% accurate model, however I have no idea on how to predict on other images on another machine. I have exported it using the "learn.export('Home_Care_Model.pkl')" as said in the documentation, with no luck.
Now in the documentation is states that I would need to define the model again with a classes and training set and so on again but now I'm on another computer so i don't have those files on it and I can't go through running it on the web as it is supposed to be run as a python script on any desktop (end goal).
What I'm going towards is where I have one file with unsorted images that then when i run the model on will separate the images into two different folders according to the prediction.
I have been searching for an answer to this and to be honest i'm not sure if i am just not understanding it well enough or something as i have come up empty with every attempt i make to get this model working.
Here is my training code:
from fastai import *
from fastai.vision import *
%matplotlib inline
%reload_ext autoreload
%autoreload 2
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
path = Path('my files path...')
print(path)
for folder in ('soap','deo'): # I have more but it will waist space.
print (folder)
verify_images(path/folder, max_size=500)
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train='.', valid_pct=0.3, ds_tfms=get_transforms(),
size=224, num_workers=4).normalize(imagenet_stats)
data.classes
from fastai.metrics import error_rate
learn= create_cnn(data, models.resnet34, metrics=error_rate)
learn
defaults.device = torch.device('cuda')
learn.fit_one_cycle(5)
learn.unfreeze()
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(4, max_lr=slice(3e-6,3e-5))
learn.save('day2Test_02')
from fastai.widgets import *
ds, idxs = DatasetFormatter().from_toplosses(learn)
ImageCleaner(ds, idxs, path)
df = pd.read_csv(path/'cleaned.csv', header='infer')
df.head()
df[(df['name'].apply(lambda x: len(x)<5))]
np.random.seed(42)
db =(ImageList.from_df(df,path).random_split_by_pct(0.2).label_from_df().transform(get_transforms(), size= 224).databunch(bs=8)).normalize(imagenet_stats)
data.classes, data.c, len(data.train_ds), len(data.valid_ds)
db.classes, db.c,len(db.train_ds), len(data.valid_ds)
learn.data = db
learn.freeze()
learn.fit_one_cycle(4)
learn.save('day2Test_02_01')
learn.unfreeze()
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(4, max_lr=slice(3e-5,3e-4))
learn.save('dat2Test_test2')
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
learn.export('day2Test_test2.pkl')

any existing code/library for image sharpness or blurriness estimation in python?

I would like to find some existing code/library for sharpness/blurriness estimation on normal images. (prefer in Python) I will need to compare the performance of different algorithms later.
I have 10000+ MRI scan images with different "quality"(sharpness/blurriness). I need to write code to filter images with certain "quality"(sharpness/blurriness) which is up to user. Hence, I am trying to research about image sharpness/blurriness estimation on medical images. My supervisor told me there are lots of existing code for sharpness/blurriness estimation on normal images(maybe it is no-reference sharpness metric) on internet. She asked me to search about them and try them on normal images first. Then try to learn about their algorithms.
I have searched about this on internet and found some pages which are relevant. However, lots of them are out of date.
For example:
On
Image sharpness metric
page,
Cumulative probability of blur detection (CPBD) https://ivulab.asu.edu/software/quality/cpbd
seems not working anymore. I guess the reason is that "imread" function is removed from new "scipy" library. (please see later code and error message) I think I can try the old version of "scipy" later. However, I would like to find some more currently available code/library about image sharpness/blurriness estimation.
Also, my working environment will be in Windows 10 or CentOS-7.
I have tried the following code with CPBD:
import sys, cpbd
from scipy import ndimage
input_image1 = ndimage.imread('D:\Work\Project\scripts\test_images\blur1.png', mode='L')
input_image2 = ndimage.imread('D:\Work\Project\scripts\test_images\clr1.png', mode='L')
print("blurry image sharpness:")
cpbd.compute(input_image1)
print("clear image sharpness:")
cpbd.compute(input_image2)
Error message from Python 3.7 shell (ran in Window 10):
Traceback (most recent call last):
File "D:\Work\Project\scripts\try_cpbd.py", line 1, in <module>
import sys, cpbd
File "D:\Program_Files_2\Python\lib\site-packages\cpbd\__init__.py", line 3, in <module>
from .compute import compute
File "D:\Program_Files_2\Python\lib\site-packages\cpbd\compute.py", line 14, in <module>
from scipy.misc import imread #Original: from scipy.ndimage import imread
ImportError: cannot import name 'imread' from 'scipy.misc' (D:\Program_Files_2\Python\lib\site-packages\scipy\misc\__init__.py)

Seems that cpbd package has not been updated from some time.
It worked for me with the following steps:
Edit "D:\Program_Files_2\Python\lib\site-packages\cpbd\compute.py":
Comment the last 4 lines starting with:
#if __name__ == '__main__':
Use the python code:
import cpbd
import cv2
input_image1 = cv2.imread('blur1.png')
if input_image1 is None:
print("error opening image")
exit()
input_image1 = cv2.cvtColor(input_image1, cv2.COLOR_BGR2GRAY)
print("blurry image sharpness:")
cpbd.compute(input_image1)

Since scipy.misc.imread is deprecated since 1.0.0, and removed in 1.2.0, I would use skimage.io.imread instead (which is in most ways a drop-in replacement).
Edit the code in cpbd/compute.py
import skimage.io
input_image1 = skimage.io.imread('blur1.png')
cv2 also works (or other options: imageio, PIL, ...) but skimage tends to be a bit easier to install/use.

The following steps worked for me:
Open the compute.py from C:\ProgramData\Anaconda3\Lib\site-packages\cpbd\compute.py or wherever you have installed it. You will find the following code:
from scipy.ndimage import imread
replace it with:
from skimage.io import imread
If you can't save the compute.py file, then copy it to desktop, edit it in the above mentioned way and replace the file in C:\ProgramData\Anaconda3\Lib\site-packages\cpbd\compute.py with it.

Following the answer from Baj Mile, I did the following and it worked for me.
opened the cpbd\compute.py file
commented the line : from scipy.ndimage import imread
Added the line: import cv2
Made the following changes to the main section:
if __name__ == '__main__':
#input_image = imread(argv[1], mode='L')
input_image=cv2.imread(argv[1])
sharpness = compute(input_image)
print('CPBD sharpness for %s: %f' % (argv[1], sharpness))
close the compute.py file.
In the main code:
import cpbd
import cv2
input_image1 = cv2.imread('testimage.jpg')
input_image1 = cv2.cvtColor(input_image1, cv2.COLOR_BGR2GRAY)
cpbd.compute(input_image1)