I am trying to complete some homework in a DeepLearning.ai course assignment.
When I try the assignment in Coursera platform everything works fine, however, when I try to do the same imports on my local machine it gives me an error,
ModuleNotFoundError: No module named 'lr_utils'
I have tried resolving the issue by installing lr_utils but to no avail.
There is no mention of this module online, and now I started to wonder if that's a proprietary to deeplearning.ai?
Or can we can resolve this issue in any other way?
You will be able to find the lr_utils.py and all the other .py files (and thus the code inside them) required by the assignments:
Go to the first assignment (ie. Python Basics with numpy) - which you can always access whether you are a paid user or not
And then click on 'Open' button in the Menu bar above. (see the image below)
.
Then you can include the code of the modules directly in your code.
As per the answer above, lr_utils is a part of the deep learning course and is a utility to download the data sets. It should readily work with the paid version of the course but in case you 'lost' access to it, I noticed this github project has the lr_utils.py as well as some data sets
https://github.com/andersy005/deep-learning-specialization-coursera/tree/master/01-Neural-Networks-and-Deep-Learning/week2/Programming-Assignments
Note:
The chinese website links did not work when I looked at them. Maybe the server storing the files expired. I did see that this github project had some datasets though as well as the lr_utils file.
EDIT: The link no longer seems to work. Maybe this one will do?
https://github.com/knazeri/coursera/blob/master/deep-learning/1-neural-networks-and-deep-learning/2-logistic-regression-as-a-neural-network/lr_utils.py
Download the datasets from the answer above.
And use this code (It's better than the above since it closes the files after usage):
def load_dataset():
with h5py.File('datasets/train_catvnoncat.h5', "r") as train_dataset:
train_set_x_orig = np.array(train_dataset["train_set_x"][:])
train_set_y_orig = np.array(train_dataset["train_set_y"][:])
with h5py.File('datasets/test_catvnoncat.h5', "r") as test_dataset:
test_set_x_orig = np.array(test_dataset["test_set_x"][:])
test_set_y_orig = np.array(test_dataset["test_set_y"][:])
classes = np.array(test_dataset["list_classes"][:])
train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
"lr_utils" is not official library or something like that.
Purpose of "lr_utils" is to fetch the dataset that is required for course.
option (didn't work for me): go to this page and there is a python code for downloading dataset and creating "lr_utils"
I had a problem with fetching data from provided url (but at least you can try to run it, maybe it will work)
option (worked for me): in the comments (at the same page 1) there are links for manually downloading dataset and "lr_utils.py", so here they are:
link for dataset download
link for lr_utils.py script download
Remember to extract dataset when you download it and you have to put dataset folder and "lr_utils.py" in the same folder as your python script that is using it (script with this line "import lr_utils").
The way I fixed this problem was by:
clicking File -> Open -> You will see the lr_utils.py file ( it does not matter whether you have paid/free version of the course).
opening the lr_utils.py file in Jupyter Notebooks and clicking File -> Download ( store it in your own folder ), rerun importing the modules. It will work like magic.
I did the same process for the datasets folder.
You can download train and test dataset directly here: https://github.com/berkayalan/Deep-Learning/tree/master/datasets
And you need to add this code to the beginning:
import numpy as np
import h5py
import os
def load_dataset():
train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")
train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features
train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels
test_dataset = h5py.File('datasets/test_catvnoncat.h5', "r")
test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features
test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels
classes = np.array(test_dataset["list_classes"][:]) # the list of classes
train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
I faced similar problem and I had followed the following steps:
1. import the following library
import numpy as np
import matplotlib.pyplot as plt
import h5py
import scipy
from PIL import Image
from scipy import ndimage
2. download the train_catvnoncat.h5 and test_catvnoncat.h5 from any of the below link:
[https://github.com/berkayalan/Neural-Networks-and-Deep-Learning/tree/master/datasets]
or
[https://github.com/JudasDie/deeplearning.ai/tree/master/Improving%20Deep%20Neural%20Networks/Week1/Regularization/datasets]
3. create a folder named datasets and paste these two files in this folder.
[ Note: datasets folder and your source code file should be in same directory]
4. run the following code
def load_dataset():
with h5py.File('datasets1/train_catvnoncat.h5', "r") as train_dataset:
train_set_x_orig = np.array(train_dataset["train_set_x"][:])
train_set_y_orig = np.array(train_dataset["train_set_y"][:])
with h5py.File('datasets1/test_catvnoncat.h5', "r") as test_dataset:
test_set_x_orig = np.array(test_dataset["test_set_x"][:])
test_set_y_orig = np.array(test_dataset["test_set_y"][:])
classes = np.array(test_dataset["list_classes"][:])
train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
5. Load the data:
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()
check datasets
print(len(train_set_x_orig))
print(len(test_set_x_orig))
your data set is ready, you may check the len of the train_set_x_orig, train_set_y variable. For mine, it was 209 and 50
I could download the dataset directly from coursera page.
Once you open the Coursera notebook you go to File -> Open and the following window will be display:
enter image description here
Here the notebooks and datasets are displayed, you can go to the datasets folder and download the required data for the assignment. The package lr_utils.py is also available for downloading.
below is your code, just save your file named "lr_utils.py" and now you can use it.
import numpy as np
import h5py
def load_dataset():
train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")
train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features
train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels
test_dataset = h5py.File('datasets/test_catvnoncat.h5', "r")
test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features
test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels
classes = np.array(test_dataset["list_classes"][:]) # the list of classes
train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
if your code file can not find you newly created lr_utils.py file just write this code:
import sys
sys.path.append("full path of the directory where you saved Ir_utils.py file")
Here is the way to get dataset from as #ThinkBonobo:
https://github.com/andersy005/deep-learning-specialization-coursera/tree/master/01-Neural-Networks-and-Deep-Learning/week2/Programming-Assignments/datasets
write a lr_utils.py file, as above answer #StationaryTraveller, put it into any of sys.path() directory.
def load_dataset():
with h5py.File('datasets/train_catvnoncat.h5', "r") as train_dataset:
....
!!! BUT make sure that you delete 'datasets/', cuz now the name of your data file is train_catvnoncat.h5
restart kernel and good luck.
I may add to the answers that you can save the file with lr_utils script on the disc and import that as a module using importlib util function in the following way.
The below code came from the general thread about import functions from external files into the current user session:
How to import a module given the full path?
### Source load_dataset() function from a file
# Specify a name (I think it can be whatever) and path to the lr_utils.py script locally on your PC:
util_script = importlib.util.spec_from_file_location("utils function", "D:/analytics/Deep_Learning_AI/functions/lr_utils.py")
# Make a module
load_utils = importlib.util.module_from_spec(util_script)
# Execute it on the fly
util_script.loader.exec_module(load_utils)
# Load your function
load_utils.load_dataset()
# Then you can use your load_dataset() coming from above specified 'module' called load_utils
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_utils.load_dataset()
# This could be a general way of calling different user specified modules so I did the same for the rest of the neural network function and put them into separate file to keep my script clean.
# Just remember that Python treat it like a module so you need to prefix the function name with a 'module' name eg.:
# d = nnet_utils.model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1000, learning_rate = 0.005, print_cost = True)
nnet_script = importlib.util.spec_from_file_location("utils function", "D:/analytics/Deep_Learning_AI/functions/lr_nnet.py")
nnet_utils = importlib.util.module_from_spec(nnet_script)
nnet_script.loader.exec_module(nnet_utils)
That was the most convenient way for me to source functions/methods from different files in Python so far.
I am coming from the R background where you can call just one line function source() to bring external scripts contents into your current session.
The above answers didn't help, some links had expired.
So, lr_utils is not a pip library but a file in the same notebook as the CourseEra website.
You can click on "Open", and it'll open the explorer where you can download everything that you would want to run in another environment.
(I used this on a browser.)
This is how i solved mine, i copied the lir_utils file and paste it in my notebook thereafter i downloaded the dataset by zipping the file and extracting it. With the following code. Note: Run the code on coursera notebook and select only the zipped file in the directory to download.
!pip install zipfile36
zf = zipfile.ZipFile('datasets/train_catvnoncat_h5.zip', mode='w')
try:
zf.write('datasets/train_catvnoncat.h5')
zf.write('datasets/test_catvnoncat.h5')
finally:
zf.close()
Related
I am trying to register a data set as a Python step with the Azure Machine Learning Studio designer. Here is my code:
import pandas as pd
from azureml.core import Workspace, Run, Dataset
def azureml_main(dataframe1 = None, dataframe2 = None):
run = Run.get_context()
ws = run. experiment.workspace
ds = Dataset.from_pandas_dataframe(dataframe1)
ds.register(workspace = ws,
name = "data set name",
description = "example description",
create_new_version = True)
return dataframe1,
I get an error saying that "create_new_version" in the ds.register line was an unexpected keyword argument. However, this keyword appears in the documentation and I need it to keep track of new versions of the file.
If I remove the argument, I get a different error: "Local data source path not supported for this operation", so it still does not work. Any help is appreciated. Thanks!
update
sharing OP's solution here for easier discovery
import pandas as pd
from azureml.core import Workspace, Run, Dataset
def azureml_main(dataframe1 = None, dataframe2 = None):
run = Run.get_context()
ws = run. experiment.workspace
datastore = ws.get_default_datastore()
ds = Dataset.Tabular.register_pandas_dataframe(
dataframe1, datastore, 'data_set_name',
description = 'data set description.')
return dataframe1,
original answer
Sorry you're struggling. You're very close!
A few things may be the culprit here.
It looks like you're using the Dataset class, which has been deprecated. I recommend trying Dataset.Tabular.register_pandas_dataframe() (docs link) instead of Dataset.from_pandas_dataframe(). (more about the Dataset API deprecation)
More conjectire here, but another thing is there might be some limitations to using dataset registration within an "Execute Python Script" (EPS) module due to:
the workspace object might not have the right permissions
you might not be able to use the register_pandas_dataframe method inside the EPS module, but might have better luck with save the dataframe first to parquet, then calling Dataset.Tabular.from_parquet_files
Hopefully something works here!
I am trying to follow along with a project written by Mike Smales - "Sound Classification using Deep Learning". In there, the author wrote a helper file called wavfilehelper.py:
wavehelper.py Code
import struct
class WavFileHelper():
def read_file_properties(self, filename):
wave_file = open(filename,"rb")
riff = wave_file.read(12)
fmt = wave_file.read(36)
num_channels_string = fmt[10:12]
num_channels = struct.unpack('<H', num_channels_string)[0]
sample_rate_string = fmt[12:16]
sample_rate = struct.unpack("<I",sample_rate_string)[0]
bit_depth_string = fmt[22:24]
bit_depth = struct.unpack("<H",bit_depth_string)[0]
return (num_channels, sample_rate, bit_depth)
In his main program he calls the helper file like this:
from helpers.wavfilehelper import WavFileHelper
wavfilehelper = WavFileHelper()
However, when I run this block of code in PyCharm, it complains "ModuleNotFoundError: No module named 'helpers.wavfilehelper'"...how can I get this helper file to work in the PyCharm environment? Do I have to put the wavehelper.py file in a special folder to be called?
Any help will be greatly appreciated!
It is important to look at (and quote in your question) the actual error messages! In this case, which line is in-error? It is not the instantiation line, but the import - Python is unable to find the module on your machine (using its system paths).
Earlier in the article, the author talks about downloading his files from GitHub (to your machine). Did you follow that step?
Web.Ref: further information about solving this error
I have tried this but it's not working for me. I am using this Git repo. I am building a desktop app and don't want users to download model. I want to ship models with build. I know transformers library looks for models in cache/torch/transformers. If it's not there then download it. I also know you can pass cache_dir parameter in pre_trained.
I am trying this.
cache = os.path.join(os.path.abspath(os.getcwd()), 'Transformation/Annotators/New Sentiment Analysis/transformers')
os.environ['TRANSFORMERS_CACHE'] = cache
if args.model_name_or_path is None:
args.model_name_or_path = 'barissayil/bert-sentiment-analysis-sst'
#Configuration for the desired transformer model
config = AutoConfig.from_pretrained(args.model_name_or_path, cache_dir=cache)
I have tried the solution in above mentioned question and tried cache_dir also. The transformer folder is in same directory with analyze.py. The whole repo and transformer folder is in New Sentiment Analysis directory.
You actually haven't showed the code which is not working but I assume you did something like the following:
from transformers import AutoConfig
import os
os.environ['TRANSFORMERS_CACHE'] = '/blabla/cache/'
config = AutoConfig.from_pretrained('barissayil/bert-sentiment-analysis-sst')
os.path.isdir('/blabla/cache/')
Output:
False
This will not create a new default location for caching, because you imported the transformers library before you have set the environment variable (I modified your linked question to make it more clear). The proper way to modify the default caching directory is setting the environment variable before importing the transformers library:
import os
os.environ['TRANSFORMERS_CACHE'] = '/blabla/cache/'
from transformers import AutoConfig
config = AutoConfig.from_pretrained('barissayil/bert-sentiment-analysis-sst')
os.path.isdir('/blabla/cache/')
Output:
True
I have followed the fastai documentation and the videos on how to create a ML model that can detect the different home care products like soaps and deodorants and so on.
I have now come to where I have the model that supposedly works with an error rate of 0.03...
to my understanding its about a 97% accurate model, however I have no idea on how to predict on other images on another machine. I have exported it using the "learn.export('Home_Care_Model.pkl')" as said in the documentation, with no luck.
Now in the documentation is states that I would need to define the model again with a classes and training set and so on again but now I'm on another computer so i don't have those files on it and I can't go through running it on the web as it is supposed to be run as a python script on any desktop (end goal).
What I'm going towards is where I have one file with unsorted images that then when i run the model on will separate the images into two different folders according to the prediction.
I have been searching for an answer to this and to be honest i'm not sure if i am just not understanding it well enough or something as i have come up empty with every attempt i make to get this model working.
Here is my training code:
from fastai import *
from fastai.vision import *
%matplotlib inline
%reload_ext autoreload
%autoreload 2
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
path = Path('my files path...')
print(path)
for folder in ('soap','deo'): # I have more but it will waist space.
print (folder)
verify_images(path/folder, max_size=500)
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train='.', valid_pct=0.3, ds_tfms=get_transforms(),
size=224, num_workers=4).normalize(imagenet_stats)
data.classes
from fastai.metrics import error_rate
learn= create_cnn(data, models.resnet34, metrics=error_rate)
learn
defaults.device = torch.device('cuda')
learn.fit_one_cycle(5)
learn.unfreeze()
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(4, max_lr=slice(3e-6,3e-5))
learn.save('day2Test_02')
from fastai.widgets import *
ds, idxs = DatasetFormatter().from_toplosses(learn)
ImageCleaner(ds, idxs, path)
df = pd.read_csv(path/'cleaned.csv', header='infer')
df.head()
df[(df['name'].apply(lambda x: len(x)<5))]
np.random.seed(42)
db =(ImageList.from_df(df,path).random_split_by_pct(0.2).label_from_df().transform(get_transforms(), size= 224).databunch(bs=8)).normalize(imagenet_stats)
data.classes, data.c, len(data.train_ds), len(data.valid_ds)
db.classes, db.c,len(db.train_ds), len(data.valid_ds)
learn.data = db
learn.freeze()
learn.fit_one_cycle(4)
learn.save('day2Test_02_01')
learn.unfreeze()
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(4, max_lr=slice(3e-5,3e-4))
learn.save('dat2Test_test2')
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
learn.export('day2Test_test2.pkl')
This is the code i'm trying to run on an virtual environment in python3.6. I use ubuntu newest version 17.10, I run the code as python3 gather_annotations.py
import numpy as np
import cv2
import argparse
from imutils.paths import list_images
from selectors import BoxSelector
#parse arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d","--dataset",required=True,help="path to images dataset...")
ap.add_argument("-a","--annotations",required=True,help="path to save annotations...")
ap.add_argument("-i","--images",required=True,help="path to save images")
args = vars(ap.parse_args())
#annotations and image paths
annotations = []
imPaths = []
#loop through each image and collect annotations
for imagePath in list_images(args["dataset"]):
#load image and create a BoxSelector instance
image = cv2.imread(imagePath)
bs = BoxSelector(image,"Image")
cv2.imshow("Image",image)
cv2.waitKey(0)
#order the points suitable for the Object detector
pt1,pt2 = bs.roiPts
(x,y,xb,yb) = [pt1[0],pt1[1],pt2[0],pt2[1]]
annotations.append([int(x),int(y),int(xb),int(yb)])
imPaths.append(imagePath)
#save annotations and image paths to disk
annotations = np.array(annotations)
imPaths = np.array(imPaths,dtype="unicode")
np.save(args["annotations"],annotations)
np.save(args["images"],imPaths)
And I get the following errors
I have this Folder named '2' where I have all the scripts and other folder named selectors where there is 2 scripts init and box_selector
2(folder)
----selectors/
------------_ init _.py
------------box_selector.py
----detector.py
----gather_annotations.py
----test.py
----train.py
How can I fix that, in the post where I got the code from says something about 'relative imports' but i couln't fix it, thank you.
You need to use . notation to access a file inside a folder..
so
from folder.python_file import ClassOrMethod
in your case
from selectors.box_selector import BoxSelector
Having__init__.py in the selectors folder is crucial to making this work.
You can as many folders as you like and can access as follows but each folder has to contain an __init__.py to work
from folder.folder1.folder2.python_file import ClassOrMethod
One possible area of confusion is that there is a different python library called "selectors" which is DIFFERENT from the selectors of this code example.
https://docs.python.org/3/library/selectors.html
I ended up rename "selectors" (including the directory) of this example to "boxselectors"
This example is from http://www.hackevolve.com/create-your-own-object-detector/