Develop Reference

When I tried to load dataset for deep learning project, I got an error like below

When executing this code:
caption_path = 'drive/MyDrive/projects/dl project/dataset/captions',
caption_path
captions = open(caption_path, 'rb').read().decode('utf-8').split('\n')
I got this error:
TypeError Traceback (most recent call last)
in ()
----> 1 captions = open(caption_path, 'rb').read().decode('utf-8').split('\n')
TypeError: expected str, bytes or os.PathLike object, not tuple
How can I solve this?

instead of:
caption_path = 'drive/MyDrive/projects/dl project/dataset/captions', caption_path
do:
caption_path = 'drive/MyDrive/projects/dl project/dataset/captions'

Why am I getting AttributeError: 'ModelMetaNames' object has no attribute 'AUTHOR_NAME' when executing it in Jupyter Notebook on IBM Watson Studio

I am following a tutorial and this step shouldn't give any errors but I stumbled upon this unexpected problem. What's going wrong here? Here is the code block:
model_props = {client.repository.ModelMetaNames.AUTHOR_NAME: "IBM",
client.repository.ModelMetaNames.NAME: "Heart Failure Prediction Model"}
published_model = client.repository.store_model(model=model_rf, pipeline=pipeline_rf,
meta_props=model_props, training_data=train_data)
Error:
AttributeError Traceback (most recent call last)
in ()
----> 1 model_props = {client.repository.ModelMetaNames.AUTHOR_NAME: "IBM",
2 client.repository.ModelMetaNames.NAME: "Heart Failure Prediction Model"}
3 published_model = client.repository.store_model(model=model_rf, pipeline=pipeline_rf, meta_props=model_props, training_data=train_data)
AttributeError: 'ModelMetaNames' object has no attribute 'AUTHOR_NAME'

See this example and you'll understand what is happening:
>>> a = "mykeyname"
>>> {a:"my value"}
{'mykeyname': 'my value'}
So you are trying to use value from client.repository.ModelMetaNames.AUTHOR_NAMEas key in your dictionary. But such attribute doesn't exists in your model.
If you want to set variable in a object or class, you can simply do:
client.repository.ModelMetaNames.AUTHOR_NAME = "IBM"
Or if you want to have dictionary object:
props = {"AUTHOR_NAME": "IBM", "OTHERKEY": "Something else"}

Pytest `pytest.raises(ValueError)` does not seem to detect a `ValueError`

EDIT. The issue was that everytime I would import the function, it would not changed with updates. For this I needed to do
import sys, importlib
importlib.reload(sys.modules['foo'])
from foo import bar
And it started working
I am trying to write a test using Pytest to detect a ValueError if a json file passed into a function is invalid. However, when I follow the example, the test doesn't detect that the ValueError was raised.
This is the function I want to test
import pytest
import json
def read_file(input_file):
try:
with open(input_file, "r", encoding='utf-8') as reader:
pre_input_data = json.load(reader)
except ValueError:
raise ValueError
And this is my test function
def test_read_file():
with pytest.raises(ValueError):
read_file("invalidJsonFile.json")
If I just run the original function, it raises the ValueError
read_file("invalidJsonFile.json")
Invalid json file: Expecting value: line 1 column 1 (char 0)
However, when I run the test, it says it did not get a ValueError
test_read_file()
Invalid json file: Expecting value: line 1 column 1 (char 0)
---------------------------------------------------------------------------
Failed Traceback (most recent call last)
<ipython-input-47-c42b81670a67> in <module>()
----> 1 test_read_file()
2 frames
<ipython-input-46-178e6c645f01> in test_read_file()
1 def test_read_file():
2 with pytest.raises(Exception):
----> 3 read_file("invalidJsonFile.json")
/usr/local/lib/python3.6/dist-packages/_pytest/python_api.py in __exit__(self, *tp)
727 __tracebackhide__ = True
728 if tp[0] is None:
--> 729 fail(self.message)
730 self.excinfo.__init__(tp)
731 suppress_exception = issubclass(self.excinfo.type, self.expected_exception)
/usr/local/lib/python3.6/dist-packages/_pytest/outcomes.py in fail(msg, pytrace)
115 """
116 __tracebackhide__ = True
--> 117 raise Failed(msg=msg, pytrace=pytrace)
118
119
Failed: DID NOT RAISE <class 'Exception'>

Are you sure you're running the same code you sent here? because in a stack trace it looks like you're reading a different file (which could be valid and then no exception will be raised, if it's empty for example).
----> 3 read_file("sampleData.csv")
Also, you do not need to except ValueError just to raise ValueError, when you use pytest.raises(ValueError): pytest will check if the exception is instanceof ValueError.

Python - Scrubadub.clean not working - Cannot properly scrub text PII + HTTP Error 503

I'm sorry and this is probably a basic question but as I am learning scrubadub and trying to get it work myself on Jupyter notebook. It kept showing - HTTP Error 503: Service Unavailable
This is what I input, which is exactly the same as the scrubadub documentation.
text = u"John is a cat"
scrubadub.clean(text, replace_with='placeholder')
u"{{NAME}} is a cat"
And this is the error message I got:
HTTPError Traceback (most recent call last)
<ipython-input-92-5b0754baae94> in <module>()
1 text = u"John is a cat"
----> 2 scrubadub.clean(text, replace_with='placeholder')
3 u"{{NAME}} is a cat"
/anaconda3/lib/python3.7/site-packages/scrubadub/__init__.py in clean(text, cls, **kwargs)
14 cls = cls or Scrubber
15 scrubber = cls()
---> 16 return scrubber.clean(text, **kwargs)
/anaconda3/lib/python3.7/site-packages/scrubadub/scrubbers.py in clean(self, text, **kwargs)
55 clean_chunks = []
56 filth = Filth()
---> 57 for next_filth in self.iter_filth(text):
58 clean_chunks.append(text[filth.end:next_filth.beg])
59 clean_chunks.append(next_filth.replace_with(**kwargs))
I also tried below code here and I get error messages as well, I'm guessing if I missing any arguments in the code...
import scrubadub
class MyFilth(scrubadub.filth.base.Filth):
type = 'mine'
class MyDetector(scrubadub.detectors.base.Detector):
filth_cls = MyFilth
def iter_filth(self, text):
# do something here
pass
scrubber = scrubadub.Scrubber()
scrubber.add_detector(MyDetector)
text = u"My stuff can be found there"
scrubadub.clean(text)
u"{{MINE}} can be found there."
StopIteration Traceback (most recent call last)
/anaconda3/lib/python3.7/site-packages/scrubadub/detectors/base.py in iter_filth(self, text)
21 if self.filth_cls.regex is None:
---> 22 raise StopIteration
23 for match in self.filth_cls.regex.finditer(text):
StopIteration:
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
<ipython-input-94-2cc23d003da7> in <module>()
11
12 text = u"My stuff can be found there"
---> 13 scrubadub.clean(text)
14 u"{{MINE}} can be found there."
/anaconda3/lib/python3.7/site-packages/scrubadub/__init__.py in clean(text, cls, **kwargs)
14 cls = cls or Scrubber
15 scrubber = cls()
---> 16 return scrubber.clean(text, **kwargs)

There is an open issue for the same on github as scrubadub seems to not working very good with python 3.7 which you are currently using.
I was able to reproduce it with 3.7 without a notebook as well. So it is a notebook issue for sure.
As an interim solution, changing your env to 3.6 (or worst case 2.7 highly not recommended) works.
https://github.com/datascopeanalytics/scrubadub/issues/40Stop Iteration issue

Adding method to Python's NoneType

I'm using BeautifulSoup to do some crawling, and want to chain find calls, for example:
soup.find('div', class_="class1").find('div', class_="class2").find('div', class_="class3")
Of course, this breaks whenever one of the divs cannot be found, throwing an
AttributeError: 'NoneType' object has no attribute 'find'
Is there a way to modify NoneType to add a find method such as
class NoneType:
def find(*args):
return None
so that I can do something like
thing = soup.find('div', class_="class1").find('div', class_="class2").find('div', class_="class3")
if thing:
do more stuff
instead of
thing1 = soup.find('div', class_="class1")
if thing1:
thing2 = thing1.find('div', class_="class2")
if thing2:
thing3 = thing2.find('div', class_="class3")
etc.
I think I might be able to do something similar by using a parser with XPath capabilities, but the question is not specific to this use case and is more about modifying/overriding built in classes.

Why not use a try/except statement instead (since you cannot modify NoneType)?
try:
thing = soup.find('div', class_="class1").find('div', class_="class2").find('div', class_="class3")
do more stuff
except AttributeError:
thing = None # if you need to do more with thing

You can't modify builtin class such as NoneType or str:
>>> nt = type(None)
>>> nt.bla = 23
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'NoneType'
For some of them (eg str), you can inherit from:
>>> class bla(str):
... def toto(self): return 1
>>> bla('2123').toto()
1
It's not possible with NoneType. And it won't help you either:
>>> class myNoneType(nt):
... def find(self): return 1
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
type 'NoneType' is not an acceptable base type

You cannot modify the class and the real question is why you would try? NoneType means there is no data there so when you attempt a .find() on that type even if it did exist you would only get null or no values from it. I would reccomend something like this.
try:
var = soup.find('div', class_="class1").find('div', class_="class2").find('div', class_="class3")
except AttributeError:
do something else instead or message saying there was no div

You can't inherit from None:
>>> class Noneish(type(None)):
... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: type 'NoneType' is not an acceptable base type

An approach might be to have a
class FindCaller(object):
def __init__(self, *a, **k):
self.a = a
self.k = k
def __call__(self, obj):
return obj.find(*self.a, **self.k)
def callchain(root, *fcs):
for fc in fcs:
root = fc(root)
if root is None: return
return root
and then do
thing = callchain(soup,
FindCaller('div', class_="class1"),
FindCaller('div', class_="class2"),
FindCaller('div', class_="class3"),
)

You can't. For good reasons...
In fact, NoneType is even less accessible than other built-in types:
type(None).foo = lambda x: x
# ---------------------------------------------------------------------------
# TypeError Traceback (most recent call last)
# <ipython-input-12-61bbde54e51b> in <module>()
# ----> 1 type(None).foo = lambda x: x
# TypeError: can't set attributes of built-in/extension type 'NoneType'
NoneType.foo = lambda x: x
# ---------------------------------------------------------------------------
# NameError Traceback (most recent call last)
# <ipython-input-13-22af1ed98023> in <module>()
# ----> 1 NoneType.foo = lambda x: x
# NameError: name 'NoneType' is not defined
int.foo = lambda x: x
# ---------------------------------------------------------------------------
# TypeError Traceback (most recent call last)
# <ipython-input-14-c46c4e33b8cc> in <module>()
# ----> 1 int.foo = lambda x: x
# TypeError: can't set attributes of built-in/extension type 'int'
As suggested above, use try: ... except AttributeError: clause.