Usually, if you want to get the source of an object, you can get it via the inspect module:
import inspect
inspect.getsource(MyObject)
However, in a Jupyter notebook, this doesn't work:
import inspect
class Foo:
def __init__(self, info):
self.info = info
a = Foo("hi")
inspect.getsource(a)
Throws the error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-14-048b6f0c2e9b> in <module>()
7 a = Foo("hi")
8
----> 9 inspect.getsource(a)
/usr/lib/python3.6/inspect.py in getsource(object)
963 or code object. The source code is returned as a single string. An
964 OSError is raised if the source code cannot be retrieved."""
--> 965 lines, lnum = getsourcelines(object)
966 return ''.join(lines)
967
/usr/lib/python3.6/inspect.py in getsourcelines(object)
950 raised if the source code cannot be retrieved."""
951 object = unwrap(object)
--> 952 lines, lnum = findsource(object)
953
954 if ismodule(object):
/usr/lib/python3.6/inspect.py in findsource(object)
763 is raised if the source code cannot be retrieved."""
764
--> 765 file = getsourcefile(object)
766 if file:
767 # Invalidate cache if needed.
/usr/lib/python3.6/inspect.py in getsourcefile(object)
679 Return None if no way can be identified to get the source.
680 """
--> 681 filename = getfile(object)
682 all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]
683 all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]
/usr/lib/python3.6/inspect.py in getfile(object)
661 return object.co_filename
662 raise TypeError('{!r} is not a module, class, method, '
--> 663 'function, traceback, frame, or code object'.format(object))
664
665 def getmodulename(path):
TypeError: <__main__.Foo object at 0x7fb9130ee518> is not a module, class, method, function, traceback, frame, or code object
If I try to find the source of Foo (using inspect.getsource(Foo)), I get:
TypeError: <module '__main__'> is a built-in class
How do I get the source of a class defined in a Jupyter notebook?
I found a "hacky way" to get the source code of a class in a Jupyter Notebook.
Assume in a cell you have:
class MyClass:
test = 2
def __init__(self):
self.L = 5
def test(self, x):
return True
#classmethod
def forward(cls, x):
return x
Then you can extract the code using:
import inspect
from IPython.core.magics.code import extract_symbols
obj = MyClass
cell_code = "".join(inspect.linecache.getlines(new_getfile(obj)))
class_code = extract_symbols(cell_code, obj.__name__)[0][0]
print(class_code)
with new_getfile defined from here:
import inspect, sys
def new_getfile(object, _old_getfile=inspect.getfile):
if not inspect.isclass(object):
return _old_getfile(object)
# Lookup by parent module (as in current inspect)
if hasattr(object, '__module__'):
object_ = sys.modules.get(object.__module__)
if hasattr(object_, '__file__'):
return object_.__file__
# If parent module is __main__, lookup by methods (NEW)
for name, member in inspect.getmembers(object):
if inspect.isfunction(member) and object.__qualname__ + '.' + member.__name__ == member.__qualname__:
return inspect.getfile(member)
else:
raise TypeError('Source for {!r} not found'.format(object))
inspect.getfile = new_getfile
Using inspect.getsource(inspect.getfile) we can get a segment of code that deals with this:
...
if isclass(object):
if hasattr(object, '__module__'):
object = sys.modules.get(object.__module__)
if hasattr(object, '__file__'):
return object.__file__
raise TypeError('{!r} is a built-in class'.format(object))
...
It seems in ipython or Jupyter notebook, the classes/functions defined or the __main__ module does not have a __file__ attribute associated with them so inspect wasn't able to retrieve a source file. In this case you can define the classes in a separate .py file so inspect is able to retrieve the file associated with it.
Related
I am defining a class:
class MyFirstClass:
pass
After, I am trying to print the definition of MyFirstClass class:
import inspect
print(inspect.getsource(MyFirstClass))
But I am getting error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_22132/2338486789.py in <module>
1 import inspect
----> 2 print(inspect.getsource(MyFirstClass))
C:\ProgramData\Anaconda3\lib\inspect.py in getsource(object)
971 or code object. The source code is returned as a single string. An
972 OSError is raised if the source code cannot be retrieved."""
--> 973 lines, lnum = getsourcelines(object)
974 return ''.join(lines)
975
C:\ProgramData\Anaconda3\lib\inspect.py in getsourcelines(object)
953 raised if the source code cannot be retrieved."""
954 object = unwrap(object)
--> 955 lines, lnum = findsource(object)
956
957 if istraceback(object):
C:\ProgramData\Anaconda3\lib\inspect.py in findsource(object)
766 is raised if the source code cannot be retrieved."""
767
--> 768 file = getsourcefile(object)
769 if file:
770 # Invalidate cache if needed.
C:\ProgramData\Anaconda3\lib\inspect.py in getsourcefile(object)
682 Return None if no way can be identified to get the source.
683 """
--> 684 filename = getfile(object)
685 all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]
686 all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]
C:\ProgramData\Anaconda3\lib\inspect.py in getfile(object)
651 if getattr(module, '__file__', None):
652 return module.__file__
--> 653 raise TypeError('{!r} is a built-in class'.format(object))
654 if ismethod(object):
655 object = object.__func__
TypeError: <class '__main__.MyFirstClass'> is a built-in class
I expected oputput is:
class MyFirstClass:
pass
How to correctly use inspect.getsource()) to get my expected output (definition of MyFirstClass class)?
I was writing some unit test in python and needed to mock a very general collections.abc.Sized, so I set off creating a Mock with a __len__ method whose return value is any int I want.
from unittest.mock import Mock
sized = Mock(**{"__len__.return_value": 8})
Which failed with AttributeErrorgave: __len__ and produced the traceback at the end of the post, while the following works:
from unittest.mock import Mock
sized = Mock()
sized.__len__.return_value = 8
# or
from unittest.mock import MagicMock
sized = MagicMock(**{"__len__.return_value": 8})
Reading down the traceback we see that:
CallableMixin's __init__ is called.
kwargs passed to Mock's __init__ method get passed to Mock.configure_mock()
the kwargs name is getattr()ed on the Mock itself
Mock __getattr__ method raises an AttributeError if attribute name is magic, that is if '__%s__' % name[2:-2] == name.
This seems like a very arbitrary limitation and I wonder if there is a reason for Mock not to accept magic methods in Mock.configure_mock().
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 Mock(**{"__len__.return_value": 0})
File ~/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py:1086, in CallableMixin.__init__(self, spec, side_effect, return_value, wraps, name, spec_set, parent, _spec_state, _new_name, _new_parent, **kwargs)
1082 def __init__(self, spec=None, side_effect=None, return_value=DEFAULT,
1083 wraps=None, name=None, spec_set=None, parent=None,
1084 _spec_state=None, _new_name='', _new_parent=None, **kwargs):
1085 self.__dict__['_mock_return_value'] = return_value
-> 1086 _safe_super(CallableMixin, self).__init__(
1087 spec, wraps, name, spec_set, parent,
1088 _spec_state, _new_name, _new_parent, **kwargs
1089 )
1091 self.side_effect = side_effect
File ~/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py:457, in NonCallableMock.__init__(self, spec, wraps, name, spec_set, parent, _spec_state, _new_name, _new_parent, _spec_as_instance, _eat_self, unsafe, **kwargs)
454 __dict__['_mock_unsafe'] = unsafe
456 if kwargs:
--> 457 self.configure_mock(**kwargs)
459 _safe_super(NonCallableMock, self).__init__(
460 spec, wraps, name, spec_set, parent,
461 _spec_state
462 )
File ~/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py:625, in NonCallableMock.configure_mock(self, **kwargs)
623 obj = self
624 for entry in args:
--> 625 obj = getattr(obj, entry)
626 setattr(obj, final, val)
File ~/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py:636, in NonCallableMock.__getattr__(self, name)
634 raise AttributeError("Mock object has no attribute %r" % name)
635 elif _is_magic(name):
--> 636 raise AttributeError(name)
637 if not self._mock_unsafe:
638 if name.startswith(('assert', 'assret', 'asert', 'aseert', 'assrt')):
AttributeError: __len__
I follow the exact same folder structure for OpenAI Gym but still get attribute not found error upon using gym.make(). My folder structure is as below.
my_gym
-->examples
---MAIN_ENV
-->my_rl_gym
--->envs
---__init__
---main_env #contains a class called Myxyz as registered below
---__init__
-->setup.ipynb
The below is how I register in the __init__ inside envs folder
from gym.envs.registration import register
register(
id='XYZ-v0',
entry_point='my_rl_gym.envs:Myxyz'
)
The below is in the __init__ inside my_rl_gym folder i.e. outside envs
# from my_rl_gym.envs.main_env import Myxyz
#### THIS above line is actually correct BUT gives error as No module named
#####'my_rl_gym.envs.main_env ' . Hence, I changed this to below line.
import main_env
The error comes upon calling the make command
env = gym.make('XYZ-v0', **env_args)
Traceback is:
----> 6 env = gym.make('XYZ-v0', **env_args)
~\anaconda3\lib\site-packages\gym\envs\registration.py in make(id, **kwargs)
674 # fmt: on
675 def make(id: str, **kwargs) -> "Env":
--> 676 return registry.make(id, **kwargs)
677
678
~\anaconda3\lib\site-packages\gym\envs\registration.py in make(self, path, **kwargs)
518 spec = self.spec(path)
519 # Construct the environment
--> 520 return spec.make(**kwargs)
521
522 def all(self):
~\anaconda3\lib\site-packages\gym\envs\registration.py in make(self, **kwargs)
137 env = self.entry_point(**_kwargs)
138 else:
--> 139 cls = load(self.entry_point)
140 env = cls(**_kwargs)
141
~\anaconda3\lib\site-packages\gym\envs\registration.py in load(name)
54 mod_name, attr_name = name.split(":")
55 mod = importlib.import_module(mod_name)
---> 56 fn = getattr(mod, attr_name)
57 return fn
58
AttributeError: module 'my_rl_gym.envs' has no attribute 'Myxyz'
What is the problem here? The folder directories are correct and the way of defining is also correct. For e.g. see https://github.com/MartinThoma/banana-gym
Is it because I need to have any setup file? because if so then I did and got the typeerror as in question Getting Type error with setup file in OpenAI gym
I am facing an issue with the MRJob library in Python and hope you can help me out in how to solve it.
I want to apply a simple MapReduce job and count the number of characters, words and lines of a text file, as shown the the original MRJob documentation (https://mrjob.readthedocs.io/en/latest/guides/quickstart.html).
My code looks as following:
%%writefile MRJob_TermCateg.py
from mrjob.job import MRJob
from mrjob.step import MRStep
import re
WORD_RE = re.compile(r'\w+')
class MRJob_TermCateg(MRJob):
def mapper(self, _, line):
yield "chars", len(line)
yield "words", len(line.split())
yield "lines", 1
def reducer(self, key, values):
yield key, sum(values)
if __name__ == '__main__':
MRJob_TermCateg.run()
I have my text file file.txt that looks as following:
This is a sentence
This is also a sentence
But, when I run the command
%run MRJob_TermCateg.py file.txt
I get the following error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
File ~\Documents\01 Jupyter Workspace\DIC\Ex 1\MRJob_TermCateg.py:20, in <module>
17 yield key, sum(values)
19 if __name__ == '__main__':
---> 20 MRJob_TermCateg.run()
File ~\Anaconda3\lib\site-packages\mrjob\job.py:616, in MRJob.run(cls)
603 """Entry point for running job from the command-line.
604
605 This is also the entry point when a mapper or reducer is run
(...)
613 * Run the entire job. See :py:meth:`run_job`
614 """
615 # load options from the command line
--> 616 cls().execute()
File ~\Anaconda3\lib\site-packages\mrjob\job.py:687, in MRJob.execute(self)
684 self.run_spark(self.options.step_num)
686 else:
--> 687 self.run_job()
File ~\Anaconda3\lib\site-packages\mrjob\job.py:645, in MRJob.run_job(self)
643 if self._should_cat_output():
644 for chunk in runner.cat_output():
--> 645 self.stdout.write(chunk)
646 self.stdout.flush()
File ~\Anaconda3\lib\site-packages\ipykernel\iostream.py:513, in OutStream.write(self, string)
503 """Write to current stream after encoding if necessary
504
505 Returns
(...)
509
510 """
512 if not isinstance(string, str):
--> 513 raise TypeError(
514 f"write() argument must be str, not {type(string)}"
515 )
517 if self.echo is not None:
518 try:
TypeError: write() argument must be str, not <class 'bytes'>
I am unsure what I am doing wrong here, as I use the same code and following the steps as described in the original documentation. Can anyone help me figuring out how to fix this?
Thanks a lot in advance!
I'd like to modify the simpy Store get functionality by altering the object that is returned in _do_get:
class mod_Store(Store):
def _do_get(self, event):
super()._do_get(event)
event.value.tagged = True
env = Environment()
s = mod_Store(env)
class thing:
pass
def putter():
while True:
yield s.put(thing())
yield env.timeout(5)
def getter():
while True:
t = yield s.get()
yield env.timeout(3)
env.process(putter())
env.process(getter())
env.run(until=20)
Basically, before returning the object back to getter, I'd like to add an attribute (tagged). But this code produces the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-50-27e09b52b41a> in getter()
19 while True:
---> 20 t = yield s.get()
21 yield env.timeout(3)
C:\ProgramData\Anaconda3\lib\site-packages\simpy\resources\base.py in __init__(self, resource)
77 self.callbacks.append(resource._trigger_put)
---> 78 resource._trigger_get(None)
79
C:\ProgramData\Anaconda3\lib\site-packages\simpy\resources\base.py in _trigger_get(self, put_event)
223 get_event = self.get_queue[idx]
--> 224 proceed = self._do_get(get_event)
225 if not get_event.triggered:
<ipython-input-50-27e09b52b41a> in _do_get(self, event)
3 super()._do_get(event)
----> 4 event.value.tagged = True
5
C:\ProgramData\Anaconda3\lib\site-packages\simpy\events.py in value(self)
132 if self._value is PENDING:
--> 133 raise AttributeError('Value of %s is not yet available' % self)
134 return self._value
AttributeError: Value of <StoreGet() object at 0x9030c88> is not yet available
The above exception was the direct cause of the following exception:
AttributeError Traceback (most recent call last)
<ipython-input-50-27e09b52b41a> in <module>()
24 env.process(getter())
25
---> 26 env.run(until=20)
C:\ProgramData\Anaconda3\lib\site-packages\simpy\core.py in run(self, until)
135 try:
136 while True:
--> 137 self.step()
138 except StopSimulation as exc:
139 return exc.args[0] # == until.value
C:\ProgramData\Anaconda3\lib\site-packages\simpy\core.py in step(self)
227 exc = type(event._value)(*event._value.args)
228 exc.__cause__ = event._value
--> 229 raise exc
AttributeError: Value of <StoreGet() object at 0x9030c88> is not yet available
Looking through base.py, I see that _trigger_get is called in the init function of Get, so it makes sense that this fails, but how can I accomplish what I want to achieve? One workaround I have is just to do the attribute assignment in a try/catch and catch AttributeErrors, but this feels like a hack.
Also, if I add a print statement at the beginning of _do_get it will actually print twice and then raise the exception, which I find weird.
Maybe you can subclass the StoreGet event and set the attribute when its succeed() method is called. You’d also need also subclass Store like this:
class TaggedStoreGet(StoreGet):
def succeed(self, value=None):
self.tagged = True
return super().succeed(value)
class TaggedStore(Store):
get = BoundClass(TaggedStoreGet)
Haven’t tested it, but I think it might work.