I have a file called sub.py, and I want to be able to call functions in it from the iPython shell. The iPython autoreload functionality has not been working very well, though. Sometimes it detects changes, sometimes it doesn't.
Instead of debugging autoreload, I was wondering if there's a way to just manually reload, or unload and load, modules in iPython. Currently I terminate the shell, start it again, re-import my module, and go from there. It would be great to be able to do a manual reload without killing the iPython shell.
I find my homebrewed %reimport to be very useful in this context:
def makemagic(f):
name = f.__name__
if name.startswith('magic_'): name = name[6:]
def wrapped(throwaway, *pargs, **kwargs): return f(*pargs,**kwargs)
if hasattr(f, '__doc__'): wrapped.__doc__ = f.__doc__
get_ipython().define_magic(name, wrapped)
return f
#makemagic
def magic_reimport(dd):
"""
The syntax
%reimport foo, bar.*
is a shortcut for the following:
import foo; foo = reload(foo)
import bar; bar = reload(bar); from bar import *
"""
ipython = get_ipython().user_ns
for d in dd.replace(',', ' ').split(' '):
if len(d):
bare = d.endswith('.*')
if bare: d = d[:-2]
exec('import xx; xx = reload(xx)'.replace('xx', d), ipython)
if bare: exec('from xx import *'.replace('xx', d), ipython)
Once gotcha is that, when there are sub-modules of packages involved, you have to reimport the sub-module, and then the top-level package:
reimport foo.bar, foo
Related
I am maintaining a python package in which I did some restructuring. Now, I want to support clients who still do from my_package.old_subpackage.foo import Foo instead of the new from my_package.new_subpackage.foo import Foo, without explicitly reintroducing many files that do the forwarding. (old_subpackage still exists, but no longer contains foo.py.)
I have learned that there are "loaders" and "finders", and my impression was that I should implement a loader for my purpose, but I only managed to implement a finder so far:
RENAMED_PACKAGES = {
'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}
# TODO: ideally, we would not just implement a "finder", but also a "loader"
# (using the importlib.util.module_for_loader decorator); this would enable us
# to get module contents that also pass identity checks
class RenamedFinder:
#classmethod
def find_spec(cls, fullname, path, target=None):
renamed = RENAMED_PACKAGES.get(fullname)
if renamed is not None:
sys.stderr.write(
f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
return importlib.util.find_spec(renamed)
return None
sys.meta_path.append(RenamedFinder())
https://docs.python.org/3.5/library/importlib.html#importlib.util.module_for_loader and related functionality, however, seem to be deprecated. I know it's not a very pythonic thing I am trying to achieve, but I would be glad to learn that it's achievable.
On import of your package's __init__.py, you can place whatever objects you want into sys.modules, the values you put in there will be returned by import statements:
from . import new_package
from .new_package import module1, module2
import sys
sys.modules["my_lib.old_package"] = new_package
sys.modules["my_lib.old_package.module1"] = module1
sys.modules["my_lib.old_package.module2"] = module2
If someone now uses import my_lib.old_package or import my_lib.old_package.module1 they will obtain a reference to my_lib.new_package.module1. Since the import machinery already finds the keys in the sys.modules dictionary, it never even begins looking for the old files.
If you want to avoid importing all the submodules immediately, you can emulate a bit of lazy loading by placing a module with a __getattr__ in sys.modules:
from types import ModuleType
import importlib
import sys
class LazyModule(ModuleType):
def __init__(self, name, mod_name):
super().__init__(name)
self.__mod_name = name
def __getattr__(self, attr):
if "_lazy_module" not in self.__dict__:
self._lazy_module = importlib.import(self.__mod_name, package="my_lib")
return self._lazy_module.__getattr__(attr)
sys.modules["my_lib.old_package"] = LazyModule("my_lib.old_package", "my_lib.new_package")
In the init file of the old module, have it import from the newer modules
Old (package.oldpkg):
foo = __import__("Path to new module")
New (package.newpkg):
class foo:
bar = "thing"
so
package.oldpkg.foo.bar is the same as package.newpkg.foo.bar
Hope this helps!
I think that this is what you are looking for:
RENAMED_PACKAGES = {
'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}
class RenamedFinder:
#classmethod
def find_spec(cls, fullname, path, target=None):
renamed = RENAMED_PACKAGES.get(fullname)
if renamed is not None:
sys.stderr.write(
f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
spec = importlib.util.find_spec(renamed)
spec.loader = cls
return spec
return None
#staticmethod
def create_module(spec):
return importlib.import_module(spec.name)
#staticmethod
def exec_module(module):
pass
sys.meta_path.append(RenamedFinder())
Still, IMO the approach that manipulates sys.modules is preferable as it is more readable, more explicit, and provides you much more control. It might become useful especially in further versions of your package when my_package.new_subpackage.foo starts to diverge from my_package.old_subpackage.foo while you would still need to provide the old one for backward compatibility. For that reason, you would maybe need to preserve the code of both anyway.
Consolidate all the old package names into my_package.
Old packages (old_package):
image_processing (class) Will be deleted and replaced by better_image_processing
text_recognition (class) Will be deleted and replaced by better_text_recognition
foo (variable) Will be moved to better_text_recognition
still_there (class) Will not move
New packages:
super_image_processing
better_text_recognition
Redirector (class of my_package):
class old_package:
image_processing = super_image_processing # Will be replaced
text_recognition = better_text_recognition # Will be replaced
Your main new module (my_package):
#imports here
class super_image_processing:
def its(gets,even,better):
pass
class better_text_recognition:
def now(better,than,ever):
pass
class old_package:
#Links
image_processing = super_image_processing
text_recognition = better_text_recognition
still_there = __import__("path to unchanged module")
This allows you to delete some files and keep the rest. If you want to redirect variables you would do:
class super_image_processing:
def its(gets,even,better):
pass
class better_text_recognition:
def now(better,than,ever):
pass
class old_package:
#Links
image_processing = super_image_processing
text_recognition = better_text_recognition
foo = text_recognition.foo
still_there = __import__("path to unchanged module")
Would this work?
I'm trying to create a process that dynamically watches jupyter notebooks, compiles them on modification and imports them into my current file, however I can't seem to execute the updated code. It only executes the first version that was loaded.
There's a file called producer.py that calls this function repeatedly:
import fs.fs_util as fs_util
while(True):
fs_util.update_feature_list()
In fs_util.py I do the following:
from fs.feature import Feature
import inspect
from importlib import reload
import os
def is_subclass_of_feature(o):
return inspect.isclass(o) and issubclass(o, Feature) and o is not Feature
def get_instances_of_features(name):
module = __import__(COMPILED_MODULE, fromlist=[name])
module = reload(module)
feature_members = getattr(module, name)
all_features = inspect.getmembers(feature_members, predicate=is_subclass_of_feature)
return [f[1]() for f in all_features]
This function is called by:
def update_feature_list(name):
os.system("jupyter nbconvert --to script {}{} --output {}{}"
.format(PATH + "/" + s3.OUTPUT_PATH, name + JUPYTER_EXTENSION, PATH + "/" + COMPILED_PATH, name))
features = get_instances_of_features(name)
for f in features:
try:
feature = f.create_feature()
except Exception as e:
print(e)
There is other irrelevant code that checks for whether a file has been modified etc.
I can tell the file is being reloaded correctly because when I use inspect.getsource(f.create_feature) on the class it displays the updated source code, however during execution it returns older values. I've verified this by changing print statements as well as comparing the return values.
Also for some more context the file I'm trying to import:
from fs.feature import Feature
class SubFeature(Feature):
def __init__(self):
Feature.__init__(self)
def create_feature(self):
return "hello"
I was wondering what I was doing incorrectly?
So I found out what I was doing wrong.
When called reload I was reloading the module I had newly imported, which was fairly idiotic I suppose. The correct solution (in my case) was to reload the module from sys.modules, so it would be something like reload(sys.modules[COMPILED_MODULE + "." + name])
I've written an IRC bot using Twisted and now I've gotten to the point where I want to be able to dynamically reload functionality.
In my main program, I do from bots.google import GoogleBot and I've looked at how to use reload to reload modules, but I still can't figure out how to do dynamic re-importing of classes.
So, given a Python class, how do I dynamically reload the class definition?
Reload is unreliable and has many corner cases where it may fail. It is suitable for reloading simple, self-contained, scripts. If you want to dynamically reload your code without restart consider using forkloop instead:
http://opensourcehacker.com/2011/11/08/sauna-reload-the-most-awesomely-named-python-package-ever/
You cannot reload the module using reload(module) when using the from X import Y form. You'd have to do something like reload(sys.modules['module']) in that case.
This might not necessarily be the best way to do what you want, but it works!
import bots.google
class BotClass(irc.IRCClient):
def __init__(self):
global plugins
plugins = [bots.google.GoogleBot()]
def privmsg(self, user, channel, msg):
global plugins
parts = msg.split(' ')
trigger = parts[0]
if trigger == '!reload':
reload(bots.google)
plugins = [bots.google.GoogleBot()]
print "Successfully reloaded plugins"
I figured it out, here's the code I use:
def reimport_class(self, cls):
"""
Reload and reimport class "cls". Return the new definition of the class.
"""
# Get the fully qualified name of the class.
from twisted.python import reflect
full_path = reflect.qual(cls)
# Naively parse the module name and class name.
# Can be done much better...
match = re.match(r'(.*)\.([^\.]+)', full_path)
module_name = match.group(1)
class_name = match.group(2)
# This is where the good stuff happens.
mod = __import__(module_name, fromlist=[class_name])
reload(mod)
# The (reloaded definition of the) class itself is returned.
return getattr(mod, class_name)
Better yet subprocess the plugins, then hypervise the subprocess, when the files change reload the plugins process.
Edit: cleaned up.
You can use the sys.modules to dynamically reload modules based on user-input.
Say that you have a folder with multiple plugins such as:
module/
cmdtest.py
urltitle.py
...
You can use sys.modules in this way to load/reload modules based on userinput:
import sys
if sys.modules['module.' + userinput]:
reload(sys.modules['module.' + userinput])
else:
' Module not loaded. Cannot reload '
try:
module = __import__("module." + userinput)
module = sys.modules["module." + userinput]
except:
' error when trying to load %s ' % userinput
When you do a from ... import ... it binds the object into the local namespace, so all you need to is re-import it. However, since the module is already loaded, it will just re-import the same version of the class so you would need to reload the module too. So this should do it:
from bots.google import GoogleBot
...
# do stuff
...
reload(bots.google)
from bots.google import GoogleBot
If for some reason you don't know the module name you can get it from GoogleBot.module.
def reload_class(class_obj):
module_name = class_obj.__module__
module = sys.modules[module_name]
pycfile = module.__file__
modulepath = string.replace(pycfile, ".pyc", ".py")
code=open(modulepath, 'rU').read()
compile(code, module_name, "exec")
module = reload(module)
return getattr(module,class_obj.__name__)
There is a lot of error checking you can do on this, if your using global variables you will probably have to figure out what happens then.
Is there a pdb equivalent to disp in gdb?
E.g. when I'm debugging C using gdb I can have variables printed on every 'step' through the code by typing:
disp var
When I'm debugging python using pdb I would like similar functionality, but disp does not seem to be there, the python pdb documentation does not seem to offer an alternative - but it seems like an odd omission?
The code bellow uses Python introspection features to add two new commands to the PDB module 0
just put the given function, and its call in a separate module, and import this module before starting debugging - you should have the 'disp' and 'undisp' commands do add and retract watches to variables.
It works by monkeypatching Python's pdb module, which is written in pure python.
# -*- coding: utf-8 -*-
def patch_pdb():
import pdb
def wrap(func):
def new_postcmd(self, *args, **kw):
result = func(self, *args, **kw)
if hasattr(self, "curframe") and self.curframe and hasattr(self, "watch_list"):
for arg in self.watch_list:
try:
print >> self.stdout, "%s: %s"% (arg, self._getval(arg)) + ", ",
except:
pass
self.stdout.write("\n")
return result #func(self, *args, **kw)
return new_postcmd
pdb.Pdb.postcmd = wrap(pdb.Pdb.postcmd)
def do_disp(self, arg):
if not hasattr(self, "watch_list"):
self.watch_list = []
self.watch_list.append(arg)
pdb.Pdb.do_disp = do_disp
def do_undisp(self, arg):
if hasattr(self, "watch_list"):
try:
self.watch_list.remove(arg)
except:
pass
pdb.Pdb.do_undisp = do_undisp
patch_pdb()
if __name__ == "__main__":
# for testing
import pdb; pdb.set_trace()
a = 0
for i in range(10):
print i
a += 2
Unfortunately I could only make it display the state of the variables as they where previously to the execution of the last command. (I tried a little bit, but monkeypatching the bdb module, which is the base for the Pdb did not seem to work as well). You can try and change the methods in either pdb.Pdb, bdb.Bdb or cmd.Cmd that are decorated by wrap to find one that is called after the debugged frame state has changed.
You can set up some aliases that will do this for you:
alias n next;; p var
alias s step;; p var
Printing a whole list of variable names is left as an exercise to the reader. Unfortunately doing it this way means that when you send the debugger an empty line, the "last command" it executes is p var rather than, for example, n. If you want to fix that, then you can use this somewhat hacky set of Pdb commands instead:
!global __stack; from inspect import stack as __stack
!global __Pdb; from pdb import Pdb as __Pdb
!global __pdb; __pdb = [__framerec[0].f_locals.get("pdb") or __framerec[0].f_locals.get("self") for __framerec in __stack() if (__framerec[0].f_locals.get("pdb") or __framerec[0].f_locals.get("self")).__class__ == __Pdb][-1]
alias s step;; p var;; !__pdb.lastcmd = "!__pdb.cmdqueue.append('s')"
alias n next;; p var;; !__pdb.lastcmd = "!__pdb.cmdqueue.append('n')"
During the pdb debugging you can type normal python code, beyond the one letter commands - so just using print var should work for you.
From what I can tell, pdb does not recognize when the source code has changed between "runs". That is, if I'm debugging, notice a bug, fix that bug, and rerun the program in pdb (i.e. without exiting pdb), pdb will not recompile the code. I'll still be debugging the old version of the code, even if pdb lists the new source code.
So, does pdb not update the compiled code as the source changes? If not, is there a way to make it do so? I'd like to be able to stay in a single pdb session in order to keep my breakpoints and such.
FWIW, gdb will notice when the program it's debugging changes underneath it, though only on a restart of that program. This is the behavior I'm trying to replicate in pdb.
The following mini-module may help. If you import it in your pdb session, then you can use:
pdb> pdbs.r()
at any time to force-reload all non-system modules except main. The code skips that because it throws an ImportError('Cannot re-init internal module main') exception.
# pdbs.py - PDB support
from __future__ import print_function
def r():
"""Reload all non-system modules, to reload stuff on pbd restart. """
import importlib
import sys
# This is likely to be OS-specific
SYS_PREFIX = '/usr/lib'
for k, v in list(sys.modules.items()):
if (
k == "__main__" or
k.startswith("pdb") or
not getattr(v, "__file__", None)
or v.__file__.startswith(SYS_PREFIX)
):
continue
print("reloading %s [%s]" % (k, v.__file__), file=sys.stderr)
importlib.reload(v)
Based on #pourhaus answer (from 2014), this recipe augments the pdb++ debugger with a reload command (expected to work on both Linux & Windows, on any Python installation).
TIP: the new reload command accepts an optional list of module-prefixes to reload (and to exclude), not to break already loaded globals when resuming debugging.
Just insert the following Python-3.6 code into your ~/.pdbrc.py file:
## Augment `pdb++` with a `reload` command
#
# See https://stackoverflow.com/questions/724924/how-to-make-pdb-recognize-that-the-source-has-changed-between-runs/64194585#64194585
from pdb import Pdb
def _pdb_reload(pdb, modules):
"""
Reload all non system/__main__ modules, without restarting debugger.
SYNTAX:
reload [<reload-module>, ...] [-x [<exclude-module>, ...]]
* a dot(`.`) matches current frame's module `__name__`;
* given modules are matched by prefix;
* any <exclude-modules> are applied over any <reload-modules>.
EXAMPLES:
(Pdb++) reload # reload everything (brittle!)
(Pdb++) reload myapp.utils # reload just `myapp.utils`
(Pdb++) reload myapp -x . # reload `myapp` BUT current module
"""
import importlib
import sys
## Derive sys-lib path prefix.
#
SYS_PREFIX = importlib.__file__
SYS_PREFIX = SYS_PREFIX[: SYS_PREFIX.index("importlib")]
## Parse args to decide prefixes to Include/Exclude.
#
has_excludes = False
to_include = set()
# Default prefixes to Exclude, or `pdb++` will break.
to_exclude = {"__main__", "pdb", "fancycompleter", "pygments", "pyrepl"}
for m in modules.split():
if m == "-x":
has_excludes = True
continue
if m == ".":
m = pdb._getval("__name__")
if has_excludes:
to_exclude.add(m)
else:
to_include.add(m)
to_reload = [
(k, v)
for k, v in sys.modules.items()
if (not to_include or any(k.startswith(i) for i in to_include))
and not any(k.startswith(i) for i in to_exclude)
and getattr(v, "__file__", None)
and not v.__file__.startswith(SYS_PREFIX)
]
print(
f"PDB-reloading {len(to_reload)} modules:",
*[f" +--{k:28s}:{getattr(v, '__file__', '')}" for k, v in to_reload],
sep="\n",
file=sys.stderr,
)
for k, v in to_reload:
try:
importlib.reload(v)
except Exception as ex:
print(
f"Failed to PDB-reload module: {k} ({v.__file__}) due to: {ex!r}",
file=sys.stderr,
)
Pdb.do_reload = _pdb_reload
What do you mean by "rerun the program in pdb?" If you've imported a module, Python won't reread it unless you explicitly ask to do so, i.e. with reload(module). However, reload is far from bulletproof (see xreload for another strategy).
There are plenty of pitfalls in Python code reloading. To more robustly solve your problem, you could wrap pdb with a class that records your breakpoint info to a file on disk, for example, and plays them back on command.
(Sorry, ignore the first version of this answer; it's early and I didn't read your question carefully enough.)
I decided to comment some lines in my input script, and after
(Pdb) run
I got pdb to recognize that change. The bad thing: it runs the script from the beginning. The good things below.
(Pdb) help run
run [args...]
Restart the debugged python program. If a string is supplied
it is split with "shlex", and the result is used as the new
sys.argv. History, breakpoints, actions and debugger options
are preserved. "restart" is an alias for "run".
May not work for more complex programs, but for a simple example using importlib.reload() using Python v3.5.3:
[user#machine ~] cat test.py
print('Test Message')
#
# start and run with debugger
#
[user#machine ~] python3 -m pdb test.py
> /home/user/test.py(1)<module>()
-> print('Test Message')
(Pdb) c
Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Test Message')
#
# in another terminal, change test.py to say "Changed Test Message"
#
#
# back in PDB:
#
(Pdb) import importlib; import test; importlib.reload(test)
Changed Test Message
<module 'test' from '/home/user/test.py'>
(Pdb) c
Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Changed Test Message')
(Pdb) c
Changed Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Changed Test Message')
ipdb %autoreload extension
6.2.0 docs document http://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html#module-IPython.extensions.autoreload :
In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: from foo import some_function
In [4]: some_function()
Out[4]: 42
In [5]: # open foo.py in an editor and change some_function to return 43
In [6]: some_function()
Out[6]: 43