I'm debugging from the python console and would like to reload a module every time I make a change so I don't have to exit the console and re-enter it. I'm doing:
>>> from project.model.user import *
>>> reload(user)
but I receive:
>>>NameError: name 'user' is not defined
What is the proper way to reload the entire user class? Is there a better way to do this, perhaps auto-updating while debugging?
Thanks.
As asked, the best you can do is
>>> from project.models.user import *
>>> import project # get module reference for reload
>>> reload(project.models.user) # reload step 1
>>> from project.models.user import * # reload step 2
it would be better and cleaner if you used the user module directly, rather than doing import * (which is almost never the right way to do it). Then it would just be
>>> from project.models import user
>>> reload(user)
This would do what you want. But, it's not very nice. If you really need to reload modules so often, I've got to ask: why?
My suspicion (backed up by previous experience with people asking similar questions) is that you're testing your module. There are lots of ways to test a module out, and doing it by hand in the interactive interpreter is among the worst ways. Save one of your sessions to a file and use doctest, for a quick fix. Alternatively, write it out as a program and use python -i. The only really great solution, though, is using the unittest module.
If that's not it, hopefully it's something better, not worse. There's really no good use of reload (in fact, it's removed in 3.x). It doesn't work effectively-- you might reload a module but leave leftovers from previous versions. It doesn't even work on all kinds of modules-- extension modules will not reload properly, or sometimes even break horribly, when reloaded.
The context of using it in the interactive interpreter doesn't leave a lot of choices as to what you are doing, and what the real best solution would be. Outside it, sometimes people used reload() to implement plugins etc. This is dangerous at best, and can frequently be done differently using either exec (ah the evil territory we find ourselves in), or a segregated process.
For python3.4+, reload has been moved to the importlib module. you can use importlib.reload(). You can refer to this post.
>>> import importlib
>>> import project # get module reference for reload
>>> importlib.reload(project.models.user) # reload step 1
>>> from project.models.user import * # reload step 2
For python3 versions before 3.4, the module to import is imp (instead of importlib)
IPython can reload modules before executing every new line:
%load_ext autoreload
%autoreload 2
Where %autoreload 2reloads "all modules (except those excluded by %aimport) every time before executing the Python code typed."
See the docs:
https://ipython.org/ipython-doc/3/config/extensions/autoreload.html
You can't use reload() in a effective way.
Python does not provide an effective support for reloading or unloading of previously imported
modules; module references makes it impractical to reload a module because references could exist in many places of your program.
Python 3 has removed reload() feature entirely.
Unfortunately you've got to use:
>>> from project.model import user
>>> reload(user)
I don't know off the top of my head of something which will automatically reload modules at the interactive prompt… But I don't see any reason one shouldn't exist (in fact, it wouldn't be too hard to implement, either…)
Now, you could do something like this:
from types import ModuleType
import sys
_reload_builtin = reload
def reload(thing):
if isinstance(thing, ModuleType):
_reload_builtin(thing)
elif hasattr(thing, '__module__') and thing.__module__:
module = sys.modules[thing.__module__]
_reload_builtin(module)
else:
raise TypeError, "reload() argument must be a module or have an __module__"
You could also try twisted.python.rebuild.rebuild.
from test_reload import add_test
where test_reload is a module, and add_test is a function
if you changed the function add_test, of course you need to reload this function.
then you can do this:
import imp
imp.reload(test_reload)
from test_reload import add_test
this will refresh the function add_test.
so you need to add
imp.reload(test_reload)
from test_reload import add_test --add this line in your code
As of Python 3.4 you can use importlib.reload(module)
>>> from importlib import reload
>>> from project.model import user
>>> reload(user)
Related
I have a function:
def f():
print('hi')
saved in a file example.py.
I start the prompt on windows and hit python
and type the following commands:
from importlib import reload
import example
from example import *
And then I change something in the file example.py and hit reload(example). But it only works when I call example.f(). Do I always have to add an extra from example import * to be able to call just f() after reloading? Is there a better way to achieve that?
No, there is no other way to achieve that other than doing an explicit from example import * after you reload the module. On the other hand, it's never a good practice in general (namespace pollution) to import anything other than a module. Importing classes and functions directly is a bad practice in general and you should always avoid that, especially from A import *. Never do that.
If you're still thinking about this, you may benefit from IPython, and the autoreload extension.
IPython is a powerful interactive shell and alternative Python interpreter with features like code completion, and syntax highlighting.
From the autoreload module page: (https://ipython.org/ipython-doc/3/config/extensions/autoreload.html)
IPython extension to reload modules before executing user code.
autoreload reloads modules automatically before entering the execution of code typed at the IPython prompt.
This makes for example the following workflow possible:
In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: from foo import some_function
In [4]: some_function()
Out[4]: 42
In [5]: # open foo.py in an editor and change some_function to return 43
In [6]: some_function()
Out[6]: 43
The module was reloaded without reloading it explicitly, and the object imported with from foo import ... was also updated.
I have a python program that loads quite a bit of data before running. As such, I'd like to be able to reload code without reloading data. With regular python, importlib.reload has been working fine. Here's an example:
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
extensions = [
Extension("foo.bar", ["foo/bar.pyx"],
language="c++",
extra_compile_args=["-std=c++11"],
extra_link_args=["-std=c++11"])
]
setup(
name="system2",
ext_modules=cythonize(extensions, compiler_directives={'language_level' : "3"}),
)
foo/bar.py
cpdef say_hello():
print('Hello!')
runner.py:
import pyximport
pyximport.install(reload_support=True)
import foo.bar
import subprocess
from importlib import reload
if __name__ == '__main__':
def reload_bar():
p = subprocess.Popen('python setup.py build_ext --inplace',
shell=True,
cwd='<your directory>')
p.wait()
reload(foo.bar)
foo.bar.say_hello()
But this doesn't seem to work. If I edit bar.pyx and run reload_bar I don't see my changes. I also tried pyximport.build_module() with no luck -- the module rebuilt but didn't reload. I'm running in a "normal" python shell, not IPython if it makes a difference.
I was able to get a solution working for Python 2.x a lot easier than Python 3.x. For whatever reason, Cython seems to be caching the shareable object (.so) file it imports your module from, and even after rebuilding and deleting the old file while running, it still imports from the old shareable object file. However, this isn't necessary anyways (when you import foo.bar, it doesn't create one), so we can just skip this anyways.
The largest problem was that python kept a reference to the old module, even after reloading. Normal python modules seem to work find, but not anything cython related. To fix this, I run execute two statements in place of reload(foo.bar)
del sys.modules['foo.bar']
import foo.bar
This successfully (though probably less efficiently) reloads the cython module. The only issue that remains in in Python 3.x running that subprocess creates a problematic shareable objects. Instead, skip that all together and let the import foo.bar work its magic with the pyximporter module, and recompile for you. I also added an option to the pyxinstall command to specify the language level to match what you've specified in the setup.py
pyximport.install(reload_support=True, language_level=3)
So all together:
runner.py
import sys
import pyximport
pyximport.install(reload_support=True, language_level=3)
import foo.bar
if __name__ == '__main__':
def reload_bar():
del sys.modules['foo.bar']
import foo.bar
foo.bar.say_hello()
input(" press enter to proceed ")
reload_bar()
foo.bar.say_hello()
Other two files remained unchanged
Running:
Hello!
press enter to proceed
-replace "Hello!" in foo/bar.pyx with "Hello world!", and press Enter.
Hello world!
Cython-extensions are not the usual python-modules and thus the behavior of the underlying OS shimmers through. This answer is about Linux, but also other OSes have similar behavior/problems (ok, Windows wouldn't even allow you to rebuild the extension).
A cython-extension is a shared object. When importing, CPython opens this shared object via ldopen and calls the init-function, i.e. PyInit_<module_name> in Python3, which among other things registers the functions/functionality provided by the extension.
Is a shared-object loaded, we no longer can unload it, because there might be some Python objects alive, which would then have dangling pointers instead of function-pointers to the functionality from the original shared-object. See for example this CPython-issue.
Another important thing: When ldopen loads a shared object with the same path as one already loaded shared object, it will not read it from the disc, but just reuse the already loaded version - even if there is a different version on the disc.
And this is the problem with our approach: As long as the resulting shared object has the same name as the old one, you will never get to see the new functionality in the interpreter without restarting it.
What are your options?
A: Use pyximport with reload_support=True
Let's assume your Cython (foo.pyx) module looks as follows:
def doit():
print(42)
# called when loaded:
doit()
Now import it with pyximport:
>>> import pyximport
>>> pyximport.install(reload_support=True)
>>> import foo
42
>>> foo.doit()
42
foo.pyx was built and loaded (we can see, it prints 42 while loading, as expected). Let's take a look at the file of foo:
>>> foo.__file__
'/home/XXX/.pyxbld/lib.linux-x86_64-3.6/foo.cpython-36m-x86_64-linux-gnu.so.reload1'
You can see the additional reload1-suffix compared to the case built with reload_support=False. Seeing the file-name, we also verify that there is no other foo.so lying in the path somewhere and being wrongly loaded.
Now, let's change 42 to 21 in the foo.pyx and reload the file:
>>> import importlib
>>> importlib.reload(foo)
21
>>> foo.doit()
42
>>> foo.__file__
'/home/XXX/.pyxbld/lib.linux-x86_64-3.6/foo.cpython-36m-x86_64-linux-gnu.so.reload2'
What happened? pyximport built an extension with a different prefix (reload2) and loaded it. It was successful, because the name/path of the new extension is different due to the new prefix and we can see 21 printed while loaded.
However, foo.doit() is still the old version! If we look up the reload-documentation, we see:
When reload() is executed:
Python module’s code is recompiled and the module-level code re-executed,
defining a new set of objects which are bound to names in
the module’s dictionary by reusing the loader which originally loaded
the module. The init function of extension modules is not called a
second time.
init (i.e. PyInit_<module_name>) isn't executed for extension (that means also for Cython-extensions), thus PyModuleDef_Init with foo-module-definition isn't called and one is stuck with the old definition bound to foo.doit. This behavior is sane, because for some extension, init-function isn't supposed to be called twice.
To fix it we have to import the module foo once again:
>>> import foo
>>> foo.doit()
21
Now foo is reloaded as good as it gets - which means there might be still old objects being in use. But I trust you to know what you do.
B: Change the name of your extensions with every version
Another strategy could be to build the module foo.pyx as foo_prefix1.so and then foo_prefix2.so and so on and load it as
>>> import foo_perfixX as foo
This is strategy used by %%cython-magic in IPython, which uses sha1-hash of the Cython-code as prefix.
One can emulate IPython's approach using imp.load_dynamic (or its implementation with help of importlib, as imp is deprecated):
from importlib._bootstrap _load
def load_dynamic(name, path, file=None):
"""
Load an extension module.
"""
import importlib.machinery
loader = importlib.machinery.ExtensionFileLoader(name, path)
# Issue #24748: Skip the sys.modules check in _load_module_shim;
# always load new extension
spec = importlib.machinery.ModuleSpec(
name=name, loader=loader, origin=path)
return _load(spec)
And now putting so-files e.g. into different folders (or adding some suffix), so dlopen sees them as different from previous version we can use it:
# first argument (name="foo") tells how the init-function
# of the extension (i.e. `PyInit_<module_name>`) is called
foo = load_dynamic("foo", "1/foo.cpython-37m-x86_64-linux-gnu.so")
# now foo has new functionality:
foo = load_dynamic("foo", "2/foo.cpython-37m-x86_64-linux-gnu.so")
Even if reloading and reloading of extension in particular is kind of hacky, for prototyping purposes I would probably go with pyximport-solution... or use IPython and %%cython-magic.
I am writing a python module and I am using many imports of other different modules.
I am bit confused that whether I should import all the necessary dependent modules in the opening of the file or shall I do it when necessary.
I also wanted to know the implications of both.
I come from C++ back ground so I am really thrilled with this feature and does not see any reason of not using __import__(), importing the modules only when needed inside my function.
Kindly throw some light on this.
To write less code, import a module at the first lines of the script, e.g.:
#File1.py
import os
#use os somewhere:
os.path.chdir(some_dir)
...
...
#use os somewhere else, you don't need to "import os" everywhere
os.environ.update(some_dict)
While sometimes you may need to import a module locally (e.g., in a function):
abc=3
def foo():
from some_module import abc #import inside foo avoids you from naming conflicts
abc(...) #call the function, nothing to do with the variable "abc" outside "foo"
Don't worry about the time consumption when calling foo() multiple times, since import statements loads modules/functions only one time. Once a module/function is imported, the object is stored in dictionary sys.modules, which is a lookup table for speedup when running the same import statement.
As #bruno desthuilliers mentioned, importing insede functions may not be that pythonic, it violates PEP8, here's a discussion I found, you should stick to importing at the top of the file most of the time.
First, __import__ isn't usually needed anywhere. It's main purpose is to support dynamic importing of things that you don't know ahead of time (think plug-ins). You can easily use the import statement inside your function:
import sys
def foo():
import this
if __name__ == "__main__":
print sys.version_info
foo()
The main advantage to importing everything up-front is that it is most customary. That's where people reading your code will go to see if something is imported or not. Also, you don't need to write import os in every function that uses os. The main downsides of this approach are that:
you can get yourself into unresolvable import loops (A imports B which imports A)
that you pull everything into memory even if you aren't going to use it.
The second problem isn't typically an issue -- very rarely do you notice the performance or memory impact of an import.
If you run into the first problem, it's likely a symptom of poorly grouped code and the common stuff should be factored into a new module C which both A and B can use.
Firstly, it's a violation of PEP8 using imports inside functions.
Calling import it's an expensive call EVEN if the module is already loaded, so if your function is gonna being called many times this will not compensate the performance gain.
Also when you call "import test" python do this:
dataFile = __ import__('test')
The only downside of imports at the top of file it's the namespace that get polluted very fast depending on complexity of the file, but if your file it's too complex it's a signal of bad design.
I'm running through the Django tutorial.
After editing a class it says:
Save these changes and start a new Python interactive shell by running python manage.py shell again:
>>> from polls.models import Poll, Choice
Is it possible to do that without quitting the shell?
No, it's not. Every edit you make to .py files will NOT get reloaded in the shell automatically. If you want to have that type of feature you would have to use
django-extensions which supports what you're after.
Not sure I agree with the other answer.
Python has a built in function reload() which, from the docs:
Reload a previously imported module. The argument must be a module object, so it must have been successfully imported before.
This is useful if you have edited the module source file using an external editor and want to try out the new version without leaving the Python interpreter. The return value is the module object (the same as the module argument).
However you have to do from polls import models and then models.Poll (as it has to be passed the actual module rather than a class) and models.Choice in your code.
That way, without leaving the shell you can run reload(models).
Edit 1:
If you can't be bothered typing models. all the time, you can also enter in your own shortcut;
from polls import models as pm
pm.Poll
pm.Choice
reload(pm)
I have encountered this situation sometimes. And it's already discussed on SO.
It's true, reload() works. But at the same time it's not a very convenient option.
>>> from polls.models import Poll, Choice
.... #Some changes are done to your polls/models.py file and saved
>>> Poll #gives you old model
>>> reload(polls.models) #Reload works at module level!
>>> from polls.models import Poll, Choice # import again from the reloaded module
>>> Poll #gives you new model
What about?
def reload_all():
import sys
module = type(sys) # this type has no name!
for m in sys.modules.values():
if isinstance(m, module):
try:
reload(m)
except:
pass
Not sure if this may have any side effect, though.
This is a very basic question - but I haven't been able to find an answer by searching online.
I am using python to control ArcGIS, and I have a simple python script, that calls some pre-written code.
However, when I make a change to the pre-written code, it does not appear to result in any change. I import this module, and have tried refreshing it, but nothing happens.
I've even moved the file it calls to another location, and the script still works fine. One thing I did yesterday was I added the folder where all my python files are to the sys path (using sys.append('path') ), and I wonder if that made a difference.
Thanks in advance, and sorry for the sloppy terminology.
It's unclear what you mean with "refresh", but the normal behavior of Python is that you need to restart the software for it to take a new look on a Python module and reread it.
If your changes isn't taken care of even after restart, then this is due to one of two errors:
The timestamp on the pyc-file is incorrect and some time in the future.
You are actually editing the wrong file.
You can with reload re-read a file even without restarting the software with the reload() command. Note that any variable pointing to anything in the module will need to get reimported after the reload. Something like this:
import themodule
from themodule import AClass
reload(themodule)
from themodule import AClass
One way to do this is to call reload.
Example: Here is the contents of foo.py:
def bar():
return 1
In an interactive session, I can do:
>>> import foo
>>> foo.bar()
1
Then in another window, I can change foo.py to:
def bar():
return "Hello"
Back in the interactive session, calling foo.bar() still returns 1, until I do:
>>> reload(foo)
<module 'foo' from 'foo.py'>
>>> foo.bar()
'Hello'
Calling reload is one way to ensure that your module is up-to-date even if the file on disk has changed. It's not necessarily the most efficient (you might be better off checking the last modification time on the file or using something like pyinotify before you reload), but it's certainly quick to implement.
One reason that Python doesn't read from the source module every time is that loading a module is (relatively) expensive -- what if you had a 300kb module and you were just using a single constant from the file? Python loads a module once and keeps it in memory, until you reload it.
If you are running in an IPython shell, then there are some magic commands that exist.
The IPython docs cover this feature called the autoreload extension.
Originally, I found this solution from Jonathan March's blog posting on this very subject (see point 3 from that link).
Basically all you have to do is the following, and changes you make are reflected automatically after you save:
In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: Import MODULE
In [4]: my_class = Module.class()
my_class.printham()
Out[4]: ham
In [5]: #make changes to printham and save
In [6]: my_class.printham()
Out[6]: hamlet
I used the following when importing all objects from within a module to ensure web2py was using my current code:
import buttons
import table
reload(buttons)
reload(table)
from buttons import *
from table import *
I'm not really sure that is what you mean, so don't hesitate to correct me. You are importing a module - let's call it mymodule.py - in your program, but when you change its contents, you don't see the difference?
Python will not look for changes in mymodule.py each time it is used, it will load it a first time, compile it to bytecode and keep it internally. It will normally also save the compiled bytecode (mymodule.pyc). The next time you will start your program, it will check if mymodule.py is more recent than mymodule.pyc, and recompile it if necessary.
If you need to, you can reload the module explicitly:
import mymodule
[... some code ...]
if userAskedForRefresh:
reload(mymodule)
Of course, it is more complicated than that and you may have side-effects depending on what you do with your program regarding the other module, for example if variables depends on classes defined in mymodule.
Alternatively, you could use the execfile function (or exec(), eval(), compile())
I had the exact same issue creating a geoprocessing script for ArcGIS 10.2. I had a python toolbox script, a tool script and then a common script. I have a parameter for Dev/Test/Prod in the tool that would control which version of the code was run. Dev would run the code in the dev folder, test from test folder and prod from prod folder. Changes to the common dev script would not run when the tool was run from ArcCatalog. Closing ArcCatalog made no difference. Even though I selected Dev or Test it would always run from the prod folder.
Adding reload(myCommonModule) to the tool script resolved this issue.
The cases will be different for different versions of python.
Following shows an example of python 3.4 version or above:
hello import hello_world
#Calls hello_world function
hello_world()
HI !!
#Now changes are done and reload option is needed
import importlib
importlib.reload(hello)
hello_world()
How are you?
For earlier python versions like 2.x, use inbuilt reload function as stated above.
Better is to use ipython3 as it provides autoreload feature.