Is there a way to do this in python 3.6+?
import -force mymodule
I just want a single python command that both:
(1) loads the module for the first time, and
(2) forces a reload of the module if it already loaded without barfing.
(This is not a duplicate question because I'm asking for something different. What I want is a single function call that will do Items (1) and (2) above as the same function call. I don't want to make a coding decision manually about if I could issue "import" or "imp.reload". I just want python code for a single function "def" that can detect which case is appropriate and proceed automatically to make the decision for me about how to import it it, (1) or (2).
I'm thinking that it something like this:
def import_force(m):
import sys
if m not in sys.modules:
import m
else:
import importlib
importlib.reload(m)
Except, I can't figure out how to pass a module name as a parameter. Just gives me an error no such module named 'm'
There is one missing step that you semi-corrected in your new answer, which is that you need to assign the new module in every scope that uses it. The easiest way is to return the module object and bind it to the name you want outside your function. Your original implementation was 90% correct:
import sys, importlib
def import_force(m):
if m not in sys.modules:
return __import__(m)
else:
return importlib.reload(sys.modules[m])
Now you can use this function from the command line to replace import, e.g.:
my_module = force_import('my_module')
Any time you find yourself using exec to perform a task for which there is so much well defined machinery already available, you have code smell. There is also no reason to re-import sys and importlib every time.
This function should do what you want:
def import_force(name):
needs_reload = name in sys.modules
module = importlib.import_module(name)
if needs_reload:
module = importlib.reload(module)
return module
# Usage example:
os = import_force('os')
An alternative approach is to write your own import hooks, which I won't describe.
However please note that this is an anti-pattern and I would discourage the practice of reloading modules at every import.
If this is for debugging purposes, then I would suggest using one of the many auto-reloader solutions available online: they watch your Python files for changes, and when you make modifications they automatically re-import the modules.
The reasons why your function didn't work are two:
The import keyword does not resolve variables, so import m does not mean "import the module which name is in the variable m", but rather it means "import the module named m".
importlib.reload wants a module object, not a module name.
import sys
import importlib
# importing with a sledgehammer... simple, effective, and it always works
def import_force(name):
module = importlib.import_module(name)
module = importlib.reload(module)
return module
#assuming mymodule.py is in the current directory
mymodule = import_force("mymodule")
It's possible! but a little bit tricky to code correctly the first time...
import sys
import importlib
def import_force(modstr):
if modstr not in sys.modules:
print("IMPORT " + modstr)
cmd = "globals()['%s'] = importlib.import_module('%s')" % (modstr, modstr)
exec(cmd)
else:
print("RELOAD " + modstr)
cmd = "globals()['%s'] = importlib.reload(%s)" % (modstr, modstr)
exec(cmd)
If you have a module file in your current directory call "mymodule.py", then use it like this:
Py> import_force("mymodule")
Version 2.0:
def import_force(modstr):
if modstr not in sys.modules:
print("IMPORT " + modstr)
globals()[modstr] = importlib.import_module(modstr)
else:
print("RELOAD " + modstr)
globals()[modstr] = importlib.reload(sys.modules[modstr])
Related
So, I've two python files:
the 1st "m12345.py"
def my():
return 'hello world'
the 2nd "1234.py":
from m12345 import *
a = m12345.my()
print(a)
On ipython I try to exec such cmds:
exec(open("f:\\temp\\m12345.py").read())
exec(open("f:\\temp\\1234.py").read())
the error for the 2nd command is:
ImportError: No module named 'm12345'
Please, help how to add the 1st file as a module for the 2nd?
First off, if you use the universal import (from m12345 import *) then you just call the my() function and not the m12345.my() or else you will get a
NameError: name 'm12345' is not defined
Secondly, you should add the following snippet in every script in which you want to have the ability of directly running it or not (when importing it).
if "__name__" = "__main__":
pass
PS. Add this to the 1st script ("m12345.py").
PS2. Avoid using the universal import method since it has the ability to mess the namespace of your script. (For that reason, it isn't considered best practice).
edit: Is the m12345.py located in the python folder (where it was installed in your hard drive)? If not, then you should add the directory it is located in the sys.path with:
import sys
sys.path.append(directory)
where directory is the string of the location where your m12345.py is located. Note that if you use Windows you should use / and not \.
However it would be much easier to just relocate the script (if it's possible).
You have to create a new module (for example m12345) by calling m12345 = imp.new_module('m12345') and then exec the python script in that module by calling exec(open('path/m12345.py').read(), m12345.__dict__). See the example below:
import imp
pyfile = open('path/m12345.py').read()
m12345 = imp.new_module('m12345')
exec(pyfile, m12345.__dict__)
If you want the module to be in system path, you can add
sys.modules['m12345'] = m12345
After this you can do
import m12345
or
from m12345 import *
I have written a script for XBMC which optionally downloads a dll and then imports a module that depends on that dll if the download was successful.
However, placing the import inside a function generates a Python syntax warning.
Simplified example:
1 def importIfPresent():
2 if chkFunction() is True:
3 from myOptionModule import *
Line 3 generates the warning, but doesn't stop the script. I can't place this code at the start outside of a function because I need to generate dialog boxes to prompt the download and then hash the file once it is downloaded to check success. I also call this same code at startup in order to check if the user has already downloaded the dll.
Is there a different/better way to do this without generating the syntax warning? Or should I just ignore the warning and leave it as is?
Thank you! Using the useful responses below, I now have:
import importlib
myOptionalModule = None
def importIfPresent():
if chkFunction is True:
try:
myOptionalModule = importlib.import_module('modulex')
except ImportError:
myOptionalModule = None
...
importIfPresent()
...
def laterFunction():
if myOptionalModule != None:
myParam = 'something expected'
myClass = getattr(myOptionalModule, 'importClassName')
myFunction = getattr(myClass, 'functionName')
result = myFunction(myClass(), myParam)
else:
callAlternativeMethod()
I am posting this back mainly to share with other beginners like myself the way I learned through the discussion to use the functionality of a module imported this way instead of the standard import statement. I'm sure that there are more elegant ways of doing this that the experts will share as well...
You're not getting the warning for doing an import inside a function, you're getting the warning for using from <module> import * inside a function. Doing a In Python3, this actually becomes a SyntaxError, not a SyntaxWarning. See this answer for why wildcard imports like this in general, and expecially inside functions are discouraged.
Also, this code isn't doing what you think it does. When you do an import inside a function, the import only takes affect inside the function. You're not importing that module into the global namespace of the file, which I believe is what you're really trying to do.
As suggested in another answer importlib can help you here:
try:
import myOptionModule as opt
except ImportError:
opt = None
def importIfPresent():
global opt
if chkFunction() is True:
opt = importlib.import_module("myOptionModule")
I beleive you need to use the importlib library to facilitate this.
The code would be at the top of the mod:
import importlib
then replace "from myOptionModule import *" with "module = importlib.import_module(myOptionModule)". You can then import the defs/classes you want or import them all by using getattr(module,NAME(S)TOIMPORT).
See if that works.
Check out chapter 30 and 31 of Learning Python by Lutz for more info.
I have a codebase where I'm cleaning up some messy decisions by the previous developer. Frequently, he has done something like:
from scipy import *
from numpy import *
...This, of course, pollutes the name space and makes it difficult to tell where an attribute in the module is originally from.
Is there any way to have Python analyze and fix this for me? Has anyone made a utility for this? If not, how might a utility like this be made?
I think PurityLake's and Martijn Pieters's assisted-manual solutions are probably the best way to go. But it's not impossible to do this programmatically.
First, you need to get a list of all names that existing in the module's dictionary that might be used in the code. I'm assuming your code isn't directly calling any dunder functions, etc.
Then, you need to iterate through them, using inspect.getmodule() to find out which module each object was originally defined in. And I'm assuming that you're not using anything that's been doubly from foo import *-ed. Make a list of all of the names that were defined in the numpy and scipy modules.
Now you can take that output and just replace each foo with numpy.foo.
So, putting it together, something like this:
for modname in sys.argv[1:]:
with open(modname + '.py') as srcfile:
src = srcfile.read()
src = src.replace('from numpy import *', 'import numpy')
src = src.replace('from scipy import *', 'import scipy')
mod = __import__(modname)
for name in dir(mod):
original_mod = inspect.getmodule(getattr(mod, name))
if original_mod.__name__ == 'numpy':
src = src.replace(name, 'numpy.'+name)
elif original_mod.__name__ == 'scipy':
src = src.replace(name, 'scipy.'+name)
with open(modname + '.tmp') as dstfile:
dstfile.write(src)
os.rename(modname + '.py', modname + '.bak')
os.rename(modname + '.tmp', modname + '.py')
If either of the assumptions is wrong, it's not hard to change the code. Also, you might want to use tempfile.NamedTemporaryFile and other improvements to make sure you don't accidentally overwrite things with temporary files. (I just didn't want to deal with the headache of writing something cross-platform; if you're not running on Windows, it's easy.) And add in some error handling, obviously, and probably some reporting.
Yes. Remove the imports and run a linter on the module.
I recommend using flake8, although it may also create a lot of noise about style errors.
Merely removing the imports and trying to run the code is probably not going to be enough, as many name errors won't be raised until you run just the right line of code with just the right input. A linter will instead analyze the code by parsing and will detect potential NameErrors without having to run the code.
This all presumes that there are no reliable unit tests, or that the tests do not provide enough coverage.
In this case, where there are multiple from module import * lines, it gets a little more painful in that you need to figure out for each and every missing name what module supplied that name. That will require manual work, but you can simply import the module in a python interpreter and test if the missing name is defined on that module:
>>> import scipy, numpy
>>> 'loadtxt' in dir(numpy)
True
You do need to take into account that in this specific case, that there is overlap between the numpy and scipy modules; for any name defined in both modules, the module imported last wins.
Note that leaving any from module import * line in place means the linter will not be able to detect what names might raise NameErrors!
I've now made a small utility for doing this which I call 'dedazzler'. It will find lines that are 'from module import *', and then expand the 'dir' of the target modules, replacing the lines.
After running it, you still need to run a linter. Here's the particularly interesting part of the code:
import re
star_match = re.compile('from\s(?P<module>[\.\w]+)\simport\s[*]')
now = str(time.time())
error = lambda x: sys.stderr.write(x + '\n')
def replace_imports(lines):
"""
Iterates through lines in a Python file, looks for 'from module import *'
statements, and attempts to fix them.
"""
for line_num, line in enumerate(lines):
match = star_match.search(line)
if match:
newline = import_generator(match.groupdict()['module'])
if newline:
lines[line_num] = newline
return lines
def import_generator(modulename):
try:
prop_depth = modulename.split('.')[1:]
namespace = __import__(modulename)
for prop in prop_depth:
namespace = getattr(namespace, prop)
except ImportError:
error("Couldn't import module '%s'!" % modulename)
return
directory = [ name for name in dir(namespace) if not name.startswith('_') ]
return "from %s import %s\n"% (modulename, ', '.join(directory))
I'm maintaining this in a more useful stand-alone utility form here:
https://github.com/USGM/dedazzler/
ok, this is what i think you could do, break the program. remove the imports and notice the errors that are made. Then import only the modules that you want, this may take a while but this is the only way I know of doing this, I will be happily surprised if someone does know of a tool to help
EDIT:
ah yes, a linter, I hadn't thought of that.
Can a python module detect if has been imported with import module or from module import *? Something like
if __something__=='something':
print 'Directly imported with "import ' + __name__ + '"'
else:
print 'Imported with "from ' + __name__ + ' import *"'
Thank you.
No, it's not possible to detect this from within the module's code. Upon the first import, the module body is executed and a new module object is inserted in sys.modules. Only after this, the requested names are inserted into the namespace of the importing module.
Upon later imports, the module body isn't even executed. So if a module is first imported as
import module
and a second time as
from module import name
it has no chance to do anything at all during the second import. In particular, it cannot check how it is imported.
While Svens answer is probably the correct one, and this might seem a bit obvious, It is what I was really looking for when I stumbled upon this question.
This module will at least know that you passed an input argument to it. While allows unit testing of just this specific script without the unit test being performed in the module that imported it.
import sys
def myfunction(blah):
return "something like: " + blah
noargs=len(sys.argv)
if noargs>1:
for i in range(noargs-1):
print myfunction(sys.argv[i+1])
However, It doesn't really help you, Emilio, if you have no input arguments. : )
I have a legacy python program which defined a module whose name conflict with a system module in the current python, what's the easiest method to fix this? This module has been imported in many place, so it would be great if no need to change the the old python source.
You could create a custom import routine using the imp module. For example, here I'm importing a module named string which shadows the real string module:
import imp
def user_import(name, path):
mod_file, mod_path, mod_desc = imp.find_module(name, [path])
return imp.load_module(name, mod_file, mod_path, mod_desc)
# load the real 'string' module
import string
print string.digits
# load the user 'string' module which lives under the path './foo'
mod = user_import('string', './foo')
print mod.digits
Output:
0123456789
thesearenotdigits
You can always rename a module when you import it, such as:
import sys as realsys
realsys.exit(1)
It seems like you'll have to do this either for the system module whose name you have overloaded or for your module, depending on which causes the least pain.
Bottom line, when you create modules, it's good to more fully qualify the name, e.g. instead of creating sys (in this example), you could have created mylib.sys.