This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Dynamic module import in Python
I intend to make a suite of files at some point soon, and the best way to organize it is to have a list, that list will be at the very top of a file, and after it will come a ridiculous amount of code to handle what that list controls and how it operates. I'm looking to write said list only once, and said list is a list of folder and file names in this format:
[(folder/filename, bool, bool, int), (folder/filename, bool, bool, int)]
As you can see, folder/filename are the same (sort of). File name is folder name with .py on the end, but doing import XXX you don't need to do import XXX.py, so I don't see this causing an issue.
The problem I'm facing is importing using this method...
for (testName, auto, hardware, bit) in testList:
print(testName)
paths = "\\" + testName
print paths
addpath(paths)
sys.modules[testName] = testName # One of a few options I've seen suggested on the net
print("Path Added")
test = testName + ".Helloworld()"
eval(test)
So for each test I have, print the name, assemble a string which contains the path ("\\testName"), for this example, print the test path, then add the path to the list (sys.path.append(path)), then print to confirm it happened, then assemble a string which will be executed by eval for the tests main module and eventually eval it.
As you can see, I'm currently having to have a list of imports at the top. I can't simply do import testName (the contents of testName are the name of the module I wish to import), as it will try to find a module called testName, not a module called the contents of testName.
I've seen a few examples of where this has been done, but can't find any which work in my circumstances. If someone could literally throw a chunk of code which does it that would be wonderful.
I'd also request that I'm not hung, drawn, nor quartered for use of eval, it is used in a very controlled environment (the list through which it cycles is within the .py file, so no "end user" should mess with it).
Not sure if I understood everything correctly, but you can import a module dynamically using __import__:
mod = __import__(testName)
mod.HelloWorld()
Edit: I wasn't aware that the use of __import__ was discouraged by the python docs for user code: __import__ documentation (as noted by Bakuriu)
This should also work and would be considered better style:
import importlib
mod = importlib.import_module(testName)
mod.HelloWorld()
Never, ever, ever mess with sys.modules directly if you don't know exactly what you are doing.
There are a lot of ways to do what you want:
The build-in __import__ function
Using imp.load_module
Using importlib.import_module
I'd avoid using __import__ directly, and go for importlib.import_module(which is also suggested at the end of the documentation of __import__).
Add the path where module resides to sys.path. Import the module using __import__ function which accepts a string variable as module name.
import sys
sys.path.insert(0, mypath) # mypath = path of module to be imported
imported_module = __import__("string_module_name") # __import__ accepts string
imported_module.myfunction() # All symbols in mymodule are now available normally
Related
When writing throwaway scripts it's often needed to load a configuration file, image, or some such thing from the same directory as the script. Preferably this should continue to work correctly regardless of the directory the script is executed from, so we may not want to simply rely on the current working directory.
Something like this works fine if defined within the same file you're using it from:
from os.path import abspath, dirname, join
def prepend_script_directory(s):
here = dirname(abspath(__file__))
return join(here, s)
It's not desirable to copy-paste or rewrite this same function into every module, but there's a problem: if you move it into a separate library, and import as a function, __file__ is now referencing some other module and the results are incorrect.
We could perhaps use this instead, but it seems like the sys.argv may not be reliable either.
def prepend_script_directory(s):
here = dirname(abspath(sys.argv[0]))
return join(here, s)
How to write prepend_script_directory robustly and correctly?
I would personally just os.chdir into the script's directory whenever I execute it. It is just:
import os
os.chdir(os.path.split(__file__)[0])
However if you did want to refactor this thing into a library, you are in essence wanting a function that is aware of its caller's state. You thus have to make it
prepend_script_directory(__file__, blah)
If you just wanted to write
prepend_script_directory(blah)
you'd have to do cpython-specific tricks with stack frames:
import inspect
def getCallerModule():
# gets globals of module called from, and prints out __file__ global
print(inspect.currentframe().f_back.f_globals['__file__'])
I think the reason it doesn't smell right is that $PYTHONPATH (or sys.path) is the proper general mechanism to use.
You want pkg_resources
import pkg_resources
foo_fname = pkg_resources.resource_filename(__name__, "foo.txt")
I have a codebase where I'm cleaning up some messy decisions by the previous developer. Frequently, he has done something like:
from scipy import *
from numpy import *
...This, of course, pollutes the name space and makes it difficult to tell where an attribute in the module is originally from.
Is there any way to have Python analyze and fix this for me? Has anyone made a utility for this? If not, how might a utility like this be made?
I think PurityLake's and Martijn Pieters's assisted-manual solutions are probably the best way to go. But it's not impossible to do this programmatically.
First, you need to get a list of all names that existing in the module's dictionary that might be used in the code. I'm assuming your code isn't directly calling any dunder functions, etc.
Then, you need to iterate through them, using inspect.getmodule() to find out which module each object was originally defined in. And I'm assuming that you're not using anything that's been doubly from foo import *-ed. Make a list of all of the names that were defined in the numpy and scipy modules.
Now you can take that output and just replace each foo with numpy.foo.
So, putting it together, something like this:
for modname in sys.argv[1:]:
with open(modname + '.py') as srcfile:
src = srcfile.read()
src = src.replace('from numpy import *', 'import numpy')
src = src.replace('from scipy import *', 'import scipy')
mod = __import__(modname)
for name in dir(mod):
original_mod = inspect.getmodule(getattr(mod, name))
if original_mod.__name__ == 'numpy':
src = src.replace(name, 'numpy.'+name)
elif original_mod.__name__ == 'scipy':
src = src.replace(name, 'scipy.'+name)
with open(modname + '.tmp') as dstfile:
dstfile.write(src)
os.rename(modname + '.py', modname + '.bak')
os.rename(modname + '.tmp', modname + '.py')
If either of the assumptions is wrong, it's not hard to change the code. Also, you might want to use tempfile.NamedTemporaryFile and other improvements to make sure you don't accidentally overwrite things with temporary files. (I just didn't want to deal with the headache of writing something cross-platform; if you're not running on Windows, it's easy.) And add in some error handling, obviously, and probably some reporting.
Yes. Remove the imports and run a linter on the module.
I recommend using flake8, although it may also create a lot of noise about style errors.
Merely removing the imports and trying to run the code is probably not going to be enough, as many name errors won't be raised until you run just the right line of code with just the right input. A linter will instead analyze the code by parsing and will detect potential NameErrors without having to run the code.
This all presumes that there are no reliable unit tests, or that the tests do not provide enough coverage.
In this case, where there are multiple from module import * lines, it gets a little more painful in that you need to figure out for each and every missing name what module supplied that name. That will require manual work, but you can simply import the module in a python interpreter and test if the missing name is defined on that module:
>>> import scipy, numpy
>>> 'loadtxt' in dir(numpy)
True
You do need to take into account that in this specific case, that there is overlap between the numpy and scipy modules; for any name defined in both modules, the module imported last wins.
Note that leaving any from module import * line in place means the linter will not be able to detect what names might raise NameErrors!
I've now made a small utility for doing this which I call 'dedazzler'. It will find lines that are 'from module import *', and then expand the 'dir' of the target modules, replacing the lines.
After running it, you still need to run a linter. Here's the particularly interesting part of the code:
import re
star_match = re.compile('from\s(?P<module>[\.\w]+)\simport\s[*]')
now = str(time.time())
error = lambda x: sys.stderr.write(x + '\n')
def replace_imports(lines):
"""
Iterates through lines in a Python file, looks for 'from module import *'
statements, and attempts to fix them.
"""
for line_num, line in enumerate(lines):
match = star_match.search(line)
if match:
newline = import_generator(match.groupdict()['module'])
if newline:
lines[line_num] = newline
return lines
def import_generator(modulename):
try:
prop_depth = modulename.split('.')[1:]
namespace = __import__(modulename)
for prop in prop_depth:
namespace = getattr(namespace, prop)
except ImportError:
error("Couldn't import module '%s'!" % modulename)
return
directory = [ name for name in dir(namespace) if not name.startswith('_') ]
return "from %s import %s\n"% (modulename, ', '.join(directory))
I'm maintaining this in a more useful stand-alone utility form here:
https://github.com/USGM/dedazzler/
ok, this is what i think you could do, break the program. remove the imports and notice the errors that are made. Then import only the modules that you want, this may take a while but this is the only way I know of doing this, I will be happily surprised if someone does know of a tool to help
EDIT:
ah yes, a linter, I hadn't thought of that.
I'm refactoring and eliminating wildcard imports on some fairly monolithic code.
Pylint seems to do a great job of listing all the unused imports that come along with a wildcard import, but what i wish it did was provide a list of used imports so I can quickly replace the wildcard import. Any quick ways of doing this? I'm about to go parse the output of pyLint and do a set.difference() on this and the dir() of the imported module. But I bet there's some tool/procedure I'm not aware of.
NB: pylint does not recommend a set of used imports. When changing this, you have to be aware of other modules importing the code you are modifying, which could use symbols which belong to the namespace of the module you are refactoring only because you have unused imports.
I recommend the following procedure to refactor from foo import *:
in an interactive shell, type:
import re
import foo as module # XXX use the correct module name here!
module_name = module.__name__
import_line = 'from %s import (%%s)' % module_name
length = len(import_line) - 3
print import_line % (',\n' + length * ' ').join([a for a in dir(module)
if not re.match('__.*[^_]{2}', a)])
replace the from foo import * line with the one printed above
run pylint, and remove the unused imports flagged by pylint
run pylint again on the whole code based, looking for imports of non existing sympols
run your unit tests
repeat with from bar import *
Here's dewildcard, a very simple tool based on Alex's initial ideas:
https://github.com/quentinsf/dewildcard
This is an old question, but I wrote something that does this based on autoflake.
See here: https://github.com/fake-name/autoflake/blob/master/autostar.py
It works the opposite way dewildcard does, in that it attempts to fully qualify all uses of wildcard items.
E.g.
from os.path import *
Is converted to
import os.path
and all uses of os.path.<func> are prepended with the proper function.
I've searched and this seems to be a simple question without a simple answer.
I have the file a/b/c.py which would be called with python -m a.b.c. I would like to obtain the value a.b.c in the module level.
USAGE = u'''\
Usage:
python -m %s -h
''' % (what_do_i_put_here,)
So when I receive the -h option, I display the USAGE without the need to actually write down the actual value in each and every script.
Do I really need to go through inspect to get the desired value?
Thanks.
EDIT: As said, there are answers (I've searched), but not simple answers. Either use inspect, use of traceback, or manipulate __file__ and __package__ and do some substring to get the answer. But nothing as simple as if I had a class in the module, I could just use myClass.__module__ and I would get the answer I want. The use of __name__ is (unfortunately) useless as it's always "__main__".
Also, this is in python 2.6 and I cannot use any other versions.
This works for me:
__loader__.fullname
Also if I do python -m b.c from a\ I get 'b.c' as expected.
Not entirely sure what the __loader__ attribute is so let me know if this is no good.
edit: It comes from PEP 302: http://www.python.org/dev/peps/pep-0302/
Interesting snippets from the link:
The load_module() method has a few responsibilities that it must
fulfill before it runs any code:
...
It should add an __loader__ attribute to the module, set to the
loader object. This is mostly for introspection, but can be used
for importer-specific extras, for example getting data associated
with an importer.
So it looks like it should work fine in all cases.
I think you're actually looking for the __name__ special variable. From the Python documentation:
Within a module, the module’s name (as a string) is available as the value of the global variable __name__.
If you run a file directly, this name will __main__. However, if you're in a module (as in the case where you're using the -m flag, or any other import), it will be the complete name of the module.
When run with -m, sys.path[0] contains the full path to the module. You could use that to build the name.
source: http://docs.python.org/using/cmdline.html#command-line
Another option may be the __package__ built in variable which is available within modules.
Number of options are there to get the path/name of the current module.
First be familiar with the use of __file__ in Python, Click here to see the usage.
It holds the name of currently loaded module.
Check/Try the following code, it will work on both Python2 & Python3.
» module_names.py
import os
print (__file__)
print (os.path.abspath(__file__))
print (os.path.realpath(__file__))
Output on MAC OS X:
MacBook-Pro-2:practice admin$ python module_names.py
module_names.py
/Users/admin/projects/Python/python-the-snake/practice/module_names.py
/Users/admin/projects/Python/python-the-snake/practice/module_names.py
So here we got the name of current module name and its absolute path.
The only way is to do path manipulation with os.getcwd(), os.path, file and whatnot, as you mentioned.
Actually, it could be a good patch to implement for optparse / argparse (which currently replace "%prog" in the usage string with os.path.basename(sys.argv[0]) -- you are using optparse, right? -- ), i.e. another special string like %module.
Why does nobody mentioned the .__module__?
When doing a self.__module__ you will get the module path.
You can also do this outside of the class:
Class A:
self.__module__ # gets module.filename
def get_module():
A.__module__ # also gets module.filename
One liner But OS dependent
it does not work in interpreter! since file is meaningless there in the interpreter and is not defined.
does not require os module to be imported.
modulename=__file__.split("\\")[-1].split('.')[0]
Explanation:
X:\apple\pythonabc.py | will output pythonabc.py
select the last element after splitting with slashes, then select the first element by splitting it with dot '.'. because first step gives module.py, second step gives 'module' only. __file__ is a unique variable and returns the filepath of current module.
Comment any flaws or has any other pitfalls.
you should hardcode a.b.c in your help, if you distribute the package as such then that's the way to call it regardless of where a is located in the filesystem, as long as it's on the PYTHONPATH it'll be imported.
I want to dynamically import a list of modules. I'm having a problem doing this. Python always yells out an ImportError and tells me my module doesn't exist.
First I get the list of module filenames and chop off the ".py" suffixes, like so:
viable_plugins = filter(is_plugin, os.listdir(plugin_dir))
viable_plugins = map(lambda name: name[:-3], viable_plugins)
Then I os.chdir to the plugins directory and map __import__ the entire thing, like so:
active_plugins = map(__import__, viable_plugins)
However, when I turn active_plugins into a list and try to access the modules within, Python will throw out an error, saying it cannot import the modules since they don't appear to be there.
What am I doing wrong?
Edit: By simply using the interactive interpreter, doing os.chdir and __import__(modulefilename) produces exactly what I need. Why isn't the above approach working, then? Am I doing something wrong with Python's more functional parts?
It says it can't do it, because even though you're changing your directory to where the modules are, that directory isn't on your import path.
What you need to do, instead of changing to the directory where the modules are located, is to insert that directory into sys.path.
import sys
sys.path.insert(0, directory_of_modules)
# do imports here.