Given a string with a module name, how do you import everything in the module as if you had called:
from module import *
i.e. given string S="module", how does one get the equivalent of the following:
__import__(S, fromlist="*")
This doesn't seem to perform as expected (as it doesn't import anything).
Please reconsider. The only thing worse than import * is magic import *.
If you really want to:
m = __import__ (S)
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir (m)
for attr in attrlist:
globals()[attr] = getattr (m, attr)
Here's my solution for dynamic naming of local settings files for Django. Note the addition below of a check to not include attributes containing '__' from the imported file. The __name__ global was being overwritten with the module name of the local settings file, which caused setup_environ(), used in manage.py, to have problems.
try:
import socket
HOSTNAME = socket.gethostname().replace('.','_')
# See http://docs.python.org/library/functions.html#__import__
m = __import__(name="settings_%s" % HOSTNAME, globals=globals(), locals=locals(), fromlist="*")
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir(m)
for attr in [a for a in attrlist if '__' not in a]:
globals()[attr] = getattr(m, attr)
except ImportError, e:
sys.stderr.write('Unable to read settings_%s.py\n' % HOSTNAME)
sys.exit(1)
The underlying problem is that I am developing some Django, but on more than one host (with colleagues), all with different settings. I was hoping to do something like this in the project/settings.py file:
from platform import node
settings_files = { 'BMH.lan': 'settings_bmh.py", ... }
__import__( settings_files[ node() ] )
It seemed a simple solution (thus elegant), but I would agree that it has a smell to it and the simplicity goes out the loop when you have to use logic like what John Millikin posted (thanks). Here's essentially the solution I went with:
from platform import node
from settings_global import *
n = node()
if n == 'BMH.lan':
from settings_bmh import *
# add your own, here...
else:
raise Exception("No host settings for '%s'. See settings.py." % node())
Which works fine for our purposes.
It appears that you can also use dict.update() on module's dictionaries in your case:
config = [__import__(name) for name in names_list]
options = {}
for conf in config:
options.update(conf.__dict__)
Update: I think there's a short "functional" version of it:
options = reduce(dict.update, map(__import__, names_list))
I didn't find a good way to do it so I took a simpler but ugly way from http://www.djangosnippets.org/snippets/600/
try:
import socket
hostname = socket.gethostname().replace('.','_')
exec "from host_settings.%s import *" % hostname
except ImportError, e:
raise e
Related
In short, is there a pythonic way to write SETTING_A = os.environ['SETTING_A']?
I want to provide a module environment.py from which I can import constants that are read from environment variables.
Approach 1:
import os
try:
SETTING_A = os.environ['SETTING_A']
SETTING_B = os.environ['SETTING_B']
SETTING_C = os.environ['SETTING_C']
except KeyError as e:
raise EnvironmentError(f'env var {e} is not defined')
Approach 2
import os
vs = ('SETTING_A', 'SETTING_B', 'SETTING_C')
try:
for v in vs:
locals()[v] = os.environ[v]
except KeyError as e:
raise EnvironmentError(f'env var {e} is not defined')
Approach 1 repeats the names of the variables, approach 2 manipulates locals and it's harder to see what constants will be importable from the module.
Is there a best practice to this problem?
Use python-decouple to set environment variables.
pip install python-decouple
Now you can create a .ini or .env file and store your variables there.
.env
SETTING_A=os.environ['SETTING_A']
SETTING_B=os.environ['SETTING_B']
SETTING_C=os.environ['SETTING_C']
Now in your settings file you can import the env variables and use them.
from decouple import config
setting_a = config('SETTING_A')
setting_b = config('SETTING_B')
setting_c = config('SETTING_C')
Note that while setting your env variables you shouldn't leave gap with "=" and you should use config('') to load them.
You can also use pydantic to achieve this
from pydantic import BaseSettings
class _Settings(BaseSettings):
SETTING_A: str
SETTING_B: str
SETTING_C: str
class SettingsHandler:
#classmethod
def generate(cls):
root_dir = os.path.dirname(os.path.abspath(__file__))
_file_path = f"{root_dir}/local.env"
if os.path.exists(ini_file_path):
return _Settings(_env_file=ini_file_path, _env_file_encoding="utf-8")
return _Settings()
SETTINGS = SettingsHandler.generate()
Now you can use this SETTINGS object. Sample usage is written below
print(SETTINGS.SETTING_A)
print(SETTINGS.SETTING_B)
print(SETTINGS.SETTING_C)
Sample env file
SETTING_A="value_of_setting_a"
SETTING_B="value_of_setting_b"
SETTING_C="value_of_setting_c"
I would like a function that, given a name which caused a NameError, can identify Python packages which could be imported to resolve it.
That part is fairly easy, and I've done it, but now I have an additional problem: I'd like to do it without causing side-effects. Here's the code I'm using right now:
def necessaryImportFor(name):
from pkgutil import walk_packages
for package in walk_packages():
if package[1] == name:
return name
try:
if hasattr(__import__(package[1]), name):
return package[1]
except Exception as e:
print("Can't check " + package[1] + " on account of a " + e.__class__.__name__ + ": " + str(e))
print("No possible import satisfies " + name)
The problem is that this code actually __import__s every module. This means that every side-effect of importing every module occurs. When testing my code I found that side-effects that can be caused by importing all modules include:
Launching tkinter applications
Requesting passwords with getpass
Requesting other input or raw_input
Printing messages (import this)
Opening websites (import antigravity)
A possible solution that I considered would be finding the path to every module (how? It seems to me that the only way to do this is by importing the module then using some methods from inspect on it), then parsing it to find every class, def, and = that isn't itself within a class or def, but that seems like a huge PITA and I don't think it would work for modules which are implemented in C/C++ instead of pure Python.
Another possibility is launching a child Python instance which has its output redirected to devnull and performing its checks there, killing it if it takes too long. That would solve the first four bullets, and the fifth one is such a special case that I could just skip antigravity. But having to start up thousands of instances of Python in this single function seems a bit... heavy and inefficient.
Does anyone have a better solution I haven't considered? Is there a simple way of just telling Python to generate an AST or something without actually importing a module, for example?
So I ended up writing a few methods which can list everything from a source file, without importing the source file.
The ast module doesn't seem particularly well documented, so this was a bit of a PITA trying to figure out how to extract everything of interest. Still, after ~6 hours of trial and error today, I was able to get this together and run it on the 3000+ Python source files on my computer without any exceptions being raised.
def listImportablesFromAST(ast_):
from ast import (Assign, ClassDef, FunctionDef, Import, ImportFrom, Name,
For, Tuple, TryExcept, TryFinally, With)
if isinstance(ast_, (ClassDef, FunctionDef)):
return [ast_.name]
elif isinstance(ast_, (Import, ImportFrom)):
return [name.asname if name.asname else name.name for name in ast_.names]
ret = []
if isinstance(ast_, Assign):
for target in ast_.targets:
if isinstance(target, Tuple):
ret.extend([elt.id for elt in target.elts])
elif isinstance(target, Name):
ret.append(target.id)
return ret
# These two attributes cover everything of interest from If, Module,
# and While. They also cover parts of For, TryExcept, TryFinally, and With.
if hasattr(ast_, 'body') and isinstance(ast_.body, list):
for innerAST in ast_.body:
ret.extend(listImportablesFromAST(innerAST))
if hasattr(ast_, 'orelse'):
for innerAST in ast_.orelse:
ret.extend(listImportablesFromAST(innerAST))
if isinstance(ast_, For):
target = ast_.target
if isinstance(target, Tuple):
ret.extend([elt.id for elt in target.elts])
else:
ret.append(target.id)
elif isinstance(ast_, TryExcept):
for innerAST in ast_.handlers:
ret.extend(listImportablesFromAST(innerAST))
elif isinstance(ast_, TryFinally):
for innerAST in ast_.finalbody:
ret.extend(listImportablesFromAST(innerAST))
elif isinstance(ast_, With):
if ast_.optional_vars:
ret.append(ast_.optional_vars.id)
return ret
def listImportablesFromSource(source, filename = '<Unknown>'):
from ast import parse
return listImportablesFromAST(parse(source, filename))
def listImportablesFromSourceFile(filename):
with open(filename) as f:
source = f.read()
return listImportablesFromSource(source, filename)
The above code covers the titular question: How do I check the contents of a Python package without running it?
But it leaves you with another question: How do I get the path to a Python package from just its name?
Here's what I wrote to handle that:
class PathToSourceFileException(Exception):
pass
class PackageMissingChildException(PathToSourceFileException):
pass
class PackageMissingInitException(PathToSourceFileException):
pass
class NotASourceFileException(PathToSourceFileException):
pass
def pathToSourceFile(name):
'''
Given a name, returns the path to the source file, if possible.
Otherwise raises an ImportError or subclass of PathToSourceFileException.
'''
from os.path import dirname, isdir, isfile, join
if '.' in name:
parentSource = pathToSourceFile('.'.join(name.split('.')[:-1]))
path = join(dirname(parentSource), name.split('.')[-1])
if isdir(path):
path = join(path, '__init__.py')
if isfile(path):
return path
raise PackageMissingInitException()
path += '.py'
if isfile(path):
return path
raise PackageMissingChildException()
from imp import find_module, PKG_DIRECTORY, PY_SOURCE
f, path, (suffix, mode, type_) = find_module(name)
if f:
f.close()
if type_ == PY_SOURCE:
return path
elif type_ == PKG_DIRECTORY:
path = join(path, '__init__.py')
if isfile(path):
return path
raise PackageMissingInitException()
raise NotASourceFileException('Name ' + name + ' refers to the file at path ' + path + ' which is not that of a source file.')
Trying the two bits of code together, I have this function:
def listImportablesFromName(name, allowImport = False):
try:
return listImportablesFromSourceFile(pathToSourceFile(name))
except PathToSourceFileException:
if not allowImport:
raise
return dir(__import__(name))
Finally, here's the implementation for the function that I mentioned I wanted in my question:
def necessaryImportFor(name):
packageNames = []
def nameHandler(name):
packageNames.append(name)
from pkgutil import walk_packages
for package in walk_packages(onerror=nameHandler):
nameHandler(package[1])
# Suggestion: Sort package names by count of '.', so shallower packages are searched first.
for package in packageNames:
# Suggestion: just skip any package that starts with 'test.'
try:
if name in listImportablesForName(package):
return package
except ImportError:
pass
except PathToSourceFileException:
pass
return None
And that's how I spent my Sunday.
I've written an IRC bot using Twisted and now I've gotten to the point where I want to be able to dynamically reload functionality.
In my main program, I do from bots.google import GoogleBot and I've looked at how to use reload to reload modules, but I still can't figure out how to do dynamic re-importing of classes.
So, given a Python class, how do I dynamically reload the class definition?
Reload is unreliable and has many corner cases where it may fail. It is suitable for reloading simple, self-contained, scripts. If you want to dynamically reload your code without restart consider using forkloop instead:
http://opensourcehacker.com/2011/11/08/sauna-reload-the-most-awesomely-named-python-package-ever/
You cannot reload the module using reload(module) when using the from X import Y form. You'd have to do something like reload(sys.modules['module']) in that case.
This might not necessarily be the best way to do what you want, but it works!
import bots.google
class BotClass(irc.IRCClient):
def __init__(self):
global plugins
plugins = [bots.google.GoogleBot()]
def privmsg(self, user, channel, msg):
global plugins
parts = msg.split(' ')
trigger = parts[0]
if trigger == '!reload':
reload(bots.google)
plugins = [bots.google.GoogleBot()]
print "Successfully reloaded plugins"
I figured it out, here's the code I use:
def reimport_class(self, cls):
"""
Reload and reimport class "cls". Return the new definition of the class.
"""
# Get the fully qualified name of the class.
from twisted.python import reflect
full_path = reflect.qual(cls)
# Naively parse the module name and class name.
# Can be done much better...
match = re.match(r'(.*)\.([^\.]+)', full_path)
module_name = match.group(1)
class_name = match.group(2)
# This is where the good stuff happens.
mod = __import__(module_name, fromlist=[class_name])
reload(mod)
# The (reloaded definition of the) class itself is returned.
return getattr(mod, class_name)
Better yet subprocess the plugins, then hypervise the subprocess, when the files change reload the plugins process.
Edit: cleaned up.
You can use the sys.modules to dynamically reload modules based on user-input.
Say that you have a folder with multiple plugins such as:
module/
cmdtest.py
urltitle.py
...
You can use sys.modules in this way to load/reload modules based on userinput:
import sys
if sys.modules['module.' + userinput]:
reload(sys.modules['module.' + userinput])
else:
' Module not loaded. Cannot reload '
try:
module = __import__("module." + userinput)
module = sys.modules["module." + userinput]
except:
' error when trying to load %s ' % userinput
When you do a from ... import ... it binds the object into the local namespace, so all you need to is re-import it. However, since the module is already loaded, it will just re-import the same version of the class so you would need to reload the module too. So this should do it:
from bots.google import GoogleBot
...
# do stuff
...
reload(bots.google)
from bots.google import GoogleBot
If for some reason you don't know the module name you can get it from GoogleBot.module.
def reload_class(class_obj):
module_name = class_obj.__module__
module = sys.modules[module_name]
pycfile = module.__file__
modulepath = string.replace(pycfile, ".pyc", ".py")
code=open(modulepath, 'rU').read()
compile(code, module_name, "exec")
module = reload(module)
return getattr(module,class_obj.__name__)
There is a lot of error checking you can do on this, if your using global variables you will probably have to figure out what happens then.
I'm working on an auto-reload feature for WHIFF
http://whiff.sourceforge.net
(so you have to restart the HTTP server less often, ideally never).
I have the following code to reload a package module "location"
if a file is added to the package directory. It doesn't work on Windows XP.
How can I fix it? I think the problem is that getmtime(dir) doesn't
change on Windows when the directory content changes?
I'd really rather not compare an os.listdir(dir) with the last directory
content every time I access the package...
if not do_reload and hasattr(location, "__path__"):
path0 = location.__path__[0]
if os.path.exists(path0):
dir_mtime = int( os.path.getmtime(path0) )
if fn_mtime<dir_mtime:
print "dir change: reloading package root", location
do_reload = True
md_mtime = dir_mtime
In the code the "fn_mtime" is the recorded mtime from the last (re)load.
... added comment: I came up with the following work around, which I think
may work, but I don't care for it too much since it involves code generation.
I dynamically generate a code fragment to load a module and if it fails
it tries again after a reload. Not tested yet.
GET_MODULE_FUNCTION = """
def f():
import %(parent)s
try:
from %(parent)s import %(child)s
except ImportError:
# one more time...
reload(%(parent)s)
from %(parent)s import %(child)s
return %(child)s
"""
def my_import(partname, parent):
f = None # for pychecker
parentname = parent.__name__
defn = GET_MODULE_FUNCTION % {"parent": parentname, "child": partname}
#pr "executing"
#pr defn
try:
exec(defn) # defines function f()
except SyntaxError:
raise ImportError, "bad function name "+repr(partname)+"?"
partmodule = f()
#pr "got", partmodule
setattr(parent, partname, partmodule)
#pr "setattr", parent, ".", partname, "=", getattr(parent, partname)
return partmodule
Other suggestions welcome. I'm not happy about this...
long time no see. I'm not sure exactly what you're doing, but the equivalent of your code:
GET_MODULE_FUNCTION = """
def f():
import %(parent)s
try:
from %(parent)s import %(child)s
except ImportError:
# one more time...
reload(%(parent)s)
from %(parent)s import %(child)s
return %(child)s
"""
to be execed with:
defn = GET_MODULE_FUNCTION % {"parent": parentname, "child": partname}
exec(defn)
is (per the docs), assuming parentname names a package and partname names a module in that package (if partname is a top-level name of the parentname package, such as a function or class, you'll have to use a getattr at the end):
import sys
def f(parentname, partname):
name = '%s.%s' % (parentname, partname)
try:
__import__(name)
except ImportError:
parent = __import__(parentname)
reload(parent)
__import__(name)
return sys.modules[name]
without exec or anything weird, just call this f appropriately.
you can try using getatime() instead.
I'm not understanding your question completely...
Are you calling getmtime() on a directory or an individual file?
There are two things about your first code snippet that concern me:
You cast the float from getmtime to int. Dependening on the frequency this code is run, you might get unreliable results.
At the end of the code you assign dir_mtime to a variable md_mtime. fn_mtime, which you check against, seems not to be updated.
Given a string with a module name, how do you import everything in the module as if you had called:
from module import *
i.e. given string S="module", how does one get the equivalent of the following:
__import__(S, fromlist="*")
This doesn't seem to perform as expected (as it doesn't import anything).
Please reconsider. The only thing worse than import * is magic import *.
If you really want to:
m = __import__ (S)
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir (m)
for attr in attrlist:
globals()[attr] = getattr (m, attr)
Here's my solution for dynamic naming of local settings files for Django. Note the addition below of a check to not include attributes containing '__' from the imported file. The __name__ global was being overwritten with the module name of the local settings file, which caused setup_environ(), used in manage.py, to have problems.
try:
import socket
HOSTNAME = socket.gethostname().replace('.','_')
# See http://docs.python.org/library/functions.html#__import__
m = __import__(name="settings_%s" % HOSTNAME, globals=globals(), locals=locals(), fromlist="*")
try:
attrlist = m.__all__
except AttributeError:
attrlist = dir(m)
for attr in [a for a in attrlist if '__' not in a]:
globals()[attr] = getattr(m, attr)
except ImportError, e:
sys.stderr.write('Unable to read settings_%s.py\n' % HOSTNAME)
sys.exit(1)
The underlying problem is that I am developing some Django, but on more than one host (with colleagues), all with different settings. I was hoping to do something like this in the project/settings.py file:
from platform import node
settings_files = { 'BMH.lan': 'settings_bmh.py", ... }
__import__( settings_files[ node() ] )
It seemed a simple solution (thus elegant), but I would agree that it has a smell to it and the simplicity goes out the loop when you have to use logic like what John Millikin posted (thanks). Here's essentially the solution I went with:
from platform import node
from settings_global import *
n = node()
if n == 'BMH.lan':
from settings_bmh import *
# add your own, here...
else:
raise Exception("No host settings for '%s'. See settings.py." % node())
Which works fine for our purposes.
It appears that you can also use dict.update() on module's dictionaries in your case:
config = [__import__(name) for name in names_list]
options = {}
for conf in config:
options.update(conf.__dict__)
Update: I think there's a short "functional" version of it:
options = reduce(dict.update, map(__import__, names_list))
I didn't find a good way to do it so I took a simpler but ugly way from http://www.djangosnippets.org/snippets/600/
try:
import socket
hostname = socket.gethostname().replace('.','_')
exec "from host_settings.%s import *" % hostname
except ImportError, e:
raise e