I need to make one function in a module platform-independent by offering several implementations, without changing any files that import it. The following works:
do_it = getattr(__import__(__name__), "do_on_" + sys.platform)
...but breaks if the module is put into a package.
An alternative would be an if/elif with hard-coded calls to the others in do_it().
Anything better?
Put the code for platform support in different files in your package. Then add this to the file people are supposed to import from:
if sys.platform.startswith("win"):
from ._windows_support import *
elif sys.platform.startswith("linux"):
from ._unix_support import *
else:
raise ImportError("my module doesn't support this system")
Use globals()['do_on_' + platform] instead of the getattr call and your original idea should work whether this is inside a package or not.
If you need to create a platform specific instance of an class you should look into the Factory Pattern:
link text
Dive Into Python offers the exceptions alternative.
Related
I would like to find out disadvantages of using exec for imports. One of the files serves as interface towards real implementations of specific functionalities depending on chosen project (framework is intended to work on several projects).
First use-case goes like this:
exec ("from API.%s.specific_API_%s import *" % (project, project))
This way I don't have to hard code anything except the variable project which is injected in the interface-module itself.
This is the other way:
if project == 'project_one':
from API.project_one.specific_API_project_one import *
elif project == 'project_two':
from API.project_two.specific_API_project_two import *
elif project == 'project_three':
from API.project_three.specific_API_project_three import *
This way I have to alter this interface-file each time new project is added to be supported.
If you need programmatic way to import modules, please use importlib or __import__ (for really specific cases). Reasons — don't re-invent the wheel, there's way to do what you want without exec. If your project variable coming from outer world, exec is a huge security issue.
Wildcard imports considered bad practice — it makes harder to maintain your codebase afterwards.
Oversimplified example of issues with exec by executing arbitrary code:
module = 'request'
func = 'urlopen'
exec("from urllib.%s import %s" % (module, func))
func = 'urlopen; print("hello python")'
exec("from urllib.%s import %s" % (module, func))
yes, your example is harder to forge, but problem stays — giving python arbitrary code to execute is overkill (with potential security gap), when you have tool built exactly for your purpose — programatically importing modules.
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.
I would like to get the path of a library before importing the library itself.
That is, something different from:
import module, os
library_path = os.path.dirname(module.__file__)
Is that possible?
Thank you.
I think what you need is imp module:
import imp
file_handle, module_path, module_doc = imp.find_module(module_name)
The second return value is the path to actual file (assuming there is one, since requested module could be a built-in). First parameter is a file handle, already opened for you.
For as long as your use case is simple, you shouldn't have any issues. If you'll try for a generic solution you will need to read imp module documentation carefully, as there are lots of possible situations and return values for this function.
https://docs.python.org/2/library/imp.html
import commands
print commands.__file__
/usr/lib/python2.7/commands.py
import os
print os.__file__
/usr/lib/python2.7/os.pyc
Yes you can, but all modules doesn't support __ file__
For instance, there are os.path.walk, os.walk and assuming another md.walk, and assume os is imported but md is not. I desire a function like
whereis('walk')
while can return os.path.walk, os.walk and md.walk.
Or if it's difficult to know there is a md.walk, how to get the imported os.path.walk and os.walk?
OK, that's was fun, here's updated solution. Some Python magic, solution for Python 2.7. Any improvements are welcome. It behaves like any other import - so be careful to wrap any of your executable code in if name == "__main__".
import inspect
import pkgutil
import sys
def whereis_in_globals(fname):
return [m for m in sys.modules.itervalues()
if module_has_function(m, fname)]
def whereis_in_locals(fname):
modules = (__import__(module_name, globals(), locals(), [fname], -1)
for _, module_name, _ in pkgutil.walk_packages(path="."))
return [m for m in modules if module_has_function(m, fname)]
def module_has_function(m, fname):
return hasattr(m, fname) and inspect.isfunction(getattr(m, fname))
if __name__ == "__main__":
# these should never raise AttributeError
for m in whereis_in_locals('walk'):
m.walk
for m in whereis_in_globals('walk'):
m.walk
Can you tell us more about your use case or larger goals? Getting this answered for non-imported modules is an interesting challenge, and I don't have an answer for the way you've phrased it but depending on your actual use case you may be interested in knowing that may IDEs (Integrated Development Environments) can sort of do what you're asking.
For example if you install PyCharm (available for free in community edition) you can type a full or partial function name (just a text string) and then have the IDE search the active project to see what you might be looking for. For example you could just type "walk" on a line and then (IIR) Ctrl-click on it to see what possible modules you might have matching functions. You can also search the other direction, seeing where in a project the function is used.
To search on a wide scale you might have to cheat a little and open your Python directory as a "project" but that's easy.
Again, you haven't provided a use case so my apologies if you are specifically looking for, e.g., an in-script way to get that result for later use in a program. If it's something you want an answer to for you, as a developer and as you work, you might look to the tools you're using to provide an answer.
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.