I am wondering if there is a way to have a python variable to behave like a python module.
Problem I currently have is that we have python bindings for our API. The bindings are automatically generated through swig and to use them someone would only needs to:
import module_name as short_name
short_name.functions()
Right now we are studying having the API to use Apache Thrift. To use it someone needs to:
client, transport = thrift_connect()
client.functions()
...
transport.close()
Problem is that we have loads of scripts and we were wondering if there is a way to have the thrift client object to behave like a module so that we don't need to modify all scripts. One idea we had was to do something like this:
client, transport = thrift_connect()
global short_name
short_name = client
__builtins__.short_name = client
This 'sort of' works. It creates a global variable 'short_name' that acts like a module, but it also generates other problems. If other files import the same module it is needed to comment those imports. Also, having a global variable is not a bright idea for maintenance purposes.
So, would there be a way to make the thrift client to behave like a module? So that people could continue to use the 'old' syntax, but under the hood the module import would trigger a connection ans return the object as the module?
EDIT 1:
It is fine for every import to open a connection. Maybe we could use some kind of singleton so that a specific interpreter can only open one connection even if it calls multiple imports on different files.
I thought about binding the transport.close() to a object termination. Could be the module itself, if that is possible.
EDIT 2:
This seems to do what I want:
client, transport = thrift_connect()
attributes = dict((name, getattr(client, name)) for name in dir(client) if not (name.startswith('__') or name.startswith('_')))
globals().update(attributes)
Importing a module shouldn't cause a network connection.
If you have mandatory setup/teardown steps then you could define a context manager:
from contextlib import contextmanager
#contextmanager
def thrift_client():
client, transport = thrift_connect()
client.functions()
try:
yield client
finally:
transport.close()
Usage:
with thrift_client() as client:
# use client here
In general, the auto-generated module with C-like API should be private e.g., name it _thrift_client and the proper pythonic API that is used outside should be written on top of it by hand in another module.
To answer the question from the title: you can make an object to behave like a module e.g., see sh.SelfWrapper and quickdraw.Module.
Related
How can I subscribe to an event using ALMemory's subscribeToEvent function, which requires the correct Python scope while using ROS (rospy) that initiates modules outside my code?
This question is similar to this Naoqi subscription question, but differs in the sense that rospy was not used, which is critical to the implementation.
My current code is as such;
mainBehaviour.py
from nao.nao import Nao
class mainBehaviour(AbstractBehaviour):
def init(self):
global mainBehaviourMethod
mainBehaviourMethod = self
self.nao.setCallOnFall("mainBehaviourMethod", "robotHasFallen")
def update(self):
print("Update")
def robotHasFallen(self, eventName, val, subscriberIdentifier):
print("FALLING")
nao.py
from naoqi import ALProxy, ALModule, ALBroker
import rospy
from math import radians
import weakref
class Nao(object):
def __init__(self):
self.nao_ip = rospy.get_param("nao_ip")
self.port = 9559
self.memory_proxy = ALProxy("ALMemory", self.nao_ip, self.port)
def setCallOnFall(self, module, method):
self.memory_proxy.subscribeToEvent("robotHasFallen", module, method)
What I want is that mainBehaviour.py, using nao.py, has its function robotHasFallen fire when the robot has fallen. However, the current code does not produce this behaviour (it ignores any fall), but does not produce an error either. I attempted to implement this using this ALMemory tutorial. However, this tutorial makes use of a single python file, in which methods aren't instantiated by ROS. Therefore I cannot make use of the line pythonModule = myModule("pythonModule"). I attempted to still obtain this Python scope (on which the answer to the previously linked question states "the python variable must have the same name than the module name you create"), by declaring a global variable pointing to self.
How can I create the desired behaviour, detecting a fallen robot using subscribeToEvent, using ROS with its consequences of not initiating modules myself and therefore not being able to pass its constructor?
I cannot install any further libraries because I use a university computer.
Your example code uses the "naoqi" library, but it's now more convenient to use the "qi" library (you can get it with "pip install qi", it's already present on your robot in versions 2.1 or above). In that version you can directly pass a callback, see here for a helper library that allows you to do events.connect("MyALMemoryKey", my_callback) (you pass the function and not just it's name - and it doesn't care where the function comes from).
Under the hood, it does this:
ALMemory.subscriber("MyALMemoryKey").signal.connect(my_callback)
(note that here ALMemory is a Service (qi framework) and not a module (naoqi framework).
You can directly use that helper lib (see the doc here, and some samples using it here), or use the same logic in your code (it's not very complicated, once you have a working example to start from).
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.
I am trying to bind hosts to specified ips in my python program. Just make it affect in the python program, so I am not going to modify the /etc/hosts file.
I tried to add a bit code to the create_connection function in socket.py for host-ip translation, like this:
host, port = address # the original code in socket.py
# My change here:
if host == "www.google.com":
host = target_ip
for res in getaddrinfo(host, port, 0, SOCK_STREAM): # the original code in socket.py
I found it works fine.
And now I want the host-ip translation only works in this python program.
So my question is: how can I make my python program import this socket.py not the build-in one when using import socket?
To make it clear, here is an example. Suppose 'test' is my work directory:
test
|--- main.py
|--- socket.py
In this case:
How can I make main.py use test/socket.py by import socket?
How can I make another modules use test/socket.py when they are
using import socket?
I think changing the module find path order may help. But I found that even if the current path('') is in the first place of sys.path already and import socket still imports the built-in scoket module.
You can monkey-patch sys.modules, placing your own module instead of the standard socket, before importing any other module which might be using it.
# myscript.py
from myproject import mysocket
import sys
sys.modules['socket'] = mysocket
# ... the rest of your code
import requests
...
For that, mysocket should expose everything which the standard socket does.
# mysocket.py
import socket as _std_socket
from socket import * # expose everything
def create_connection(address, *args, **kwargs):
if address == ...:
address = ...
return _std_socket.create_connection(address, *args, **kwargs)
This might be an over-simplification of what mysocket.py should look like. Youd' likely need to add some definitions before this can be used in production, but you get the idea.
Another approach would be to monkey-patch the socket module itself, i.e. overwrite names inside the original module.
# myscript.py
import socket
def create_connection2(...):
...
socket.create_connection = create_connection2
# ... the rest of your code
import requests
...
I prefer the former approach, becuase it is cleaner in the sense you don't need to go inside the module, only to hide it and override some things in it from the outside.
You can use relative imports to locally use a socket.py module. However, to do this your project must be structured as a package.
from . import socket
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.
Summary: when a certain python module is imported, I want to be able to intercept this action, and instead of loading the required class, I want to load another class of my choice.
Reason: I am working on some legacy code. I need to write some unit test code before I start some enhancement/refactoring. The code imports a certain module which will fail in a unit test setting, however. (Because of database server dependency)
Pseduo Code:
from LegacyDataLoader import load_me_data
...
def do_something():
data = load_me_data()
So, ideally, when python excutes the import line above in a unit test, an alternative class, says MockDataLoader, is loaded instead.
I am still using 2.4.3. I suppose there is an import hook I can manipulate
Edit
Thanks a lot for the answers so far. They are all very helpful.
One particular type of suggestion is about manipulation of PYTHONPATH. It does not work in my case. So I will elaborate my particular situation here.
The original codebase is organised in this way
./dir1/myapp/database/LegacyDataLoader.py
./dir1/myapp/database/Other.py
./dir1/myapp/database/__init__.py
./dir1/myapp/__init__.py
My goal is to enhance the Other class in the Other module. But since it is legacy code, I do not feel comfortable working on it without strapping a test suite around it first.
Now I introduce this unit test code
./unit_test/test.py
The content is simply:
from myapp.database.Other import Other
def test1():
o = Other()
o.do_something()
if __name__ == "__main__":
test1()
When the CI server runs the above test, the test fails. It is because class Other uses LegacyDataLoader, and LegacydataLoader cannot establish database connection to the db server from the CI box.
Now let's add a fake class as suggested:
./unit_test_fake/myapp/database/LegacyDataLoader.py
./unit_test_fake/myapp/database/__init__.py
./unit_test_fake/myapp/__init__.py
Modify the PYTHONPATH to
export PYTHONPATH=unit_test_fake:dir1:unit_test
Now the test fails for another reason
File "unit_test/test.py", line 1, in <module>
from myapp.database.Other import Other
ImportError: No module named Other
It has something to do with the way python resolves classes/attributes in a module
You can intercept import and from ... import statements by defining your own __import__ function and assigning it to __builtin__.__import__ (make sure to save the previous value, since your override will no doubt want to delegate to it; and you'll need to import __builtin__ to get the builtin-objects module).
For example (Py2.4 specific, since that's what you're asking about), save in aim.py the following:
import __builtin__
realimp = __builtin__.__import__
def my_import(name, globals={}, locals={}, fromlist=[]):
print 'importing', name, fromlist
return realimp(name, globals, locals, fromlist)
__builtin__.__import__ = my_import
from os import path
and now:
$ python2.4 aim.py
importing os ('path',)
So this lets you intercept any specific import request you want, and alter the imported module[s] as you wish before you return them -- see the specs here. This is the kind of "hook" you're looking for, right?
There are cleaner ways to do this, but I'll assume that you can't modify the file containing from LegacyDataLoader import load_me_data.
The simplest thing to do is probably to create a new directory called testing_shims, and create LegacyDataLoader.py file in it. In that file, define whatever fake load_me_data you like. When running the unit tests, put testing_shims into your PYTHONPATH environment variable as the first directory. Alternately, you can modify your test runner to insert testing_shims as the first value in sys.path.
This way, your file will be found when importing LegacyDataLoader, and your code will be loaded instead of the real code.
The import statement just grabs stuff from sys.modules if a matching name is found there, so the simplest thing is to make sure you insert your own module into sys.modules under the target name before anything else tries to import the real thing.
# in test code
import sys
import MockDataLoader
sys.modules['LegacyDataLoader'] = MockDataLoader
import module_under_test
There are a handful of variations on the theme, but that basic approach should work fine to do what you describe in the question. A slightly simpler approach would be this, using just a mock function to replace the one in question:
# in test code
import module_under_test
def mock_load_me_data():
# do mock stuff here
module_under_test.load_me_data = mock_load_me_data
That simply replaces the appropriate name right in the module itself, so when you invoke the code under test, presumably do_something() in your question, it calls your mock routine.
Well, if the import fails by raising an exception, you could put it in a try...except loop:
try:
from LegacyDataLoader import load_me_data
except: # put error that occurs here, so as not to mask actual problems
from MockDataLoader import load_me_data
Is that what you're looking for? If it fails, but doesn't raise an exception, you could have it run the unit test with a special command line tag, like --unittest, like this:
import sys
if "--unittest" in sys.argv:
from MockDataLoader import load_me_data
else:
from LegacyDataLoader import load_me_data