I want to allow two different access to a particular functions. I'm not the author of sample.test_foo and I am making use of the #service decorator to work in these two ways.
The method say sample.test_foo is decorated by #service and when directly imported and accessed, it will run certain code and return the result.
Other mode I want is to make the function use a cache to fetch the data. How I'm doing this is, I'm asking the users to set a context variable - cache_mode_modules - and users are expected to add "sample". The decorator checks if the parent module is in the env, then it fetches from cache otherwise calls the method. How I fetch from cache etc are bit complex and isn't related to the question.
I really want to change the second mode of access. I want to create an API which takes in "sample" module and returns me a different callable.
import sample
new_sample = magic_wrapper(sample)
sample.test_foo() -> calls the actual func
new_sample.foo() -> calls the cache
The magic_wrapper need to set some module variable, say "_cache_mode=True" which I can use it in my decorator and decide how to access.
Note that even if package is passed to the magic_wrapper, I need the functionality to work.
I tried the below one
In [1]: def test(module):
...: SPEC_OS = importlib.util.find_spec(module)
...: a = importlib.util.module_from_spec(SPEC_OS)
...: SPEC_OS.loader.exec_module(a)
...: a._cache_mode = True
...: return a
It works for module but not for package. Can someone suggest if it is possible to have the magic_wrapper ?
From your comment, I think you are trying to change the behavior of specific functions in the package/module by overriding defaults and not actually make changes to the package/module core functionality. If that is right, functions are first class objects in Python so they can be passed around just like any other object and thus set as a default argument to another function.
Example:
def function_in_module(b, default=False):
if default:
return -b
return b
def magic_wrapper(b, func=function_in_module):
return func(b, default=True)
print(magic_wrapper(10)) # prints -10
print(function_in_module(10)) # prints 10
Same is true with an imported from an imported module. If the above function_in_module were actually in module, you could make a file magic_wrapper.py:
from module import function_in_module
def magic_wrapper(b, func=function_in_module):
return func(b, default=True)
and in main.py:
from module import function_in_module
from magic_wrapper import magic_wrapper
print(magic_wrapper(10)) # prints -10
print(function_in_module(10)) # prints 10
Related
I have a Python class in a base_params.py module within an existing codebase, which looks like this:
import datetime
class BaseParams:
TIMESTAMP = datetime.datetime.now()
PATH1 = f'/foo1/bar/{TIMESTAMP}/baz'
PATH2 = f'/foo2/bar/{TIMESTAMP}/baz'
Callers utilize it this way:
from base_params import BaseParams as params
print(params.PATH1)
Now, I want to replace the TIMESTAMP value with one that is dynamically specified at runtime (through e.g. CLI arguments).
Is there a way to do this in Python without requiring my callers to refactor their code in a dramatic way? This is currently confounding me because the contents of the class BaseParams get executed at 'compile' time, so there is no opportunity there to pass in a dynamic value as it's currently structured. And in some of my existing code, this object is being treated as "fully ready" at 'compile' time, for example, its values are used as function argument defaults:
def some_function(value1, value2=params.PATH1):
...
I am wondering if there is some way to work with Python modules and/or abuse Python's __special_methods__ to get this existing code pattern working more or less as-is, without a deeper refactoring of some kind.
My current expectation is "this is not really possible" because of that last example, where the default value is being specified in the function signature. But I thought I should check with the Python wizards to see if there may be a suitably Pythonic way around this.
Yes, you just need to make sure that the command line argument is parsed before the class is defined and before any function that uses the class's attribute as a default argument is defined (but that should already be the case).
(using sys.argv for sake of simplicity. It is better to use an actual argument parser such as argparse)
import datetime
import sys
class BaseParams:
try:
TIMESTAMP = sys.argv[1]
except IndexError:
TIMESTAMP = datetime.datetime.now()
PATH1 = f'/foo1/bar/{TIMESTAMP}/baz'
PATH2 = f'/foo2/bar/{TIMESTAMP}/baz'
print(BaseParams.TIMESTAMP)
$ python main.py dummy-argument-from-cli
outputs
dummy-argument-from-cli
while
$ python main.py
outputs
2021-06-26 02:32:12.882601
You can still totally replace the value of a class attribute after the class has been defined:
BaseParams.TIMESTAMP = <whatever>
There are definitely some more "magic" things you can do though, such as a class factory of some kind. Since Python 3.7 you can also take advantage of module __getattr__ to create a kind of factory for the BaseParams class (PEP 562)
In base_params.py you might rename BaseParams to _BaseParams or BaseParamsBase or something like that :)
Then at the module level define:
def __getattr__(attr):
if attr == 'BaseParams':
params = ... # whatever code you need to determine class attributes for BaseParams
return type('BaseParams', (_BaseParams,), params)
raise AttributeError(attr)
I have an architecture, where I use wrapper for calling functions from package module. Inside the module there is a function that calls another three. I need to override one of them in run-time. Exactly I need to change parameters that are forwarded to another set of functions being called.
Here is a case sample:
a.py
import b_wrapper as wrapper
def foo():
if wrapper.bar(parameter):
"""some more code goes here"""
b_wrapper.py
import some.package.module as module
def bar(parameter):
return module.baz(veryImportantParameter, parameter)
file.py
def functionThree(par): # needs to be overwritten
"""more functions called forwarding par as a parameter"""
def baz(veryImportantParameter, parameter)
functionOne(veryImportantParameter, otherParameters)
functionTwo(veryImportantParameter, someMoreParameters)
functionThree(veryImportantParameter, parameterToChange, evenMoreParameters)
What I tried to do is overriding in wrapper file, didn't work out, as other functions are interfering with it. As reference used this post.
I'm not quite sure that this is doable, because of unique functions that are called inside this module, also looking for alternatives that won't require overriding portion of module.
Edit: mixing up arguments and parameters is intentional for demonstration purpose only.
I have these modules:
base_module, module_1
base_module_extend, module_2.
These modules have such dependencies:
module_1 depends on base_module.
base_module_extend depends on base_module.
module_2 depends on base_module_extend.
If module_1 and module_2 are not installed at the same time, then everything works fine, but if they are installed, then dependencies change in the order those modules were installed.
So for example if I install module_2 and then module_1, when I start using module_2 when you call super in that module code, it finds class that was defined in base_module instead of base_module_extend first and then some things break, because it does not find some attributes/arguments which are defined in base_module_extend.
And if I install in opposite order: first install module_1 and module_2, then it when I use module_2, it works fine, because it calls correct classes from super.
Is there a way to make those modules call correct parent classes if they are installed both without caring at the order they were installed?
Or is it limitation and there is nothing you can do about it?..
Update
Example of what is broken if you install in wrong order (the first scenario):
This is a method defined in base_module:
def export_data(self, export_time=fields.Datetime.now()):
"""
This method can be overridden to implement custom data exportation
"""
self.ensure_one()
res = []
return res
This is a method that was changed in module_1:
def export_data(self, export_time=fields.Datetime.now()):
res = super(SFTPImportExport, self).export_data(export_time)
if self.process == 'custom_qty_available':
res = self._export_qty_available(export_time)
elif self.process == 'custom_sale':
res = self._export_sale(export_time)
return res
This is a method that was changed in base_module_extend:
def export_data(
self, export_time=fields.Datetime.now(), external_data=False):
"""
Overrides export_data to be able to provide external_data that
is already generated and can be directly returned to be used in export
#param export_time: time of export
#param external_data: used to bypass specific export
method and directly use data provided
"""
res = super(SFTPImportExport, self).export_data(export_time)
return res
And this is a same method that was changed in module_2:
def export_data(self, export_time=fields.Datetime.now(), external_data=False):
"""
Adds support for custom_invoice process
"""
res = super(SFTPImportExport, self).export_data(export_time=export_time, external_data=external_data)
if self.process == 'custom_invoice':
res = external_data
return res
Update2
When last method version (the one that is defined in module_2) is executed, I get this error:
"/opt/odoo/projects/project/sftp_import_export_extend/models/sftp_import_export.py", line 47, in export_sftp
export_time=now, external_data=external_data)
ValueError: "export_data() got an unexpected keyword argument 'external_data'" while evaluating
u'invoice_validate()'
So it looks like it get wrong parent class, the one that does not have such keyword argument
super() tends to choose wrong(unexpected) parent in Python. As an alternative you can use direct call of Python function from the desired superclass, instead of using super(), this is much more reliable way.
I do not know a directory structure and source file names of your Odoo module, so I'll try to guess the structure approximately and give you description of what should be done. Please adapt it to your case (to have correct module name, source file name, etc).
If the desired python class, say named as SFTPImportExport that contains the desired export_data method is defined in the following source file inside the base_module_extend Odoo module (relative path): base_module_extend/models/somefile.py than, instead of using super(), you can call it directly like:
import desired supercalss:
from openerp.base_module_extend.models.somefile import SFTPImportExport as ParentSFTPImportExport
and call it's method directly:
ParentSFTPImportExport.export_data(self, export_time=export_time, external_data=external_data)
In this case the exact function you desire will be called, as you directly refer to it. I suggest to do the same, i.e. directly call desired function, in the both modules, to be sure that they never mix their target functions.
I have a function called get_account(param1,param2)
in run time I need to replace this function with the function mock_get_account(param1,param2)
so when the system calls get_account(param1,param2) I need the mock_get_account(param1,param2) to be called instead.
I tried this code:
package.get_account=self.mock_get_account
package.get_account(x,y)
but still the get_account runs instead of the mock_get_account
I'm new to python and I don't know if this is even possible but I have seen the lamda function and I know that function programming is possible in python. Thanks
Edit:
if i do the following:
package.get_account=self.mock_get_account
package.get_account(x,y)
then every thing is ok, meaning the mock_get_account is called, but in mu code I the following code i do a post self.client.post(url, data=data, follow=True) that triggers the package.get_account and this is not working:
package.get_account=self.mock_get_account
package.get_account(x,y)
#the folowing call will trigger the package.get_account(x,y) function in a django url #callback
self.client.post(url, data=data, follow=True)
meaning it calls the old function, also get_account(param1,param2) is defined in side a file, and is not a child function of a class and mock_get_account(self,param1,param2) is defined in a class Test and is called inside the Test.test_account - function
This is very opinionated and does not (directly) answer your question, but hopefully solves your problem.
A better practice is to use a subclass with your mock_get_account's implementation override the parent get_account method, example below:
class A(object):
def get_account(self):
return 1
def post(self):
return self.get_account()
class B(A):
def get_account(self):
return 2 # your original mock_get_account implementation
a = A()
print(a.get_account())
b = B()
print(b.post()) # this .post will trigger the overridden implementation of get_account
My guess is that the code implementing self.client.post has access to get_account through an import statement that looks like from package import get_account.
from package import get_account will first load package if it hasn't been already imported. Then it will look for a name get_account in that module, and whatever object that was bound to will be bound in the importing package's namespace, also under the name get_account. Thereafter the two names refer to the same object, but they are not the same name.
So if your mocking code comes along after this point, it sets the name get_account in package to instead refer to mock_get_account. But that'll only affect code that reads get_account from package again; anything that's already imported that name specially won't be affected.
If the code behind self.client.post instead had access only to package through import package, and was calling package.get_account it would work, because it's then only the object representing the package module that has been bound in the importing module's namespace. package.get_account would be reading an attribute of that object, and so would get whatever the current value is. If the from package import get_account appeared at function local scope rather than module scope, then this would behave similarly.
If I'm correct and your code is structured this way, then it's unfortunately not really package.get_account you need to rebind to a mock, but the get_account name in the module where self.client.post comes from (as well as any other modules which may call it).
I am doing some heavy commandline stuff (not really web based) and am new to Python, so I was wondering how to set up my files/folders/etc. Are there "header" files where I can keep all the DB connection stuff?
How/where do I define classes and objects?
Just to give you an example of a typical Python module's source, here's something with some explanation. This is a file named "Dims.py". This is not the whole file, just some parts to give an idea what's going on.
#!/usr/bin/env python
This is the standard first line telling the shell how to execute this file. Saying /usr/bin/env python instead of /usr/bin/python tells the shell to find Python via the user's PATH; the desired Python may well be in ~/bin or /usr/local/bin.
"""Library for dealing with lengths and locations."""
If the first thing in the file is a string, it is the docstring for the module. A docstring is a string that appears immediately after the start of an item, which can be accessed via its __doc__ property. In this case, since it is the module's docstring, if a user imports this file with import Dims, then Dims.__doc__ will return this string.
# Units
MM_BASIC = 1500000
MILS_BASIC = 38100
IN_BASIC = MILS_BASIC * 1000
There are a lot of good guidelines for formatting and naming conventions in a document known as PEP (Python Enhancement Proposal) 8. These are module-level variables (constants, really) so they are written in all caps with underscores. No, I don't follow all the rules; old habits die hard. Since you're starting fresh, follow PEP 8 unless you can't.
_SCALING = 1
_SCALES = {
mm_basic: MM_BASIC,
"mm": MM_BASIC,
mils_basic: MILS_BASIC,
"mil": MILS_BASIC,
"mils": MILS_BASIC,
"basic": 1,
1: 1
}
These module-level variables have leading underscores in their names. This gives them a limited amount of "privacy", in that import Dims will not let you access Dims._SCALING. However, if you need to mess with it, you can explicitly say something like import Dims._SCALING as scaling.
def UnitsToScale(units=None):
"""Scales the given units to the current scaling."""
if units is None:
return _SCALING
elif units not in _SCALES:
raise ValueError("unrecognized units: '%s'." % units)
return _SCALES[units]
UnitsToScale is a module-level function. Note the docstring and the use of default values and exceptions. No spaces around the = in default value declarations.
class Length(object):
"""A length. Makes unit conversions easier.
The basic, mm, and mils properties can be used to get or set the length
in the desired units.
>>> x = Length(mils=1000)
>>> x.mils
1000.0
>>> x.mm
25.399999999999999
>>> x.basic
38100000L
>>> x.mils = 100
>>> x.mm
2.54
"""
The class declaration. Note the docstring has things in it that look like Python command line commands. These care called doctests, in that they are test code in the docstring. More on this later.
def __init__(self, unscaled=0, basic=None, mm=None, mils=None, units=None):
"""Constructs a Length.
Default contructor creates a length of 0.
>>> Length()
Length(basic=0)
Length(<float>) or Length(<string>) creates a length with the given
value at the current scale factor.
>>> Length(1500)
Length(basic=1500)
>>> Length("1500")
Length(basic=1500)
"""
# Straight copy
if isinstance(unscaled, Length):
self._x = unscaled._x
return
# rest omitted
This is the initializer. Unlike C++, you only get one, but you can use default arguments to make it look like several different constructors are available.
def _GetBasic(self): return self._x
def _SetBasic(self, x): self._x = x
basic = property(_GetBasic, _SetBasic, doc="""
This returns the length in basic units.""")
This is a property. It allows you to have getter/setter functions while using the same syntax as you would for accessing any other data member, in this case, myLength.basic = 10 does the same thing as myLength._SetBasic(10). Because you can do this, you should not write getter/setter functions for your data members by default. Just operate directly on the data members. If you need to have getter/setter functions later, you can convert the data member to a property and your module's users won't need to change their code. Note that the docstring is on the property, not the getter/setter functions.
If you have a property that is read-only, you can use property as a decorator to declare it. For example, if the above property was to be read-only, I would write:
#property
def basic(self):
"""This returns the length in basic units."""
return self._x
Note that the name of the property is the name of the getter method. You can also use decorators to declare setter methods in Python 2.6 or later.
def __mul__(self, other):
"""Multiplies a Length by a scalar.
>>> Length(10)*10
Length(basic=100)
>>> 10*Length(10)
Length(basic=100)
"""
if type(other) not in _NumericTypes:
return NotImplemented
return Length(basic=self._x * other)
This overrides the * operator. Note that you can return the special value NotImplemented to tell Python that this operation isn't implemented (in this case, if you try to multiply by a non-numeric type like a string).
__rmul__ = __mul__
Since code is just a value like anything else, you can assign the code of one method to another. This line tells Python that the something * Length operation uses the same code as Length * something. Don't Repeat Yourself.
Now that the class is declared, I can get back to module code. In this case, I have some code that I want to run only if this file is executed by itself, not if it's imported as a module. So I use the following test:
if __name__ == "__main__":
Then the code in the if is executed only if this is being run directly. In this file, I have the code:
import doctest
doctest.testmod()
This goes through all the docstrings in the module and looks for lines that look like Python prompts with commands after them. The lines following are assumed to be the output of the command. If the commands output something else, the test is considered to have failed and the actual output is printed. Read the doctest module documentation for all the details.
One final note about doctests: They're useful, but they're not the most versatile or thorough tests available. For those, you'll want to read up on unittests (the unittest module).
Each python source file is a module. There are no "header" files. The basic idea is that when you import "foo" it'll load the code from "foo.py" (or a previously compiled version of it). You can then access the stuff from the foo module by saying foo.whatever.
There seem to be two ways for arranging things in Python code. Some projects use a flat layout, where all of the modules are at the top-level. Others use a hierarchy. You can import foo/bar/baz.py by importing "foo.bar.baz". The big gotcha with hierarchical layout is to have __init__.py in the appropriate directories (it can even be empty, but it should exist).
Classes are defined like this:
class MyClass(object):
def __init__(self, x):
self.x = x
def printX(self):
print self.x
To create an instance:
z = MyObject(5)
You can organize it in whatever way makes the most sense for your application. I don't exactly know what you're doing so I can't be certain what the best organization would be for you, but you can pretty much split it up as you see fit and just import what you need.
You can define classes in any file, and you can define as many classes as you would like in a script (unlike Java). There are no official header files (not like C or C++), but you can use config files to store info about connecting to a DB, whatever, and use configparser (a standard library function) to organize them.
It makes sense to keep like things in the same file, so if you have a GUI, you might have one file for the interface, and if you have a CLI, you might keep that in a file by itself. It's less important how your files are organized and more important how the source is organized into classes and functions.
This would be the place to look for that: http://docs.python.org/reference/.
First of all, compile and install pip: http://pypi.python.org/pypi/pip. It is like Ubuntu's apt-get. You run it via a Terminal by typing in pip install package-name. It has a database of packages, so you can install/uninstall stuff quite easily with it.
As for importing and "header" files, from what I can tell, if you run import foo, Python looks for foo.py in the current folder. If it's not there, it looks for eggs (folders unzipped in the Python module directory) and imports those.
As for defining classes and objects, here's a basic example:
class foo(foobar2): # I am extending a class, in this case 'foobar2'. I take no arguments.
__init__(self, the, list, of, args = True): # Instead, the arguments get passed to me. This still lets you define a 'foo()' objects with three arguments, only you let '__init__' take them.
self.var = 'foo'
def bar(self, args):
self.var = 'bar'
def foobar(self): # Even if you don't need arguments, never leave out the self argument. It's required for classes.
print self.var
foobar = foo('the', 'class', 'args') # This is how you initialize me!
Read more on this in the Python Reference, but my only tip is to never forget the self argument in class functions. It will save you a lot of debugging headaches...
Good luck!
There's no some fixed structure for Python programs, but you can take Django project as an example. Django project consists of one settings.py module, where global settings (like your example with DB connection properties) are stored and pluggable applications. Each application has it's own models.py module, which stores database models and, possibly, other domain specific objects. All the rest is up to you.
Note, that these advices are not specific to Python. In C/C++ you probably used similar structure and kept settings in XML. Just forget about headers and put settings in plain in .py file, that's all.