Develop Reference

Wrap a package/module/callable to a new one with added attributes

I want to allow two different access to a particular functions. I'm not the author of sample.test_foo and I am making use of the #service decorator to work in these two ways.
The method say sample.test_foo is decorated by #service and when directly imported and accessed, it will run certain code and return the result.
Other mode I want is to make the function use a cache to fetch the data. How I'm doing this is, I'm asking the users to set a context variable - cache_mode_modules - and users are expected to add "sample". The decorator checks if the parent module is in the env, then it fetches from cache otherwise calls the method. How I fetch from cache etc are bit complex and isn't related to the question.
I really want to change the second mode of access. I want to create an API which takes in "sample" module and returns me a different callable.
import sample
new_sample = magic_wrapper(sample)
sample.test_foo() -> calls the actual func
new_sample.foo() -> calls the cache
The magic_wrapper need to set some module variable, say "_cache_mode=True" which I can use it in my decorator and decide how to access.
Note that even if package is passed to the magic_wrapper, I need the functionality to work.
I tried the below one
In [1]: def test(module):
...: SPEC_OS = importlib.util.find_spec(module)
...: a = importlib.util.module_from_spec(SPEC_OS)
...: SPEC_OS.loader.exec_module(a)
...: a._cache_mode = True
...: return a
It works for module but not for package. Can someone suggest if it is possible to have the magic_wrapper ?

From your comment, I think you are trying to change the behavior of specific functions in the package/module by overriding defaults and not actually make changes to the package/module core functionality. If that is right, functions are first class objects in Python so they can be passed around just like any other object and thus set as a default argument to another function.
Example:
def function_in_module(b, default=False):
if default:
return -b
return b
def magic_wrapper(b, func=function_in_module):
return func(b, default=True)
print(magic_wrapper(10)) # prints -10
print(function_in_module(10)) # prints 10
Same is true with an imported from an imported module. If the above function_in_module were actually in module, you could make a file magic_wrapper.py:
from module import function_in_module
def magic_wrapper(b, func=function_in_module):
return func(b, default=True)
and in main.py:
from module import function_in_module
from magic_wrapper import magic_wrapper
print(magic_wrapper(10)) # prints -10
print(function_in_module(10)) # prints 10

Refactoring a Python class to take a dynamic parameter without changing calling pattern dramatically

I have a Python class in a base_params.py module within an existing codebase, which looks like this:
import datetime
class BaseParams:
TIMESTAMP = datetime.datetime.now()
PATH1 = f'/foo1/bar/{TIMESTAMP}/baz'
PATH2 = f'/foo2/bar/{TIMESTAMP}/baz'
Callers utilize it this way:
from base_params import BaseParams as params
print(params.PATH1)
Now, I want to replace the TIMESTAMP value with one that is dynamically specified at runtime (through e.g. CLI arguments).
Is there a way to do this in Python without requiring my callers to refactor their code in a dramatic way? This is currently confounding me because the contents of the class BaseParams get executed at 'compile' time, so there is no opportunity there to pass in a dynamic value as it's currently structured. And in some of my existing code, this object is being treated as "fully ready" at 'compile' time, for example, its values are used as function argument defaults:
def some_function(value1, value2=params.PATH1):
...
I am wondering if there is some way to work with Python modules and/or abuse Python's __special_methods__ to get this existing code pattern working more or less as-is, without a deeper refactoring of some kind.
My current expectation is "this is not really possible" because of that last example, where the default value is being specified in the function signature. But I thought I should check with the Python wizards to see if there may be a suitably Pythonic way around this.

Yes, you just need to make sure that the command line argument is parsed before the class is defined and before any function that uses the class's attribute as a default argument is defined (but that should already be the case).
(using sys.argv for sake of simplicity. It is better to use an actual argument parser such as argparse)
import datetime
import sys
class BaseParams:
try:
TIMESTAMP = sys.argv[1]
except IndexError:
TIMESTAMP = datetime.datetime.now()
PATH1 = f'/foo1/bar/{TIMESTAMP}/baz'
PATH2 = f'/foo2/bar/{TIMESTAMP}/baz'
print(BaseParams.TIMESTAMP)
$ python main.py dummy-argument-from-cli
outputs
dummy-argument-from-cli
while
$ python main.py
outputs
2021-06-26 02:32:12.882601

You can still totally replace the value of a class attribute after the class has been defined:
BaseParams.TIMESTAMP = <whatever>
There are definitely some more "magic" things you can do though, such as a class factory of some kind. Since Python 3.7 you can also take advantage of module __getattr__ to create a kind of factory for the BaseParams class (PEP 562)
In base_params.py you might rename BaseParams to _BaseParams or BaseParamsBase or something like that :)
Then at the module level define:
def __getattr__(attr):
if attr == 'BaseParams':
params = ... # whatever code you need to determine class attributes for BaseParams
return type('BaseParams', (_BaseParams,), params)
raise AttributeError(attr)

How to properly deal with optional features in python

I'm working on python packages that implement scientific models and I'm wondering what is the best way to handle optional features.
Here's the behavior I'd like:
If some optional dependencies can't be imported (plotting module on a headless machine for example), I'd like to disable the functions using these modules in my classes, warn the user if he tries to use them and all that without breaking the execution.
so the following script would work in any cases:
mymodel.dostuff()
mymodel.plot() <= only plots if possible, else display log an error
mymodel.domorestuff() <= get executed regardless of the result of the previous statement
So far the options I see are the following:
check in the __init __.py for available modules and keep a list of
them (but how to properly use it in the rest of the package?)
for each function relying on optional dependencies have a try import ...
except ... statement
putting functions depending on a particular module in a separated file
These options should work, but they all seem to be rather hacky and hard to maintain. what if we want to drop a dependency completely? or make it mandatory?

The easiest solution, of course, is to simply import the optional dependencies in the body of the function that requires them. But the always-right PEP 8 says:
Imports are always put at the top of the file, just after any module
comments and docstrings, and before module globals and constants.
Not wanting to go against the best wishes of the python masters, I take the following approach, which has several benefits...
First, import with an try-except
Say one of my functions foo needs numpy, and I want to make it an optional dependency. At the top of the module, I put:
try:
import numpy as _numpy
except ImportError:
_has_numpy = False
else:
_has_numpy = True
Here (in the except block) would be the place to print a warning, preferably using the warnings module.
Then throw the exception in the function
What if the user calls foo and doesn't have numpy? I throw the exception there and document this behaviour.
def foo(x):
"""Requires numpy."""
if not _has_numpy:
raise ImportError("numpy is required to do this.")
...
Alternatively you can use a decorator and apply it to any function requiring that dependency:
#requires_numpy
def foo(x):
...
This has the benefit of preventing code duplication.
And add it as an optional dependency to your install script
If you're distributing code, look up how to add the extra dependency to the setup configuration. For example, with setuptools, I can write:
install_requires = ["networkx"],
extras_require = {
"numpy": ["numpy"],
"sklearn": ["scikit-learn"]}
This specifies that networkx is absolutely required at install time, but that the extra functionality of my module requires numpy and sklearn, which are optional.
Using this approach, here are the answers to your specific questions:
What if we want to make a dependency mandatory?
We can simply add our optional dependency to our setup tool's list of required dependencies. In the example above, we move numpy to install_requires. All of the code checking for the existence of numpy can then be removed, but leaving it in won't cause your program to break.
What if we want to drop a dependency completely?
Simply remove the check for the dependency in any function that previously required it. If you implemented the dependency check with a decorator, you could just change it so that it simply passes the original function through unchanged.
This approach has the benefit of placing all of the imports at the top of the module so that I can see at a glance what is required and what is optional.

I would use the mixin style of composing a class. Keep optional behaviour in separate classes and subclass those classes in your main class. If you detect that the optional behaviour is not possible then create a dummy mixin class instead. For example:
model.py
import numpy
import plotting
class Model(PrimaryBaseclass, plotting.Plotter):
def do_something(self):
...
plotting.py
from your_util_module import headless as _headless
__all__ = ["Plotter"]
if _headless:
import warnings
class Plotter:
def plot(self):
warnings.warn("Attempted to plot in a headless environment")
else:
class Plotter:
"""Expects an attribute called `data' when plotting."""
def plot(self):
...
Or, as an alternative, use decorators to describe when a function might be unavailable.
eg.
class unavailable:
def __init__(self, *, when):
self.when = when
def __call__(self, func):
if self.when:
def dummy(self, *args, **kwargs):
warnings.warn("{} unavailable with current setup"
.format(func.__qualname__))
return dummy
else:
return func
class Model:
#unavailable(when=headless)
def plot(self):
...

replace functions with a different function in python

I have a function called get_account(param1,param2)
in run time I need to replace this function with the function mock_get_account(param1,param2)
so when the system calls get_account(param1,param2) I need the mock_get_account(param1,param2) to be called instead.
I tried this code:
package.get_account=self.mock_get_account
package.get_account(x,y)
but still the get_account runs instead of the mock_get_account
I'm new to python and I don't know if this is even possible but I have seen the lamda function and I know that function programming is possible in python. Thanks
Edit:
if i do the following:
package.get_account=self.mock_get_account
package.get_account(x,y)
then every thing is ok, meaning the mock_get_account is called, but in mu code I the following code i do a post self.client.post(url, data=data, follow=True) that triggers the package.get_account and this is not working:
package.get_account=self.mock_get_account
package.get_account(x,y)
#the folowing call will trigger the package.get_account(x,y) function in a django url #callback
self.client.post(url, data=data, follow=True)
meaning it calls the old function, also get_account(param1,param2) is defined in side a file, and is not a child function of a class and mock_get_account(self,param1,param2) is defined in a class Test and is called inside the Test.test_account - function

This is very opinionated and does not (directly) answer your question, but hopefully solves your problem.
A better practice is to use a subclass with your mock_get_account's implementation override the parent get_account method, example below:
class A(object):
def get_account(self):
return 1
def post(self):
return self.get_account()
class B(A):
def get_account(self):
return 2 # your original mock_get_account implementation
a = A()
print(a.get_account())
b = B()
print(b.post()) # this .post will trigger the overridden implementation of get_account

My guess is that the code implementing self.client.post has access to get_account through an import statement that looks like from package import get_account.
from package import get_account will first load package if it hasn't been already imported. Then it will look for a name get_account in that module, and whatever object that was bound to will be bound in the importing package's namespace, also under the name get_account. Thereafter the two names refer to the same object, but they are not the same name.
So if your mocking code comes along after this point, it sets the name get_account in package to instead refer to mock_get_account. But that'll only affect code that reads get_account from package again; anything that's already imported that name specially won't be affected.
If the code behind self.client.post instead had access only to package through import package, and was calling package.get_account it would work, because it's then only the object representing the package module that has been bound in the importing module's namespace. package.get_account would be reading an attribute of that object, and so would get whatever the current value is. If the from package import get_account appeared at function local scope rather than module scope, then this would behave similarly.
If I'm correct and your code is structured this way, then it's unfortunately not really package.get_account you need to rebind to a mock, but the get_account name in the module where self.client.post comes from (as well as any other modules which may call it).

How is a Python project set up?

I am doing some heavy commandline stuff (not really web based) and am new to Python, so I was wondering how to set up my files/folders/etc. Are there "header" files where I can keep all the DB connection stuff?
How/where do I define classes and objects?

Just to give you an example of a typical Python module's source, here's something with some explanation. This is a file named "Dims.py". This is not the whole file, just some parts to give an idea what's going on.
#!/usr/bin/env python
This is the standard first line telling the shell how to execute this file. Saying /usr/bin/env python instead of /usr/bin/python tells the shell to find Python via the user's PATH; the desired Python may well be in ~/bin or /usr/local/bin.
"""Library for dealing with lengths and locations."""
If the first thing in the file is a string, it is the docstring for the module. A docstring is a string that appears immediately after the start of an item, which can be accessed via its __doc__ property. In this case, since it is the module's docstring, if a user imports this file with import Dims, then Dims.__doc__ will return this string.
# Units
MM_BASIC = 1500000
MILS_BASIC = 38100
IN_BASIC = MILS_BASIC * 1000
There are a lot of good guidelines for formatting and naming conventions in a document known as PEP (Python Enhancement Proposal) 8. These are module-level variables (constants, really) so they are written in all caps with underscores. No, I don't follow all the rules; old habits die hard. Since you're starting fresh, follow PEP 8 unless you can't.
_SCALING = 1
_SCALES = {
mm_basic: MM_BASIC,
"mm": MM_BASIC,
mils_basic: MILS_BASIC,
"mil": MILS_BASIC,
"mils": MILS_BASIC,
"basic": 1,
1: 1
}
These module-level variables have leading underscores in their names. This gives them a limited amount of "privacy", in that import Dims will not let you access Dims._SCALING. However, if you need to mess with it, you can explicitly say something like import Dims._SCALING as scaling.
def UnitsToScale(units=None):
"""Scales the given units to the current scaling."""
if units is None:
return _SCALING
elif units not in _SCALES:
raise ValueError("unrecognized units: '%s'." % units)
return _SCALES[units]
UnitsToScale is a module-level function. Note the docstring and the use of default values and exceptions. No spaces around the = in default value declarations.
class Length(object):
"""A length. Makes unit conversions easier.
The basic, mm, and mils properties can be used to get or set the length
in the desired units.
>>> x = Length(mils=1000)
>>> x.mils
1000.0
>>> x.mm
25.399999999999999
>>> x.basic
38100000L
>>> x.mils = 100
>>> x.mm
2.54
"""
The class declaration. Note the docstring has things in it that look like Python command line commands. These care called doctests, in that they are test code in the docstring. More on this later.
def __init__(self, unscaled=0, basic=None, mm=None, mils=None, units=None):
"""Constructs a Length.
Default contructor creates a length of 0.
>>> Length()
Length(basic=0)
Length(<float>) or Length(<string>) creates a length with the given
value at the current scale factor.
>>> Length(1500)
Length(basic=1500)
>>> Length("1500")
Length(basic=1500)
"""
# Straight copy
if isinstance(unscaled, Length):
self._x = unscaled._x
return
# rest omitted
This is the initializer. Unlike C++, you only get one, but you can use default arguments to make it look like several different constructors are available.
def _GetBasic(self): return self._x
def _SetBasic(self, x): self._x = x
basic = property(_GetBasic, _SetBasic, doc="""
This returns the length in basic units.""")
This is a property. It allows you to have getter/setter functions while using the same syntax as you would for accessing any other data member, in this case, myLength.basic = 10 does the same thing as myLength._SetBasic(10). Because you can do this, you should not write getter/setter functions for your data members by default. Just operate directly on the data members. If you need to have getter/setter functions later, you can convert the data member to a property and your module's users won't need to change their code. Note that the docstring is on the property, not the getter/setter functions.
If you have a property that is read-only, you can use property as a decorator to declare it. For example, if the above property was to be read-only, I would write:
#property
def basic(self):
"""This returns the length in basic units."""
return self._x
Note that the name of the property is the name of the getter method. You can also use decorators to declare setter methods in Python 2.6 or later.
def __mul__(self, other):
"""Multiplies a Length by a scalar.
>>> Length(10)*10
Length(basic=100)
>>> 10*Length(10)
Length(basic=100)
"""
if type(other) not in _NumericTypes:
return NotImplemented
return Length(basic=self._x * other)
This overrides the * operator. Note that you can return the special value NotImplemented to tell Python that this operation isn't implemented (in this case, if you try to multiply by a non-numeric type like a string).
__rmul__ = __mul__
Since code is just a value like anything else, you can assign the code of one method to another. This line tells Python that the something * Length operation uses the same code as Length * something. Don't Repeat Yourself.
Now that the class is declared, I can get back to module code. In this case, I have some code that I want to run only if this file is executed by itself, not if it's imported as a module. So I use the following test:
if __name__ == "__main__":
Then the code in the if is executed only if this is being run directly. In this file, I have the code:
import doctest
doctest.testmod()
This goes through all the docstrings in the module and looks for lines that look like Python prompts with commands after them. The lines following are assumed to be the output of the command. If the commands output something else, the test is considered to have failed and the actual output is printed. Read the doctest module documentation for all the details.
One final note about doctests: They're useful, but they're not the most versatile or thorough tests available. For those, you'll want to read up on unittests (the unittest module).

Each python source file is a module. There are no "header" files. The basic idea is that when you import "foo" it'll load the code from "foo.py" (or a previously compiled version of it). You can then access the stuff from the foo module by saying foo.whatever.
There seem to be two ways for arranging things in Python code. Some projects use a flat layout, where all of the modules are at the top-level. Others use a hierarchy. You can import foo/bar/baz.py by importing "foo.bar.baz". The big gotcha with hierarchical layout is to have __init__.py in the appropriate directories (it can even be empty, but it should exist).
Classes are defined like this:
class MyClass(object):
def __init__(self, x):
self.x = x
def printX(self):
print self.x
To create an instance:
z = MyObject(5)

You can organize it in whatever way makes the most sense for your application. I don't exactly know what you're doing so I can't be certain what the best organization would be for you, but you can pretty much split it up as you see fit and just import what you need.
You can define classes in any file, and you can define as many classes as you would like in a script (unlike Java). There are no official header files (not like C or C++), but you can use config files to store info about connecting to a DB, whatever, and use configparser (a standard library function) to organize them.
It makes sense to keep like things in the same file, so if you have a GUI, you might have one file for the interface, and if you have a CLI, you might keep that in a file by itself. It's less important how your files are organized and more important how the source is organized into classes and functions.

This would be the place to look for that: http://docs.python.org/reference/.
First of all, compile and install pip: http://pypi.python.org/pypi/pip. It is like Ubuntu's apt-get. You run it via a Terminal by typing in pip install package-name. It has a database of packages, so you can install/uninstall stuff quite easily with it.
As for importing and "header" files, from what I can tell, if you run import foo, Python looks for foo.py in the current folder. If it's not there, it looks for eggs (folders unzipped in the Python module directory) and imports those.
As for defining classes and objects, here's a basic example:
class foo(foobar2): # I am extending a class, in this case 'foobar2'. I take no arguments.
__init__(self, the, list, of, args = True): # Instead, the arguments get passed to me. This still lets you define a 'foo()' objects with three arguments, only you let '__init__' take them.
self.var = 'foo'
def bar(self, args):
self.var = 'bar'
def foobar(self): # Even if you don't need arguments, never leave out the self argument. It's required for classes.
print self.var
foobar = foo('the', 'class', 'args') # This is how you initialize me!
Read more on this in the Python Reference, but my only tip is to never forget the self argument in class functions. It will save you a lot of debugging headaches...
Good luck!

There's no some fixed structure for Python programs, but you can take Django project as an example. Django project consists of one settings.py module, where global settings (like your example with DB connection properties) are stored and pluggable applications. Each application has it's own models.py module, which stores database models and, possibly, other domain specific objects. All the rest is up to you.
Note, that these advices are not specific to Python. In C/C++ you probably used similar structure and kept settings in XML. Just forget about headers and put settings in plain in .py file, that's all.