Develop Reference

Lambda multiple functions implementation in Python like Linux Pipeline command

I am implementing this requirement:
As part of a data processing pipeline, complete the implementation of the pipeline method:
The method should accept a variable number of functions, and it
should return a new function that accepts one parameter arg.
The returned function should call the first function in the pipeline
with the parameter arg, and call the second function with the result
of the first function.
The returned function should continue calling each function in the
pipeline in order, following the same pattern, and return the value
from the last function.
For example, pipeline(lambda x: x * 3, lambda x: x + 1, lambda x: x / 2) then calling the returned function with 3 should return 5.0.
My code
def pipeline(*funcs):
def helper(arg):
argCount = len(funcs)
if argCount > 0:
# Iterate over all the arguments and call each lamba's function
res = []
for elem in funcs:
if(len(res) > 0):
helper = elem(res.pop())
else:
helper = elem(arg)
res.append(helper)
helper = res.pop()
else:
return helper
print('before returning, helper value is: ', helper)
return helper
fun = pipeline(lambda x: x * 3, lambda x: x + 1, lambda x: x / 2)
print('final result: ', fun(3)) #should print 5.0
Question
None is returned. Why?
before returning, helper value is: 5.0
final result: None

The problem is that you don't execute a return right after you print. You do have a return in the else branch just before it, but not in the if block. Also, the return helper you have further below does not belong to the def helper function block, so you need one more return helper. I would in fact omit the else block, and just always do the return, like this:
def pipeline(*funcs):
def helper(arg):
argCount = len(funcs)
if argCount > 0:
# Iterate over all the arguments and call each lamba's function
res = []
for elem in funcs:
if(len(res) > 0):
helper = elem(res.pop())
else:
helper = elem(arg)
res.append(helper)
helper = res.pop()
print('before returning, helper value is: ', helper)
return helper # <-------
return helper
It is not really clear why you have a list res, since there is only one value to pass from one function to the next. You could just use arg for this purpose. Furthermore, you use helper in two different senses (function & value) which is quite confusing. The code can be simplified to this:
def pipeline(*funcs):
def helper(arg):
for elem in funcs:
arg = elem(arg)
return arg
return helper

Do not invent what is already available in Python
from functools import reduce
pipeline = [lambda x: x * 3, lambda x: x + 1, lambda x: x / 2]
val = reduce(lambda x, f: f(x), pipeline, 3)
print(val) # 5.0

You do: print('before returning, helper value is: ', helper)... and then do not actually return anything from helper, so it implicitly returns None.

def pipeline(*args):
def helper(num):
for i in args:
total=i(num)
num=total
return total
return helper
fun = pipeline(lambda x: x * 3, lambda x: x + 1, lambda x: x / 2)
print(fun(3)) #should print 5.0

how to nest multiple python function to create a sort of pipeline?

Hello i have 3 functions f1(), f2() and f3(). The output of the previous is the input of the next. meaning output = f3(f2(f1(data))).
instead of writing
def outp(data):
o1 = f1(data)
o2= f2(o1)
o3 = f3(02)
return o3
output=outp(data)
is there a way to this by simply providing a list of functions to some other general function and let it handle the chaining together?

You could simply run a for loop with an assignment:
>>> f1 = int
>>> f2 = float
>>> f3 = lambda x: x * 2
>>> i = '3'
>>> for func in (f1, f2, f3):
... i = func(i)
... print(i)
...
3
3.0
6.0

I stumbled across your question and I found it really interesting.
Here is my approach on a pipeline of functions:
def call_pipeline(transition_list, *args):
"""
transition_list = [func1, func2, func3...
]
- func1 output is the input for func2 etc
"""
result = args
for f in transition_list:
result = f(*result)
print(*result)
def f1(x, y):
return [x + y]
def f2(z):
return [z**2, z]
def f3(w, r):
return [w * r]
def f4(t):
return ["Final", t]
def test_pipeline():
transition_list = [f1, f2, f3, f4]
call_pipeline(transition_list, *[1, 2])
As long as you make sure that every function inside the pipeline returns a list and the next function can properly process the output of the previous one, this should work fine.

It is fairly easy to define a compose function that handles simple single-argument functions:
def compose(*args):
if not args:
return lambda x: x # Identity function
else:
return compose(args[0], compose(args[1:]))
outp = compose(f3, f2, f1) # Note the order of the arguments
You can also use the reduce function (functools.reduce in Python 3):
outp = reduce(lambda f, g: lambda x: f(g(x)), [f3, f2, f1], lambda x:x)
You can omit the third argument if you are certain the list of functions won't be empty.

You can define a composition function that operates on a sequence of functions:
def compose(a, b):
def fcn(*args, **kwargs):
return a(b(*args, **kwargs))
return fcn
def compose_all(*fcns):
return reduce(compose, fcns)
def compose_all_1(*fcns):
return reduce(compose, fcns, lambda x: x)
The function compose is the basic building block that takes two functions and returns their composition. With this elementary idiom you can extend this to an arbitrary sequence of functions in compose_all. The variant compose_all_1 works even on a 1- or 0-element input sequence.

You can define a pipeline builder, that you can reuse to build pipelines.
Each pipeline is a function that can be invoked with one argument or none (rarely used).
When a pipeline is called it sequentially run every functions with previous function return value as argument.
from functools import reduce
import warnings
# pipeline function builder
def pipe(*funcs):
"""
Usage:
pipe(
lambda x: x+1,
lambda x: x+20
)(50)
Order of execution of functions is FIFO.
FIFO = First In First Out
"""
# if no arguments return an identity function and warn the developer
if not funcs:
warnings.warn("""pipe() is useless when called without arguments.
Please provide functions as arguments or remove call to pipe().
""")
return lambda x=None: x # Identity function
def runner(value=None):
return reduce(lambda acc, curr: curr(acc), funcs, value)
return runner
# test 1
pipeline = pipe()
print(pipeline())
# None
# test 2
pipeline = pipe()
print(pipeline(8))
# 8
# test 3
pipeline = pipe(
lambda unused: 50
)
print(pipeline())
# 50
# test 4 -> Real Usage
pipeline = pipe(
lambda age: age >= 18,
lambda is_adult: 'Adult' if is_adult else 'Minor'
)
print(pipeline(12))
# Minor
print(pipeline(20))
# Adult

Expressive way compose generators in Python

I really like Python generators. In particular, I find that they are just the right tool for connecting to Rest endpoints - my client code only has to iterate on the generator that is connected the the endpoint. However, I am finding one area where Python's generators are not as expressive as I would like. Typically, I need to filter the data I get out of the endpoint. In my current code, I pass a predicate function to the generator and it applies the predicate to the data it is handling and only yields data if the predicate is True.
I would like to move toward composition of generators - like data_filter(datasource( )). Here is some demonstration code that shows what I have tried. It is pretty clear why it does not work, what I am trying to figure out is what is the most expressive way of arriving at the solution:
# Mock of Rest Endpoint: In actual code, generator is
# connected to a Rest endpoint which returns dictionary(from JSON).
def mock_datasource ():
mock_data = ["sanctuary", "movement", "liberty", "seminar",
"formula","short-circuit", "generate", "comedy"]
for d in mock_data:
yield d
# Mock of a filter: simplification, in reality I am filtering on some
# aspect of the data, like data['type'] == "external"
def data_filter (d):
if len(d) < 8:
yield d
# First Try:
# for w in data_filter(mock_datasource()):
# print(w)
# >> TypeError: object of type 'generator' has no len()
# Second Try
# for w in (data_filter(d) for d in mock_datasource()):
# print(w)
# I don't get words out,
# rather <generator object data_filter at 0x101106a40>
# Using a predicate to filter works, but is not the expressive
# composition I am after
for w in (d for d in mock_datasource() if len(d) < 8):
print(w)

data_filter should apply len on the elements of d not on d itself, like this:
def data_filter (d):
for x in d:
if len(x) < 8:
yield x
now your code:
for w in data_filter(mock_datasource()):
print(w)
returns
liberty
seminar
formula
comedy

More concisely, you can do this with a generator expression directly:
def length_filter(d, minlen=0, maxlen=8):
return (x for x in d if minlen <= len(x) < maxlen)
Apply the filter to your generator just like a regular function:
for element in length_filter(endpoint_data()):
...
If your predicate is really simple, the built-in function filter may also meet your needs.

You could pass a filter function that you apply for each item:
def mock_datasource(filter_function):
mock_data = ["sanctuary", "movement", "liberty", "seminar",
"formula","short-circuit", "generate", "comedy"]
for d in mock_data:
yield filter_function(d)
def filter_function(d):
# filter
return filtered_data

What I would do is define filter(data_filter) to receive a generator as input and return a generator with values filtered by data_filter predicate (regular predicate, not aware of generator interface).
The code is:
def filter(pred):
"""Filter, for composition with generators that take coll as an argument."""
def generator(coll):
for x in coll:
if pred(x):
yield x
return generator
def mock_datasource ():
mock_data = ["sanctuary", "movement", "liberty", "seminar",
"formula","short-circuit", "generate", "comedy"]
for d in mock_data:
yield d
def data_filter (d):
if len(d) < 8:
return True
gen1 = mock_datasource()
filtering = filter(data_filter)
gen2 = filtering(gen1) # or filter(data_filter)(mock_datasource())
print(list(gen2))
If you want to further improve, may use compose which was the whole intent I think:
from functools import reduce
def compose(*fns):
"""Compose functions left to right - allows generators to compose with same
order as Clojure style transducers in first argument to transduce."""
return reduce(lambda f,g: lambda *x, **kw: g(f(*x, **kw)), fns)
gen_factory = compose(mock_datasource,
filter(data_filter))
gen = gen_factory()
print(list(gen))
PS: I used some code found here, where the Clojure guys expressed composition of generators inspired by the way they do composition generically with transducers.
PS2: filter may be written in a more pythonic way:
def filter(pred):
"""Filter, for composition with generators that take coll as an argument."""
return lambda coll: (x for x in coll if pred(x))

Here is a function I have been using to compose generators together.
def compose(*funcs):
""" Compose generators together to make a pipeline.
e.g.
pipe = compose(func1, func2, func3)
result = pipe(range(0, 5))
"""
return lambda x: reduce(lambda f, g: g(f), list(funcs), x)
Where funcs is a list of generator functions. So your example would look like
pipe = compose(mock_datasource, data_filter)
print(list(pipe))
This is not original

Python Generator "chain" in a for loop

I'm trying to set up a "processing pipeline" for data that I'm reading in from a data source, and applying a sequence of operators (using generators) to each item as it is read.
Some sample code that demonstrates the same issue.
def reader():
yield 1
yield 2
yield 3
def add_1(val):
return val + 1
def add_5(val):
return val + 5
def add_10(val):
return val + 10
operators = [add_1, add_5, add_10]
def main():
vals = reader()
for op in operators:
vals = (op(val) for val in vals)
return vals
print(list(main()))
Desired : [17, 18, 19]
Actual: [31, 32, 33]
Python seems to not be saving the value of op each time through the for loop, so it instead applies the third function each time. Is there a way to "bind" the actual operator function to the generator expression each time through the for loop?
I could get around this trivially by changing the generator expression in the for loop to a list comprehension, but since the actual data is much larger, I don't want to be storing it all in memory at any one point.

You can force a variable to be bound by creating the generator in a new function. eg.
def map_operator(operator, iterable):
# closure value of operator is now separate for each generator created
return (operator(item) for item in iterable)
def main():
vals = reader()
for op in operators:
vals = map_operator(op, vals)
return vals
However, map_operator is pretty much identical to the map builtin (in python 3.x). So just use that instead.

You can define a little helper which composes the functions but in reverse order:
import functools
def compose(*fns):
return functools.reduce(lambda f, g: lambda x: g(f(x)), fns)
I.e. you can use compose(f,g,h) to generate a lambda expression equivalent to lambda x: h(g(f(x))). This order is uncommon, but ensures that your functions are applied left-to-right (which is probably what you expect):
Using this, your main becomes just
def main():
vals = reader()
f = compose(add_1, add_5, add_10)
return (f(v) for v in vals)

This may be what you want - create a composite function:
import functools
def compose(functions):
return functools.reduce(lambda f, g: lambda x: g(f(x)), functions, lambda x: x)
def reader():
yield 1
yield 2
yield 3
def add_1(val):
return val + 1
def add_5(val):
return val + 5
def add_10(val):
return val + 10
operators = [add_1, add_5, add_10]
def main():
vals = map(compose(operators), reader())
return vals
print(list(main()))

The reason for this problem is that you are creating a deeply nested generator of generators and evaluate the whole thing after the loop, when op has been bound to the last element in the list -- similar to the quite common "lambda in a loop" problem.
In a sense, your code is roughly equivalent to this:
for op in operators:
pass
print(list((op(val) for val in (op(val) for val in (op(val) for val in (x for x in [1, 2, 3])))))
One (not very pretty) way to fix this would be to zip the values with another generator, repeating the same operation:
def add(n):
def add_n(val):
return val + n
return add_n
operators = [add(n) for n in [1, 5, 10]]
import itertools
def main():
vals = (x for x in [1, 2, 3])
for op in operators:
vals = (op(val) for (val, op) in zip(vals, itertools.repeat(op)))
return vals
print(list(main()))

Python one-liner to call list of functions

I've got some old code where I stored lists of functions in Python as class attributes. These lists are used as a sort of event hook.
To call each function in the list with appropriate arguments, I've used one-liners, mixing map with lambda expressions. I'm now concerned that there is unnecessary overhead in using lambda expressions like this.. I guess the recommended way would be to drop both map and lambda and just use a standard for loop, for readability.
Is there a better (read faster) one-liner to do this, though?
For example:
class Foo:
"""Dummy class demonstrating event hook usage."""
pre = [] # list of functions to call before entering loop.
mid = [] # list of functions to call inside loop, with value
post = [] # list of functions to call after loop.
def __init__(self, verbose=False, send=True):
"""Attach functions when initialising class."""
self._results = []
if verbose:
self.mid.append( self._print )
self.mid.append( self._store )
if send:
self.post.append( self._send )
def __call__(self, values):
# call each function in self.pre (no functions there)
map( lambda fn: fn(), self.pre )
for val in values:
# call each function in self.mid, with one passed argument
map( lambda fn: fn(val), self.mid )
# call each fn in self.post, with no arguments
map( lambda fn: fn(), self.post )
def _print(self, value):
"""Print argument, when verbose=True."""
print value
def _store(self, value):
"""Store results"""
self._results.append(value)
def _send(self):
"""Send results somewhere"""
# create instance of Foo
foo = Foo(verbose=True)
# equivalent to: foo.__call__( ... )
foo( [1, 2, 3, 4] )
Is there a better way to write those one-liner map calls?

The recommended way is definitely to use for loops, however, if you insist on using map, then operator.methodcaller might be just what you need:
>>> def foo(*args):
... print 'foo',args
...
>>> def bar(*args):
... print 'bar',args
...
>>> from operator import methodcaller
>>>
>>> map(methodcaller('__call__',1,2,3),[foo,bar])
foo (1, 2, 3)
bar (1, 2, 3)
[None, None]
A word of caution about using map for this -- It won't work if you port your code to python 3 since map became lazy.
You could also use list comprehensions pretty trivially (and that works on python3 also):
[fn() for fn in self.pre]
[fn(val) for fn in self.mid]
etc.

First of all "I'm concerned that there is unnecessary overhead" is no way to optimise your code. Use a profiler to find the hotspots.
Secondly, your code could do with comments to let the reader know what is going on.
Finally, until proven otherwise, the following is a fine way to accomplish the task:
for func in self.pre: func()
#apply every function in self.mid to every value in values
for func,val in itertools.product(self.mid, values):
func(val)
If you wanted to capture the values, you could use a list comprehension; if you wanted to delay evaluation, you could use a generator expression.

>>> def chain(*fn):
>>> return lambda *args, **kwargs: [_(*args, **kwargs) for _ in fn]
>>>
>>> def add(x, y):
>>> return(x + y)
>>>
>>> def multiply(x, y):
>>> return(x * y)
>>>
>>> chained = chain(add, multiply)
>>> chained(2, 6)
[8, 12]