Develop Reference

How to make reduce function take three parameters in Python?

In python,reduce takes a function that accepts only two arguments.
Is there any elegant way to have a reduce that can take more than two arguments?
For example:
from operator import add
list = [1,add,2,add,3,add,4]
func = lambda operand1,operator,operand2: operator(operand1,operand2)
reduce(func,list) # mock reduce. Expected result is 10(1+2+3+4=10)
EDIT:
The reduce3 function is used to calculate the result of the abstract syntax tree(AST).
Every children of AST can be either a node or another tree.
For simplicity, assume that leaf nodes can only be number,addition symbol(+) or subtraction symbol(-).To calculate the result, one intuitive idea is to firstly work out the result of every nodes(can be implemented by recursively calls the method), and then combine them together.
For example, here's a simple AST:
Tree(
Node(1),
Node("+"),
Tree(
Node(2),
"+",
Node(3)
),
Node("-"),
Node(4)
)
After recursion, we get the following result.
If there is a reduce take 3 arguments, we can easily combine the result.
Tree(
Node(1),
Node("+"),
Node(5)
Node("-"),
Node(4)
)

This gives the exact result without 3 arguments.
It creates a partial function on each argument if arguments are like 1, add and calls the function if arguments are like partial,2
from operator import add
from functools import reduce, partial
lst = [1,add,2,add,3,add,4]
def operation_reduce(x, y):
if callable(x):
return x(y)
else:
return partial(y, x)
print(reduce(operation_reduce, lst)) #10

reduce can't be changed, but you could provide a callable that handles the differing inputs:
from functools import reduce
from operator import add, sub
L = [1, add, 2, add, 3, add, 4, sub]
class Reducer:
def __init__(self):
self.pending = 0
def __call__(self, total, value):
if callable(value):
total = value(total, self.pending)
else:
self.pending = value
return total
func = Reducer()
result = reduce(func, L, 0)
print(result)
Or you could batch the inputs in tuples, and use a callable that could handle those
def reducer(total, next_):
value, operation = next_
return operation(total, value)
result = reduce(reducer, zip(L[::2], L[1::2]), 0)
print(result)

I'd be interested to hear what is your desired use case as I believe reduce taking more than 2 arguments is not elegant, period. Here's my rationale (though it's a bit hard arguing in abstractness, with no concrete use case):
Assume the reducing operation takes 3 arguments as in your example. Let's also make initial accumulator explicit, reduce3(func,list,acc). Then in every call to func:
first argument is accumulator as usual
second argument is always an element at the odd position in the list
third argument is always at an even position in the list
Even positioned elements from the list are treated by the func differently than odd positioned elements. We can ever really need such a reduce3 function if the elements at odd and even positions are of different type!*
It is the case in the example. Mixing elements of different type in one list requires more caution not to make a mistake (mess up the order of some 2 elements and boom the reduce3 breaks) and IMO should be avoided in general.
Modifying your example:
list = [2,3,4] # we'l include 1 as initial acc
ops = [add, add, add]
func = lambda acc, pair : pair[0](acc, pair[1])
reduce(func, zip(ops, list), 1) # result is 10(1+2+3+4=10)
Reducing over a list of tuples is a general solution to your question. If mathematical expressions is what you want to evaluate you should consider their tree structure, which gets lost when collapsed to a list.
Edit:
Indeed you want to evaluate mathematical expressions. Consider changing the way you are parsing these expressions so that for the example the AST looks like:
Tree("-",
Tree("+",
Node(1),
Tree("+",
Node(2),
Node(3))),
Node(4))
# -
# / \
# + 4
# / \
# 1 +
# / \
# 2 3
That is internal nodes are always tagged with binary operation and it is only leafs that carry numbers. Now reducing tree to a value is trivial but at a cost of parsing doing most of the work.
# Here assuming some specifics of the Tree and Node classes.
# strToOpt is mapping of operation symbols to functions.
def evaluate(expr): # expr : Tree or Node
if (isinstance(expr, Node)): # expr : Node
return expr.x
else: # expr : Tree
return strToOp[expr.op](evaluate(expr.left), evaluate(expr.right))
If you allow for non-binary tree nodes then swap the "else" branch with reduce like reduce(strToOpt[expr.op], expr.children).
I understand that this is very different approach. Reduce3 shouldn't be that hard to implement if you want to stick to it, right? For some help on parsing math from scratch in python you could look at the old project of mine BUT it's amateur code written by a student (me). On the other hand would be unfair not to share, here you go: parsing
*Not necessarily of different type in a programming sense, but certainly of different type or meaning to us.

Manual product with unknown number of arguments

The following examples give the same result:
A.
product = []
for a in "abcd":
for b in "xy":
product.append((a,b))
B.
from itertools import product
list(product("abcd","xy"))
How can I calculate the cartesian product like in example A when I don't know the number of arguments n?
REASON I'm asking this:
Consider this piece of code:
allocations = list(product(*strategies.values()))
for alloc in allocations:
PWC[alloc] = [a for (a,b) in zip(help,alloc) if coalitions[a] >= sum(b)]
The values of the strategies dictionary are list of tuples, help is an auxiliary variable (a list with the same length of every alloc) and coalitions is another dictionary that assigns to the tuples in help some numeric value.
Since strategies values are sorted, I know that the if statement won't be true anymore after a certain alloc. Since allocations is a pretty big list, I would avoid tons of comparisons and tons of sums if I could use the example algorithm A.

You can do:
items = ["abcd","xy"]
from itertools import product
list(product(*items))
The list items can contain an arbitrary number of strings and it'll the calculation with product will provide you with the Cartesian product of those strings.
Note that you don't have to turn it into a list - you can iterate over it and stop when you no longer wish to continue:
for item in product(*items):
print(item)
if condition:
break

If you just want to abort the allocations after you hit a certain condition, and you want to avoid generating all the elements from the cartesian product for those, then simply don’t make a list of all combinations in the first place.
itertools.product is lazy that means that it will only generate a single value of the cartesian product at a time. So you never need to generate all elements, and you also never need to compare the elements then. Just don’t call list() on the result as that would iterate the whole sequence and store all possible combinations in memory:
allocations = product(*strategies.values())
for alloc in allocations:
PWC[alloc] = [a for (a,b) in zip(help,alloc) if coalitions[a] >= sum(b)]
# check whether you can stop looking at more values from the cartesian product
if someCondition(alloc):
break
It’s just important to note how itertools.product generates the values, what pattern it follows. It’s basically equivalent to the following:
for a in firstIterable:
for b in secondIterable:
for c in thirdIterable:
…
for n in nthIterable:
yield (a, b, c, …, n)
So you get an increasing pattern from the left side of your iterables. So make sure that you order the iterables in a way that you can correctly specify a break condition.

Algorithm to reverse list items in place?

I'm learning about data structures and algorithm efficiency in my CS class right now, and I've created an algorithm to return a new list of items that in reverse order from the original list. I'm trying to figure out how to do this in place with Python lists using recursion, but the solution is eluding me. Here is my code:
def reverse_list(in_list):
n = len(in_list)
print('List is: ', in_list)
if n <= 2:
in_list[0], in_list[-1] = in_list[-1], in_list[0]
return in_list
else:
first = [in_list[0]]
last = [in_list[-1]]
temp = reverse_list(in_list[1:-1])
in_list = last + temp + first
print('Now list is: ', in_list)
return in_list
if __name__ == '__main__':
list1 = list(range(1, 12))
list2 = reverse_list(list1)
print(list2)
As a side note, this is O(n) in average case due to it being n/2 right?

You really don't want to be using recursion here as it doesn't really simplify the problem for the extra overhead involved with the function calls:
list3 = list(range(1, 10))
for i in range(len(list3)/2):
list3[i], list3[len(list3)-i-1] = list3[len(list3)-i-1], list3[i]
print(list3)
if you wanted to approach it using recursion, you'd pass in an index counter to each call of your function till you had gotten half way through the list. This would give you O(n/2) runtime.

Your n <= 2 case is good. It is "in-place" in both of the usual senses. It modifies the same list as is passed in, and it only uses a constant amount of memory in addition to the list passed in.
Your general case is where you have problems. It uses a lot of extra memory (n/2 levels of recursion, each of which holds two lists across the recursive call, so all those lists at all levels have to exist at once), and it does not modify the list passed in (rather, it returns a new list).
I don't want to do too much for you, but the key to solving this is to pass one (or two if you prefer) extra parameter(s) into the recursive step. Either an optional parameter to your function, or define a recursive helper function that has them. Use this to indicate what region of the list to reverse in-place.
As someone mentioned in comments, CPython doesn't have a tail-recursion optimization. This means that no algorithm that uses call-recursion like this is ever going to be truly "in-place", since it will use linear amounts of stack. But "in-place" in the sense of "modifies the original list instead of returning a new one" is certainly doable.

Python: Sorting a list by class objects

Working on a project for CS1, and I am close to cracking it, but this part of the code has stumped me! The object of the project is to create a list of the top 20 names in any given year by referencing a file with thousands of names on it. Each line in each file contains the name, gender, and how many times it occurs. This file is seperated by gender (so female names in order of their occurences followed by male names in order of their occurences). I have gotten the code to a point where each entry is contained within a class in a list (so this list is a long list of memory entries). Here is the code I have up to this point.
class entry():
__slots__ = ('name' , 'sex' , 'occ')
def mkEntry( name, sex, occ ):
dat = entry()
dat.name = name
dat.sex = sex
dat.occ = occ
return dat
##test = mkEntry('Mary', 'F', '7065')
##print(test.name, test.sex, test.occ)
def readFile(fileName):
fullset = []
for line in open(fileName):
val = line.split(",")
sett = mkEntry(val[0] , val[1] , int(val[2]))
fullset.append(sett)
return fullset
fullset = readFile("names/yob1880.txt")
print(fullset)
What I am wondering if I can do at this point is can I sort this list via usage of sort() or other functions, but sort the list by their occurrences (dat.occ in each entry) so in the end result I will have a list sorted independently of gender and then at that point I can print the first entries in the list, as they should be what I am seeking. Is it possible to sort the list like this?

Yes, you can sort lists of objects using sort(). sort() takes a function as an optional argument key. The key function is applied to each element in the list before making the comparisons. For example, if you wanted to sort a list of integers by their absolute value, you could do the following
>>> a = [-5, 4, 6, -2, 3, 1]
>>> a.sort(key=abs)
>>> a
[1, -2, 3, 4, -5, 6]
In your case, you need a custom key that will extract the number of occurrences for each object, e.g.
def get_occ(d): return d.occ
fullset.sort(key=get_occ)
(you could also do this using an anonymous function: fullset.sort(key=lambda d: d.occ)). Then you just need to extract the top 20 elements from this list.
Note that by default sort returns elements in ascending order, which you can manipulate e.g. fullset.sort(key=get_occ, reverse=True)

This sorts the list by using the occ property in descending order:
fullset.sort(key=lambda x: x.occ, reverse=True)

You mean you want to sort the list only by the occ? sort() has a parameter named key, you can do like this:
fullset.sort(key=lambda x: x.occ)

I think you just want to sort on the value of the 'occ' attribute of each object, right? You just need to use the key keyword argument to any of the various ordering functions that Python has available. For example
getocc = lambda entry: entry.occ
sorted(fullset, key=getocc)
# or, for in-place sorting
fullset.sort(key=getocc)
or perhaps some may think it's more pythonic to use operator.attrgetter instead of a custom lambda:
import operator
getocc = operator.attrgetter('occ')
sorted(fullset, key=getocc)
But it sounds like the list is pretty big. If you only want the first few entries in the list, sorting may be an unnecessarily expensive operation. For example, if you only want the first value you can get that in O(N) time:
min(fullset, key=getocc) # Same getocc as above
If you want the first three, say, you can use a heap instead of sorting.
import heapq
heapq.nsmallest(3, fullset, key=getocc)
A heap is a useful data structure for getting a slice of ordered elements from a list without sorting the whole list. The above is equivalent to sorted(fullset, key=getocc)[:3], but faster if the list is large.
Hopefully it's obvious you can get the three largest with heapq.nlargest and the same arguments. Likewise you can reverse any of the sorts or replace min with max.

Python converting nested loops to simple line

In Python 2.7.x I have two lists I would like a function that returns the first value (not index) as shown below
def first_incorrect_term(polynomial, terms):
for index in range(len(polynomial), len(terms)):
if evaluate(polynomial, index) != terms[index-1]:
return evaluate(polynomial, index)
Let us assume evaluate is a function that works. I would like to replace these three lines which looks Object Oriented into something that uses the "find" or some such function in Python.
Basically I am iterating through the indices of the second list beyond the number terms in the polynomial (as I am confident the first X terms will match), evaluating it and comparing with the expected terms. For the first instance where the terms do not match I would like the evaluated polynomial returned.
I am looking for a replacement of these 3 lines using a Python find/lambda or some such thing, this is because I can definitely see I am not using the Python power as described for example in the link
PS: This is somewhat related to a Project Euler problem, however I have solved it using the snippet above and would like to improve my "Python" skills :)

Firstly, use yield to make a generator version of your function:
def incorrect_terms(polynomial, terms):
for index in range(len(polynomial), len(terms)):
eval = evaluate(polynomial,index)
if eval != terms[index-1]:
yield (polynomial, index, eval)
Then the first result is the first mismatch:
mismatches = incorrect_terms(polynomial, terms)
first_mismatch = mismatches.next()
I think you actually want to iterate over all the values of terms, not the values after polynomial's length, in which case you can zip:
results = (evaluate(polynomial,index) for index in count(0))
pairsToCompare = itertools.izip(results, terms)
mismatches = (pair for pair in pairsToCompare if pair[0] != pair[1])
first_mismatch = mismatches.next()
Assuming here that evaluate(polynomial, n) is calculating the nth term for a given polynomial, and that these are being compared with the values in terms.

I would do it using generator expressions, but they don't fit in one line as well:
def first_incorrect_term(polynomial, terms):
evaled = ((index, evaluate(polynomial, index)) for index in range(len(polynomial), len(terms)))
return next((val for index, val in evaled if val != terms[index-1]), None)