Basically, if I were to write a function with variable return elements, like so:
def func(elem1=True, elem2=True, elem3=True, elem4=False):
x = MyClass()
ret = []
if elem1:
ret.extend([x.func1()])
if elem2:
ret.extend([x.obj1])
if elem3:
ret.extend([x.func2().attr1])
if elem4:
ret.extend(x.list_obj3)
return ret
Things get rather long and windy. Is it possible to do something like this perhaps:
def func(elem1=True, elem2=True, elem3=True, elem4=False):
x = MyClass()
return [x.func1() if elem1,
x.obj1 if elem2,
x.func2().attr1 if elem3,
x.list_obj3 if elem4]
How neat is that!?
I know this can be done:
def func(elem1=True, elem2=True, elem3=True, elem4=False):
x = MyClass()
ret = [x.func1(), x.obj1, x.func2().attr1, x.list_obj3]
choices = [elem1, elem2, elem3, elem4]
return [r for i, r in enumerate(ret) if choices[i]]
but I would like to not calculate the elements if the user does not want them; it is a little expensive to calculate some of them.
If you hide your operations in lambdas then you can use lazy evaluation:
def func(elem1=True, elem2=True, elem3=True, elem4=False):
x = MyClass()
return [L() for inc,L in (
(elem1, lambda: x.func1()),
(elem2, lambda: x.obj1),
(elem3, lambda: x.func2().attr1),
(elem4, lambda: x.list_obj3),
) if inc]
Asking a slightly different question, can you get behaviour like matlab/octave, where you only calculate the first two results if you are assigning to two variables, without computing results 3 and 4?
For example:
a, b = func()
Python can't quite do it since func() doesn't know how many return values it wants, but you can get close using:
from itertools import islice
def func():
x = MyClass()
yield x.fun c1()
yield x.obj1
yield x.func2().attr1
yield x.list_obj3
a, b = islice(func(), 2)
I'm not sure it is better, but you could add array indexing semantics using a decorator, which would allow you to write:
#sliceable
def func():
...
a, b = func()[:2]
This is easy enough to implement:
from itertools import islice
class SlicedIterator(object):
def __init__(self, it):
self.it = it
def __iter__(self):
return self.it
def __getitem__(self, idx):
if not isinstance(idx, slice):
for _ in range(idx): next(self.it)
return next(self.it)
return list(islice(self.it, idx.start, idx.stop, idx.step))
def sliceable(f):
def wraps(*args, **kw):
return SlicedIterator(f(*args, **kw))
return wraps
Testing:
#sliceable
def f():
print("compute 1")
yield 1
print("compute 2")
yield 2
print("compute 3")
yield 3
print("compute 4")
yield 4
print("== compute all four")
a, b, c, d = f()
print("== compute first two")
a, b = f()[:2]
print("== compute one only")
a = f()[0]
print("== all as a list")
a = f()[:]
gives:
== compute all four
compute 1
compute 2
compute 3
compute 4
== compute first two
compute 1
compute 2
== compute one only
compute 1
== all as a list
compute 1
compute 2
compute 3
compute 4
Related
The Question
Is there a straightforward algorithm for figuring out if a variable is "used" within a given scope?
In a Python AST, I want to remove all assignments to variables that are not otherwise used anywhere, within a given scope.
Details
Motivating example
In the following code, it is obvious to me (a human), that _hy_anon_var_1 is unused, and therefore the _hy_anon_var_1 = None statements can be removed without changing the result:
# Before
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
_hy_anon_var_1 = None
else:
n = 3 * n + 1
_hy_anon_var_1 = None
yield n
# After
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
else:
n = 3 * n + 1
yield n
Bonus version
Extend this to []-lookups with string literals as keys.
In this example, I would expect _hyx_letXUffffX25['x'] to be eliminated as unused, because _hyx_letXUffffX25 is local to h, so _hyx_letXUffffX25['x'] is essentially the same thing as a local variable. I would then expect _hyx_letXUffffX25 itself to be eliminated once there are no more references to it.
# Before
def h():
_hyx_letXUffffX25 = {}
_hyx_letXUffffX25['x'] = 5
return 3
# After
def h():
return 3
From what I can tell, this is somewhat of an edge case, and I think the basic algorithmic problem is the same.
Definition of "used"
Assume that no dynamic name lookups are used in the code.
A name is used if any of these are true in a given scope:
It is referenced anywhere in an expression. Examples include: an expression in a return statement, an expression on the right-hand side of an assignment statement, a default argument in a function definition, being referenced inside a local function definition, etc.
It is referenced on the left-hand side of an "augmented assignment" statement, i.e. it is an augtarget therein. This might represent "useless work" in a lot of programs, but for the purpose of this task that's OK and distinct from being an entirely unused name.
It is nonlocal or global. These might be useless nonlocals or globals, but because they reach beyond the given scope, it is OK for my purposes to assume that they are "used".
Please let me know in the comments if this seems incorrect, or if you think I am missing something.
Examples of "used" and "unused"
Example 1: unused
Variable i in f is unused:
def f():
i = 0
return 5
Example 2: unused
Variable x in f is unused:
def f():
def g(x):
return x/5
x = 10
return g(100)
The name x does appear in g, but the variable x in g is local to g. It shadows the variable x created in f, but the two x names are not the same variable.
Variation
If g has no parameter x, then x is in fact used:
def f():
x = 10
def g():
return x/5
return g(100)
Example 3: used
Variable i in f is used:
def f():
i = 0
return i
Example 4: used
Variable accum in silly_map and silly_sum is used in both examples:
def silly_map(func, data):
data = iter(data)
accum = []
def _impl():
try:
value = next(data)
except StopIteration:
return accum
else:
accum.append(value)
return _impl()
return _impl()
def silly_any(func, data):
data = iter(data)
accum = False
def _impl():
nonlocal accum, data
try:
value = next(data)
except StopIteration:
return accum
else:
if value:
data = []
accum = True
else:
return _impl()
return _impl()
The solution below works in two parts. First, the syntax tree of the source is traversed and all unused target assignment statements are discovered. Second, the tree is traversed again via a custom ast.NodeTransformer class, which removes these offending assignment statements. The process is repeated until all unused assignment statements are removed. Once this is finished, the final source is written out.
The ast traverser class:
import ast, itertools, collections as cl
class AssgnCheck:
def __init__(self, scopes = None):
self.scopes = scopes or cl.defaultdict(list)
#classmethod
def eq_ast(cls, a1, a2):
#check that two `ast`s are the same
if type(a1) != type(a2):
return False
if isinstance(a1, list):
return all(cls.eq_ast(*i) for i in itertools.zip_longest(a1, a2))
if not isinstance(a1, ast.AST):
return a1 == a2
return all(cls.eq_ast(getattr(a1, i, None), getattr(a2, i, None))
for i in set(a1._fields)|set(a2._fields) if i != 'ctx')
def check_exist(self, t_ast, s_path):
#traverse the scope stack and remove scope assignments that are discovered in the `ast`
s_scopes = []
for _ast in t_ast:
for sid in s_path[::-1]:
s_scopes.extend(found:=[b for _, b in self.scopes[sid] if AssgnCheck.eq_ast(_ast, b) and \
all(not AssgnCheck.eq_ast(j, b) for j in s_scopes)])
self.scopes[sid] = [(a, b) for a, b in self.scopes[sid] if b not in found]
def traverse(self, _ast, s_path = [1]):
#walk the ast object itself
_t_ast = None
if isinstance(_ast, ast.Assign): #if assignment statement, add ast object to current scope
self.traverse(_ast.targets[0], s_path)
self.scopes[s_path[-1]].append((True, _ast.targets[0]))
_ast = _ast.value
if isinstance(_ast, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)):
s_path = [*s_path, (nid:=(1 if not self.scopes else max(self.scopes)+1))]
if isinstance(_ast, (ast.FunctionDef, ast.AsyncFunctionDef)):
self.scopes[nid].extend([(False, ast.Name(i.arg)) for i in _ast.args.args])
_t_ast = [*_ast.args.defaults, *_ast.body]
self.check_exist(_t_ast if _t_ast is not None else [_ast], s_path) #determine if any assignment statement targets have previously defined names
if _t_ast is None:
for _b in _ast._fields:
if isinstance((b:=getattr(_ast, _b)), list):
for i in b:
self.traverse(i, s_path)
elif isinstance(b, ast.AST):
self.traverse(b, s_path)
else:
for _ast in _t_ast:
self.traverse(_ast, s_path)
Putting it all together:
class Visit(ast.NodeTransformer):
def __init__(self, asgn):
super().__init__()
self.asgn = asgn
def visit_Assign(self, node):
#remove assignment nodes marked as unused
if any(node.targets[0] == i for i in self.asgn):
return None
return node
def remove_assgn(f_name):
tree = ast.parse(open(f_name).read())
while True:
r = AssgnCheck()
r.traverse(tree)
if not (k:=[j for b in r.scopes.values() for k, j in b if k]):
break
v = Visit(k)
tree = v.visit(tree)
return ast.unparse(tree)
print(remove_assgn('test_name_assign.py'))
Output Samples
Contents of test_name_assign.py:
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
_hy_anon_var_1 = None
else:
n = 3 * n + 1
_hy_anon_var_1 = None
yield n
Output:
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
else:
n = 3 * n + 1
yield n
Contents of test_name_assign.py:
def h():
_hyx_letXUffffX25 = {}
_hyx_letXUffffX25['x'] = 5
return 3
Output:
def h():
return 3
Contents of test_name_assign.py:
def f():
i = 0
return 5
Output:
def f():
return 5
Contents of test_name_assign.py:
def f():
x = 10
def g():
return x/5
return g(100)
Ouptut:
def f():
x = 10
def g():
return x / 5
return g(100)
I have the following generator function which adds two numbers:
def add():
while True:
x = yield "x="
y = yield "y="
print (x+y)
And I can call it like this:
x=add()
next(x)
'x='
x.send(2)
'y='
x.send(3)
# 5
I thought it would be trivial to add in an init so that I don't have to do the next and I can just start sending it values, and so I did:
def init(_func):
def func(*args, **kwargs):
x=_func(*args, **kwargs)
next(x)
return x
return func
Or, simplifying it to receive no input variables like the function above:
def init(func):
x=func()
next(x)
return x
I thought that doing:
x=init(add) # doesn't print the "x=" line.
x.send(2)
'y='
x.send(3)
5
Would work, but it seems it acts just like as if the init is not there at all. Why does this occur, and how can I get rid of that behavior?
It seems to work for me. Tried
def add():
while True:
x = yield 'x='
y = yield 'y='
print (x+y)
def init(func):
x=func()
next(x)
return x
a = init(add)
a.send(5)
a.send(10)
For me this returns 15, as expected.
[update]
After your update, I think you might just want to print out the a.send():
def add():
while True:
x = yield 'x='
y = yield 'y='
print (x+y)
def init(func):
x=func()
print(next(x))
return x
a = init(add)
print(a.send(5))
a.send(10)
Your code works as-is, however, if you want to print the output that occurs before the field yield statement, then you can adapt the init method to do just that. For example:
def init(func):
x=func()
a=next(x)
if a: print (a) # this line will print the output, in your case 'x='
return x
And now you have:
>>> x=init(add)
x=
>>> x.send(2)
'y='
>>> x.send(3)
5
And finally, to keep your more generalized approach, you can do something like the following with a decorator:
def init_coroutine(_func):
def func(*args, **kwargs):
x=_func(*args, **kwargs)
_ = next(x)
if _: print (_)
return x
return func
#init_coroutine
def add():
while True:
x = yield "x="
y = yield "y="
print (x+y)
>>> x=add()
x=
>>> x.send(2)
'y='
>>> x.send(3)
5
I want to combine any two functions, funcA and funcB, so they create one function that take the input of funcB, send the result to funcA and return that result
funcC(args) = funcA(funcB(args))
For example:
def sub(a, b):
return a - b
def neg(a):
return -a
For this example, my question is how to create a negsub function such that
negsub(3, 4) = 1 # neg(sub(3, 4)) = 1
negsub(2, 0) = -2 # neg(sub(2, 0)) = -2
and so on.
This is probably the most straightforward way:
def negsub(a, b):
return neg(sub(a, b))
But if you want a function that builds functions out of other functions:
def combine(outer, inner):
return lambda *args: outer(inner(*args))
Now you can do:
negsub = combine(neg, sub)
negsub(3, 4)
def sub(a, b):
return a - b
def neg(a):
return -a
def negsub(a, b):
return neg(sub(a, b))
I wrote a generator function that I instantiate then call over and over again, each time this increments the number.
def pagecnt():
n = 1
while True:
yield n
n += 1
pg = pagecnt()
print(next(pg))
print(next(pg))
print(next(pg))
This prints 1, 2 and then 3. Is there some way to combine the generator and instantiation into a new function so that I can just call
print(newfunc())
print(newfunc())
print(newfunc())
to get 1, 2 and 3?
EDIT: I don't just want to call it only 3 times. It's used to generate page number. So I don't know in advance how many times I am going to call it. And between each call, there's lots of code to calculate stuff and generate graphs.
Just create a function which instantiates your generator and calls next n times
def combine_func(n):
pg = pagecnt()
for _ in range(n):
print(next(pg))
Or we can define a wrapper function which takes in the generator instance, and returns a function which we can call to get the next element
def combine_func(pg):
def next_elem():
return next(pg)
return next_elem
pg = pagecnt()
cf = combine_func(pg)
print(cf())
print(cf())
print(cf())
You could make a simple Counter class with __call__() defined:
class Counter():
def __init__(self):
self.count = 0
def __call__(self):
self.count += 1
return self.count
c = Counter()
print(c())
print(c())
print(c())
prints:
1
2
3
You could also capture the iterator in a closure and return a lambda function:
from itertools import count
def Counter():
it = count(1)
return lambda: next(it)
c = Counter()
print(c())
print(c())
print(c())
prints the same as above. In both cases it would be easy to pass in a start value if you wanted to begin somewhere other than 0.
Edit:
This is the same as using count but with your custom generator:
def Counter():
def pagecnt():
n = 1
while True:
yield n
n += 1
it = pagecnt()
return lambda: next(it)
c = Counter()
print(c())
print(c())
print(c())
I have the following example in which the next method of a class is supposed to return the values from two generators:
class Test():
def __next__(self):
g1, g2 = self._gen1(), self._gen2()
return next(g1), next(g2)
def _gen1(self):
i = 0
while True:
yield i
i += 2
def _gen2(self):
i = 1
while True:
yield i
i += 2
However, when I call next for this class, the values are not incremented.
>>> t = Test()
>>> next(t)
>>> (0, 1)
>>> next(t)
>>> (0, 1)
What is wrong? Is there a more eloquent way to write this class?
Although I have no idea what you are trying to accomplish, here is a cleaned up version which (I think) does what you want.
class Test():
def __init__(self):
self.g1 = self._gen2()
self.g2 = self._gen1()
def __next__(self):
return next(self.g1), next(self.g2)
def _gen1(self):
i = 0
while True:
yield i
i += 2
def _gen2(self):
i = 1
while True:
yield i
i += 2
t = Test()
print(next(t))
print(next(t))
print(next(t))
Your code doesn't work because it recreates the generator functions every time __next__() is called, which effectively resets them back to their initial state before their next next() values are returned:
def __next__(self):
g1, g2 = self._gen1(), self._gen2() # Don't do this here.
return next(g1), next(g2)
You can fix that by adding an __init__() method and initializing them in it:
class Test:
def __init__(self):
self.g1, self.g2 = self._gen1(), self._gen2() # Initialize here.
def __next__(self):
return next(self.g1), next(self.g2)
...
A more eloquent and slightly more concise way to do it which likewise will avoid the problem would be to use the builtin zip() function to create an "iterator of generators" that will return pairs of next values from each generator every time it's called. Another advantage is it's very easy to extend to handle even more generators simply just changing the __init__() method.
Here's what I mean:
class Test:
def __init__(self):
self.generators = zip(self._gen1(), self._gen2())
def __next__(self):
return next(self.generators)
def _gen1(self):
i = 0
while True:
yield i
i += 2
def _gen2(self):
i = 1
while True:
yield i
i += 2
t = Test()
for _ in range(3):
print(next(t))
Output:
(0, 1)
(2, 3)
(4, 5)