normal way:
for x in myList:
myFunc(x)
you must use a variable x
use
map(myFunc,myList)
and in fact you must use this to make above work
list(map(myFunc,myList))
that would build a list,i don't need to build a list
maybe some one would suggest me doing this
def func(l):
for x in l:
....
that is another topic
is there something like this?
every(func,myList)
The 'normal way' is definitely the best way, although itertools does offer the consume recipe for whatever reason you might need it:
import collections
from itertools import islice
def consume(iterator, n):
"Advance the iterator n-steps ahead. If n is none, consume entirely."
# Use functions that consume iterators at C speed.
if n is None:
# feed the entire iterator into a zero-length deque
collections.deque(iterator, maxlen=0)
else:
# advance to the empty slice starting at position n
next(islice(iterator, n, n), None)
This could be used like:
consume(imap(func, my_list), None) # On python 3 use map
This function performs the fastest as it avoids python for loop overhead by using functions which run on the C side.
AFAIK there is no 'foreach' shortcut in the standard library, but such a thing is very easy to implement:
def every(fun, iterable):
for i in iterable:
fun(i)
If you just want myList to be modified to contain myFunc(x) for all x in myList then you could try a list comprehension which also requires a variable, but doesn't let the variable leak out of the scope of the comprehension:
myList = [myFunc(x) for x in myList]
Hope that helps
Related
This code is part of a challenge that requires the code to give back permutations of a string with no duplicates. The code executes but under some of the challenges it doesn't pass because of the time gate and i dont know a way to make it execute faster.
from itertools import permutations as perm
def permutations(string):
permList = list(perm(string))
joinedList = [''.join(tups) for tups in permList]
ans = []
[ans.append(x) for x in joinedList if x not in ans]
return ans
Again code runs for certain examples but examples with large strings and alot of matches the code takes too long and fails the challenge.
If you want to prevent duplicates, use a set, not lists. Your code takes forever because you're constantly scanning the list while you are inserting new data. Instead you can do a constant time lookup / replacement
And you can save on storage costs by using generator comprehension rather than list comprehension
def permutations(string):
permList = (''.join(p) for p in perm(string)))
result = set()
for p in permList:
result.add(p)
return list(result)
Ideally, you should keep the output in a generator for as long as possible; evaluating it takes time and space.
Here we maintain a set of seen elements in order to avoid yielding them again thus keeping each unique.
import itertools
def unique_permutations(seq):
seen = set()
for p in itertools.permutations(seq):
if p not in seen:
seen.add(p)
yield(p)
for p in unique_permutations('aaab'):
print(p)
I have an iterator it that I'm assuming already sorted but I would like to raise an exception if it isn't.
Data from iterator is not in memory so I do not want to use sorted() builtin because AFAIK it puts the whole iterator in a list.
The solution I'm using now is to wrap the iterator in a generator function like this:
def checkSorted(it):
prev_v = it.next()
yield prev_v
for v in it:
if v >= prev_v:
yield v
prev_v = v
else:
raise ValueError("Iterator is not sorted")
So that I can use it like this:
myconsumer(checkSorted(it))
Does someone know if there are better solutions?
I know that my solution works but it seems quite strange (at least to me) writing a module on my own to accomplish such a trivial task. I'm looking for a simple one liner or builtin solution (If it exists)
Basically your solution is almost as elegant as it gets (you could of course put it in an utility module if you find it generally useful). You could if you wanted it use an infinity object to cut the code down a bit, but then you have to include a class definition as well which grows the code again (unless you inline the class definition):
def checkSorted(it):
prev = type("", (), {"__lt__": lambda a, b: False})()
for x in it:
if prev < x:
raise ValueError("Not sorted")
prev = x
yield x
The first line is using the type to first create a class and then instantiate it. Objects of this class compares less than to anything (infinity object).
The problem with doing a one-liner is that you have to deal with three constructs: you have to update state (assignment), throw an exception and doing a loop. You could easily perform these by using statements, but making them into a oneliner will mean that you will have to try to put the statements on the same line - which in turn will result in problem with the loop and if-constructs.
If you want to put the whole thing into an expression you will have to use dirty tricks to do these, the assignment and looping the iterutils can provide and the throwing can be done by using the throw method in a generator (which can be provided in an expression too):
imap( lambda i, j: (i >= j and j or (_ for _ in ()).throw(ValueError("Not sorted"))), *(lambda pre, it: (chain([type("", (), {"__ge__": lambda a, b: True})()], pre), it))(*tee(it)))
The last it is the iterator you want to check and the expression evaluates to a checked iterator. I agree it's not good looking and not obvious what it does, but you asked for it (and I don't think you wanted it).
As an alternative i suggest to use itertools.izip_longest (and zip_longest in python 3 )to create a generator contains consecutive pairs :
You can use tee to create 2 independent iterators from a first iterable.
from itertools import izip_longest,tee
def checkSorted(it):
pre,it=tee(it)
next(it)
for i,j in izip_longest(pre,it):
if j:
if i >= j:
yield i
else:
raise ValueError("Iterator is not sorted")
else :
yield i
Demo :
it=iter([5,4,3,2,1])
print list(checkSorted(it))
[5, 4, 3, 2, 1]
it=iter([5,4,3,2,3])
print list(checkSorted(it))
Traceback (most recent call last):
File "/home/bluebird/Desktop/ex2.py", line 19, in <module>
print list(checkSorted(it))
File "/home/bluebird/Desktop/ex2.py", line 10, in checkSorted
raise ValueError("Iterator is not sorted")
ValueError: Iterator is not sorted
Note : Actually I think there is no need to yield the values of your iterable wen you have them already.So as a more elegant way I suggest to use a generator expression within all function and return a bool value :
from itertools import izip,tee
def checkSorted(it):
pre,it=tee(it)
next(it)
return all(i>=j for i,j in izip(pre,it))
I am trying to make program that prints all the possible combinations for a to zzz. I tried to add a save state feature, and it works fine but there is this bug.
Let's say I interrupted the program when it printed something like e. When I execute the program again, it works fine until z but after z instead of printing aa it prints ba and continues from ba. This happens right after it prints zz too. it prints baa instead of aaa. How can I fix this?
Here is what I did so far:
import pickle,os,time
alphabet="abcdefghijklmnopqrstuvwxyz"
try:
if os.path.isfile("save.pickle")==True:
with open("save.pickle","rb") as f:
tryn=pickle.load(f)
for i in range(3):
a=[x for x in alphabet]
for j in range(i):
a=[x+i for x in alphabet for i in a]
b=a[tryn:]
for k in b:
print(k)
time.sleep(0.01)
tryn+=1
else:
tryn=0
for i in range(3):
a=[x for x in alphabet]
for j in range(i):
a=[x+i for x in alphabet for i in a]
for k in a:
print(k)
tryn+=1
time.sleep(0.01)
except KeyboardInterrupt:
with open("save.pickle","wb") as f:
pickle.dump(tryn,f)
If you're using python2, or python3 as the tag suggests, this exists in the standard library already. See itertools, product py2, and product py3, for a simple way to solve this problem.
for i in range(3):
a=[x for x in alphabet]
for j in range(i):
a=[x+i for x in alphabet for i in a]
b=a[tryn:]
Here's your bug. You skip the first tryn strings of every length, rather than just the first tryn strings. This would be easier to recognize in the output if it weren't for the following:
for k in b:
print(k)
time.sleep(0.01)
tryn+=1
You modify tryn, the number of things you're skipping. When you print out length-2 strings, you skip a number of them equal to the number of length-1 strings. When you print out length-3 strings, you skip a number of them equal to the number of length-2 strings. If tryn were bigger than the number of length-1 strings, you would skip even more.
your problem is almost certainly here:
a=[x for x in alphabet]
for j in range(i):
a=[x+i for x in alphabet for i in a]
Perhaps you shouldn't assign the in-loop value to a, but instead use a different name? Otherwise, you are changing what you use every time through the loop....
Edit: More detail. So, technically user2357112's answer is more correct, but I'm amending mine. The initial answer was just from a quick reading, so the other answer is close to the original intent. But, the original version is inefficient (for more reasons than not using product :), since you are generating the inner loops more than once. So let's walk through why this is a bad idea, as an educational exercise:
Initial algorithm:
for i in range(n):
assign a to alphabet
for j in range(i):
i times, we rewrite a to be all combinations of the current set against the alphabet.
Note that for this algorithm, to generate the length(n) product, we have to generate all previous products length(n-1), length(n-2), ..., length(1). But you aren't saving those.
You'd be better off doing something like this:
sum_list = alphabet[:]
#get a copy
product_list = alphabet[:]
#Are we starting at 0, or 1? In any case, skip the first, since we preloaded it
for i in range(1, n):
# Your existing list comprehension was equivalent here, and could still be used
# it MIGHT be faster to do '%s%s'%(x,y) instead of x+y... but maybe not
# with these short strings
# This comprehension takes the result of the last iteration, and makes the next iteration
product_list = [x+y for x,y in product(product_list, alphabet)]
# So product list is JUST the list for range (n) - i.e. if we are on loop 2, this
# is aaa...zzz. But you want all lengths together. So, as you go, add these
# sublists to a main list.
sum_list.extend(product_list)
Overall, you are doing a lot less work.
Couple other things:
You're using i as a loop variable, then re-using it in the loop comprehension. This is conflicting, and probably not working the way you'd expect.
If this is to learn how to write save/restore type apps... it's not a good one. Note that the restore function is re-calculating every value to be able to get back where it left off - if you could rewrite this algorithm to write more information out to the file (such as the current value of product_list) and make it more generator-like, then it will actually work more like a real-world example.
Here is how I would suggest solving this problem in Python. I didn't implement the save state feature; this sequence is not a really long one and your computer should be able to produce this sequence pretty fast, so I don't think it is worth the effort to try to make it cleanly interruptable.
import itertools as it
def seq(alphabet, length):
for c in range(1, length+1):
for p in it.product(alphabet, repeat=c):
yield ''.join(p)
alphabet="abcdefghijklmnopqrstuvwxyz"
for x in seq(alphabet, 3):
print(x)
If you really wanted to, you could make a one-liner using itertools. I think this is too hard to read and understand; I prefer the above version. But this does work and will be somewhat faster, due to the use of itertools.chain and itertools.imap() rather than a Python for loops.
import itertools as it
def seq(alphabet, length):
return it.imap(''.join, it.chain.from_iterable(it.product(alphabet, repeat=c) for c in range(1, length+1)))
alphabet="abcdefghijklmnopqrstuvwxyz"
for x in seq(alphabet, 3):
print(x)
In Python 3.x you could just use map() rather than itertools.imap().
I have a structure with an x amount of lists in lists, and each list an x amount of tuples. I don't know beforehand how many nested lists there are, or how many tuples in each list.
I want to make dictionaries out of all the tuples and because I don't know the depth of the lists I want to use recursion. What I did was
def tupleToDict(listOfList, dictList):
itemDict = getItems(list) # a function that makes a dictionary out of all the tuples in list
dictList.append(itemDict)
for nestedList in listOfList:
getAllNestedItems(nestedList, dictList)
return dictList
this works, but I end up with a huge list at the end. I would rather return the itemDict at every round of recursion. However, I don't know how to (if it is possible) return a value without stopping the recursion.
You're looking for yield:
def tupleToDict(listOfList):
yield getItems(listofList)
for nestedList in listOfList:
for el in getAllNestedItems(nestedList):
yield el
In Python 3.3+, you can replace the last two lines with a yield from.
You may want to rewrite your function to be iterative:
def tupleToDict(listOfList):
q = [listOfList]
while q:
l = q.pop()
yield getItems(l)
for nestedList in listOfList:
q += getAllNestedItems(nestedList)
Who are you going to return it to? I mean if your thread is busy running the recursive algorithm, who gets the "interim results" to process?
Best bet is to tweak your algorithm to include some processing before it recurses again.
I'm not sure what you're trying to do, but you could try to make a recursive generator by using the yield statement to return the dict at the desired intervals. Either that or shove copies of it into a global list?
You got two possible solutions:
The generator approach: a function with a yield statement, which may be a hassle to implement in a recursive function. (Look at phihags proposal for an example)
The callback approach: You call a helper-function/method from inside the recursion and can monitor the progress through a second outer function.
Here a non-recursive recursion example: ;-)
def callback(data):
print "from the depths of recursion: {0}".format(data)
def recursion(arg, callbackfunc):
arg += 1
callbackfunc(arg)
if arg <10:
recursion(arg, callbackfunc)
return arg
print recursion(1, callback)
I have a loop of the following type:
a = range(10)
b = [something]
for i in range(len(a)-1):
b.append(someFunction(b[-1], a[i], a[i+1]))
However the for-loop is killing a lot of performance. I have try to write a windows generator to give me 2 elements everything time but it still require explicit for-loop in the end. Is there a way to make this shorter and more efficient in a pythonic way?
Thanks
edit: I forgot the element in b.. sorry guys. However the solution to my previous problem is very helpful in other problem I have too. Thanks.
Consider this
def make_b( a, seed ):
yield seed
for a,b in zip( a[:-1], a[1:] ):
seed= someFunction( seed, a, b )
yield seed
Which lets you do this
a = xrange(10)
b= list(make_b(a,something))
Note that you can often use this:
b = make_b(a)
Instead of actually creating b as a list. b as a generator function saves you considerable storage (and some time) because you may not really need a list object in the first place. Often, you only need something iterable.
Similarly for a. It does not have to be a list, merely something iterable -- like a generator function with a yield statement.
For your initially stated problem of mapping a function over pairs of an input sequence the following will work, and is about as efficient as it gets while staying in Python land.
from itertools import tee
a = range(10)
a1, a2 = tee(a)
a2.next()
b = map(someFunction, a1, a2)
As for the expanded problem where you need to access the result of the previous iteration - this kind of inner state is present in the functional concept unfold. But Python doesn't include an unfold construct, and for a good reason for loops are more readable in this case and most likely faster too. As for making it more Pythonic, I suggest lifting the pairwise iteration out to a function and create an explicit loop variable.
def pairwise(seq):
a, b = tee(seq)
b.next()
return izip(a, b)
def unfold_over_pairwise(unfolder, seq, initial):
state = initial
for cur_item, next_item in pairwise(seq):
state = unfolder(state, cur_item, next_item)
yield state
b = [something]
b.extend(unfold_over_pairwise(someFunction, a, initial=b[-1]))
If the looping overhead really is a problem, then someFunction must be something really simple. In that case it probably is best to write the whole loop in a faster language, such as C.
Some loop or other will always be around, but one possibility that might reduce overhead is:
import itertools
def generate(a, item):
a1, a2 = itertools.tee(a)
next(a2)
for x1, x2 in itertools.izip(a1, a2):
item = someFunction(item, x1, x2)
yield item
to be used as:
b.extend(generate(a, b[-1]))
Try something like this:
a = range(10)
b = [something]
s = len(b)
b+= [0] * (len(a) - 1)
[ b.__setitem__(i, someFunction(b[i-1], a[i-s], a[i-s+1])) for i in range(s, len(b))]
Also:
using functions from itertools should
be useful also (earlier posts)
maybe you can rewrite someFunction and use map instead of list
comprehension