Efficient way to reverse a large iterator - python

This may sound a bit insane but I have an iterator with N = 10**409 elements. Is there a way to get items from the end of this "list"? I.e. when I call next(iterator) it gives me what I want to be the last thing, but to get to what I want to be first thing I would need to call next(iterator) N times.
If I do something like list(iterator).reverse() it will of course crash due to lack of memory.
Edit: how the iterator is being used with a simplified example:
# prints all possible alphabetical character combinations that can fit in a tweet
chars = "abcdefghijklmnopqrstuvwxyz "
cproduct = itertools.product(chars,repeat=250)
for subset in cproduct:
print(''.join(subset))
# will start with `aaaaaaaa...aaa`
# but I want it to start with `zzz...zzz`

For some problems, you can compute the elements in reverse. For the example you provide, one can simply reverse the items you are taking the product of.
In this example, we reverse the symbols before taking the product to get the "reverse iterator":
>>> symbols = "abc"
>>> perms = itertools.product(symbols, repeat=5)
>>> perms = ["".join(x) for x in perms]
>>> perms
['aaaaa', 'aaaab', 'aaaac', 'aaaba', 'aaabb',
...,
'cccbb', 'cccbc', 'cccca', 'ccccb', 'ccccc']
>>> perms_rev = itertools.product(symbols[::-1], repeat=5)
>>> perms_rev = ["".join(x) for x in perms_rev]
>>> perms_rev
['ccccc', 'ccccb', 'cccca', 'cccbc', 'cccbb',
...,
'aaabb', 'aaaba', 'aaaac', 'aaaab', 'aaaaa']
>>> perms_rev == perms[::-1]
True

Related

most pythonic way to compare substrings l in list L to string S & edit S according to l in L?

The list ['a','a #2','a(Old)'] should become {'a'} because '#' and '(Old)' are to be excised and a list of duplicates isn't needed. I struggled to develop a list comprehension with a generator and settled on this since I knew it'd work and valued time more than looking good:
l = []
groups = ['a','a #2','a(Old)']
for i in groups:
if ('#') in i: l.append(i[:i.index('#')].strip())
elif ('(Old)') in i: l.append(i[:i.index('(Old)')].strip())
else: l.append(i)
groups = set(l)
What's the slick way to get this result?
Here is general solution, if you want to clean elements of list lst from parts in wastes:
lst = ['a','a #2','a(Old)']
wastes = ['#', '(Old)']
cleaned_set = {
min([element.split(waste)[0].strip() for waste in wastes])
for element in arr
}
You could write this whole expression in a single set comprehension
>>> groups = ['a','a #2','a(Old)']
>>> {i.split('#')[0].split('(Old)')[0].strip() for i in groups}
{'a'}
This will get everything preceding a # and everything preceding '(Old)', then trim off whitespace. The remainder is placed into a set, which only keeps unique values.
You could define a helper function to apply all of the splits and then use a set comprehension.
For example:
lst = ['a','a #2','a(Old)', 'b', 'b #', 'b(New)']
splits = {'#', '(Old)', '(New)'}
def split_all(a):
for s in splits:
a = a.split(s)[0]
return a.strip()
groups = {split_all(a) for a in lst}
#{'a', 'b'}

Cut character string every two commas

I would like to separate my string every both commas but I can not, can you help me.
This is what I want: ['nb1,nb2','nb3,nb4','nb5,nb6']
Here is what I did :
a= 'nb1,nb2,nb3,nb4,nb5,nb6'
compteur=0
for i in a:
if i==',' :
compteur+=1
if compteur%2==0:
print compteur
test = a.split(',', compteur%2==0 )
print a
print test
The result:
2
4
nb1,nb2,nb3,nb4,nb5,nb6
['nb1', 'nb2,nb3,nb4,nb5,nb6']
Thanks you by advances for you answers
You can use regex
In [12]: re.findall(r'([\w]+,[\w]+)', 'nb1,nb2,nb3,nb4,nb5,nb6')
Out[12]: ['nb1,nb2', 'nb3,nb4', 'nb5,nb6']
A quick fix could be to simply first separate the elements by commas and then join the elements by two together again. Like:
sub_result = a.split(',')
result = [','.join(sub_result[i:i+2]) for i in range(0,len(sub_result),2)]
This gives:
>>> result
['nb1,nb2', 'nb3,nb4', 'nb5,nb6']
This will also work if the number of elements is odd. For example:
>>> a = 'nb1,nb2,nb3,nb4,nb5,nb6,nb7'
>>> sub_result = a.split(',')
>>> result = [','.join(sub_result[i:i+2]) for i in range(0,len(sub_result),2)]
>>> result
['nb1,nb2', 'nb3,nb4', 'nb5,nb6', 'nb7']
You use a zip operation of the list with itself to create pairs:
a = 'nb1,nb2,nb3,nb4,nb5,nb6'
parts = a.split(',')
# parts = ['nb1', 'nb2', 'nb3', 'nb4', 'nb5', 'nb6']
pairs = list(zip(parts, parts[1:]))
# pairs = [('nb1', 'nb2'), ('nb2', 'nb3'), ('nb3', 'nb4'), ('nb4', 'nb5'), ('nb5', 'nb6')]
Now you can simply join every other pair again for your output:
list(map(','.join, pairs[::2]))
# ['nb1,nb2', 'nb3,nb4', 'nb5,nb6']
Split the string by comma first, then apply the common idiom to partition an interable into sub-sequences of length n (where n is 2 in your case) with zip.
>>> s = 'nb1,nb2,nb3,nb4,nb5,nb6'
>>> [','.join(x) for x in zip(*[iter(s.split(','))]*2)]
['nb1,nb2', 'nb3,nb4', 'nb5,nb6']

Given a list of string, determine if one string is a prefix of another string

I want to write a Python function which checks if one string is a prefix string of another; not an arbitrary sub string of another; must be prefix. If it is, return True. For instance,
list = ['abc', 'abcd', 'xyx', 'mno']
Return True because 'abc' is a prefix of 'abcd'.
list = ['abc', 'xyzabc', 'mno']
Return False
I tried the startwith() and list comprehension, but it didn't quite work.
Appreciate for any help or pointers.
Let us first sort the given lst w.r.t length of the string, due to the known fact that sub strings always have length less than or equal to the original string, so after sorting we have strings with smaller length at the start of the list, and then we iterate over the sorted list comparing the current element with all the elements next to it, This small optimization would reduce the complexity of the problem as now we don't have to comapre each element with every other element.
lst1 = ['abc', 'abcd', 'xyx', 'mno']
lst2 = ['abc', 'xyzabc', 'mno']
lst3 = ["abc", "abc"]
def check_list(lst):
lst = list(set(lst)) #if you want to avoid redundant strings.
lst.sort(key = lambda x:len(x))
n = len(lst)
for i in xrange(n):
for j in xrange(i+1, n):
if lst[j].startswith(lst[i]):
return True
return False
print check_list(lst1)
print check_list(lst2)
print check_list(lst3)
>>> True
>>> False
>>> False #incase you use lst = list(set(lst))
Using itertools
import itertools
list1 = ["abc", "xyz", "abc123"]
products = itertools.product(list1, list1)
is_substringy = any(x.startswith(y) for x, y in products if x != y)
This isn't very optimised, but depending on the amount of data you've got to deal with, the code is fairly elegant (and short); that might trump speed in your use case.
This assumes that you don't have pure repeats in the list however (but you don't have that in your example).
import itertools
mlist = ['abc', 'abcd', 'xyx', 'mno']
#combination of list elements, 2-by-2. without repetition
In [638]: for i,j in itertools.combinations(mlist,2):
print (i,j)
.....:
('abc', 'abcd')
('abc', 'xyx')
('abc', 'mno')
('abcd', 'xyx')
('abcd', 'mno')
('xyx', 'mno')
#r holds the final result. if there is any pair where one is a prefixed of another
r=False
In [639]: for i,j in itertools.combinations(mlist,2):
r = r or i.startswith(j) # if i is the prefix of j. logical or
r = r or j.startswith(i) # if j is the prefix of i
.....:
In [640]: r
Out[640]: True

Use of slice command with split command?

fh=open('asd.txt')
data=fh.read()
fh.close()
name=data.split('\n')[0][1:]
seq=''.join(data.split('\n')[1:])
print name
print seq
In this code, the 3rd line means "take only first line with first character removed" while the 4th line means "leave the first line and join the next remaining lines".
I cannot get the logic of these two lines.
Can anyone explain me how these two slice operators ([0][1:]) are used together?
Thanx
Edited: renamed file variable (which is a keyword, too) to data.
Think of it like this: file.split('\n') gives you a list of strings. So the first indexing operation, [0], gives you the first string in the list. Now, that string itself is a "list" of characters, so you can then do [1:] to get every character after the first. It's just like starting with a two-dimensional list (a list of lists) and indexing it twice.
When confused by a complex expression, do it it steps.
>>> data.split('\n')[0][1:]
>>> data
>>> data.split('\n')
>>> data.split('\n')[0]
>>> data.split('\n')[0][1:]
That should help.
lets do it by steps, (I think I know what name and seq is):
>>> file = ">Protein kinase\nADVTDADTSCVIN\nASHRGDTYERPLK" <- that's what you get reading your (fasta) file
>>> lines = file.split('\n') <- make a list of lines
>>> line_0 = lines[0] <- take first line (line numbers start at 0)
>>> name = line_0[1:] <- give me line items [x:y] (from x to y)
>>> name
'Protein kinase'
>>>
>>> file = ">Protein kinase\nADVTDADTSCVIN\nASHRGDTYERPLK"
>>> lines = file.split('\n')
>>> seqs = lines[1:] <- gime lines [x:y] (from x to y)
>>> seq = ''.join(seqs)
>>> seq
'ADVTDADTSCVINASHRGDTYERPLK'
>>>
in slice [x:y], x is included, y is not included. When you want to arrive to the end of the list just do not indicate y -> [x:] (from item of index x to the end)
Each set of [] just operates on the list that split returns, and the resulting
list or string then used without assigning it to another variable first.
Break down the third line like this:
lines = file.split('\n')
first_line = lines[0]
name = first_line[1:]
Break down the fourth line like this:
lines = file.split('\n')
all_but_first_line = lines[1:]
seq = ''.join(all_but_first_line)
take this as an example
myl = [["hello","world","of","python"],["python","is","good"]]
so here myl is a list of list. So, myl[0] means first element of list which is equal to ['hello', 'world', 'of', 'python'] but when you use myl[0][1:] it means selecting first element from list which is represented by myl[0] and than from the resulting list(myl[0]) select every element except first one(myl[0][1:]). So output = ['world', 'of', 'python']

extracting data from matchobjects

I have a long sequence with multiple repeats of a specific string( 'say GAATTC') randomly throughout the sequence string. I'm currently using the regular expression .span() to provide with me with the indices of where the pattern 'GAATTC' is found. Now I want to use those indices to slice the pattern between the G and A (i.e. 'G|AATTC').
How do I use the data from the match object to slice those out?
If I understand you correctly, you have the string and an index where the sequence GAATTC starts, so do you need this (i here is the m.start for the group)?
>>> seq = "GAATTC"
>>> s = "AATCCTGAGAATTCAAC"
>>> i = 8 # the index where seq starts in s
>>> s[i:]
'GAATTCAAC'
>>> s[i:i+len(seq)]
'GAATTC'
That extracts it. You can also slice the original sequence at the G like this:
>>> s[:i+1]
'AATCCTGAG'
>>> s[i+1:]
'AATTCAAC'
>>>
If what you want to do is replace the 'GAATTC' by the 'G|AATTC' one (not sure of what you want to do in the end), I think that you can manage this without regex:
>>> string = 'GAATTCAAGAATTCTTGAATTCGAATTCAATATATA'
>>> string.replace('GAATTC', 'G|AATTC')
'G|AATTCAAG|AATTCTTG|AATTCG|AATTCAATATATA'
EDIT: ok, this way can be adapted to suit what you want to do:
>>> groups = string.replace('GAATTC', 'G|AATTC').split('|')
>>> groups
['G', 'AATTCAAG', 'AATTCTTG', 'AATTCG', 'AATTCAATATATA']
>>> map(len, groups)
[1, 8, 8, 6, 13]

Categories

Resources