Develop Reference

separate the abnormal reads of DNA (A,T,C,G) templates

I have millions of DNA clone reads and few of them are misreads or error. I want to separate the clean reads only.
For non biological background:
DNA clone consist of only four characters (A,T,C,G) in various permutation/combination. Any character, symbol or sign other that "A","T","C", and "G" in DNA is an error.
Is there any way (fast/high throughput) in python to separate the clean reads only.
Basically I want to find a way through which I can separate a string which has nothing but "A","T","C","G" alphabet characters.
Edit
correct_read_clone: "ATCGGTTCATCGAATCCGGGACTACGTAGCA"
misread_clone: "ATCGGNATCGACGTACGTACGTTTAAAGCAGG" or "ATCGGTT#CATCGAATCCGGGACTACGTAGCA" or "ATCGGTTCATCGAA*TCCGGGACTACGTAGCA" or "AT?CGGTTCATCGAATCCGGGACTACGTAGCA" etc
I have tried the below for loop
check_list=['A','T','C','G']
for i in clone:
if i not in check_list:
continue
but the problem with this for loop is, it iterates over the string and match one by one which makes this process slow. To clean millions of clone this delay is very significant.

If these are the nucleotide sequences with an error in 2 of them,
a = 'ATACTGAGTCAGTACGTACTGAGTCAGTACGT'
b = 'AACTGAGTCAGTACGTACTGAGTCAAGTCAGTACGTSACTGAGTCAGTACGT'
c = 'ATUACTGAGTCAGTACGT'
d = 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
e = 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
f = 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT'
g = 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
test = a, b, c, d, e, f, g
try:
misread_counter = 0
correct_read_clone = []
for clone in test:
if len(set(list(clone))) <= 4:
correct_read_clone.append(clone)
else:
misread_counter +=1
print(f'Unclean sequences: {misread_counter}')
print(correct_read_clone)
Output:
Unclean sequences: 2
['ATACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT', 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT']
This way the for loop only has to attend each full sequence in a list of clones, rather than looping over each character of every sequence.
or if you want to know which ones have the errors you can make two lists:
misread_clone = []
correct_read_clone = []
for clone in test:
bases = len(set(list(clone)))
misread_clone.append(clone) if bases > 4 else correct_read_clone.append(clone)
print(f'misread sequences count: {len(misread_clone)}')
print(correct_read_clone)
Output:
misread sequences count: 2
['ATACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT', 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT', 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT']

I don't think you're going to get too many significant improvements for this. Most operations on a string are going to be O(N), and there isn't much you can do to get it to O(log(N)) or O(1). The checking for the values in ACTG is also O(N), leading to a worse case of O(n*m), where n and m are the lengths of the string and ACTG.
One thing you could do is cast the string into a set, which would be O(N), check if the length of set is more than 4 (which should be impossible if the only characters are ACTG) and if not, loop through the set and do the check against ACTG. I am assuming that it is possible that a clone could possibly be a string such as "AACCAA!!" which results in a set of ['A', 'C', '!'] in which case the length would be less than or equal to 4, but still be unclean/incorrect.
clones = [ "ACGTATCG", "AGCTGACGAT", "AGTACGATCAGT", "ACTGAGTCAGTACGT", "AGTACGTACGATCAGTACGT", "AAACCS", "AAACCCCCGGGGTTTS"]
for clone in clones:
if len(set(clone)) > 4:
print(f"unclean: {clone}")
else:
for char in clone:
if char not in "ACTG":
print(f"unclean: {clone}")
break
else:
print(f"clean: {clone}")
Since len(set) is O(1), that could potentially skip the need to check against ACTG. If it is less than or equal to 4, then the check would be O(n*m) again, but in this case the n is guaranteed to be less than 4 while your m stays the same at 4. The final process becomes O(n) rather than O(n*m), where n and m are the lengths of the set and ACTG. Since you are now checking against a set and anything other than ACTG will be unclean, n has a cap of 5. This means that no matter how large the original string is, doing the ACTG check on the set will be worst case O(5*4) and is thus essentially O(1) (Big O notation is about scale rather than exact values).
However, whether or not this is actually faster would depend on the length of the original string. It may end up taking more time if the original string is short. This would be unlikely, since the string would have to be very short, but can be the case.
You may get more time saved by tackling the amount of entries which you have noted is very large, if possible you may want to consider if you can split this into smaller groups to run them asynchronously. However, at the end of the day none of these are going to actively scale down your time. They would reduce the time taken since you'd be cutting out a constant scale from the time complexity or running a few at the same time, but at the end of the day it's still an O(N*M), with N and M being the number and length of strings, and there isn't anything that can really change that.

try this:
def is_clean_read(read):
for char in read:
if char not in ['A', 'T', 'C', 'G']:
return False
return True
reads = [ "ACGTATCG", "AGCTGACGAT", "AGTACGATCAGT", "ACTGAGTCAGTACGT", "AGTACGTACGATCAGTACGT"]
clean_reads = [read for read in reads if is_clean_read(read)]
print(clean_reads)

ok stealing from answer https://stackoverflow.com/a/75393987/9877065 by Shorn, tried to add multiprocessing, you can play with the lenght of my orfs list in the first part of the code and then try to change the number_of_processes = XXX to different values from 1 to your system max : multiprocessing.cpu_count(), code :
import time
from multiprocessing import Pool
from datetime import datetime
a = 'ATACTGAGTCAGTACGTACTGAGTCAGTACGT'
b = 'AACTGAGTCAGTACGTACTGAGTCAAGTCAGTACGTSACTGAGTCAGTACGT'
c = 'ATUACTGAGTCAGTACGT'
d = 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
e = 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
f = 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT'
g = 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
aa = 'ATACTGAGTCAGTACGTACTGAGTCAGTACGT'
bb = 'AACTGAGTCAGTACGTACTGAGTCAAGTCAGTACGTSACTGAGTCAGTACGT'
cc = 'ATUACTGAGTCAGTACGT'
dd = 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
ee = 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
ff = 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT'
gg = 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
aaa = 'ATACTGAGTCAGTACGTACTGAGTCAGTACGT'
bbb = 'AACTGAGTCAGTACGTACTGAGTCAAGTCAGTACGTSACTGAGTCAGTACGT'
ccc = 'ATUACTGAGTCAGTACGT'
ddd = 'AAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
eee = 'AACTGAGTCAGTAAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
fff = 'AAGTACGTACTGAGTCAGTACGTACTCAGTACGT'
ggg = 'ATCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGTACTGAGTCAGTACGT'
kkk = 'AAAAAAAAAAAAAAAAAAAAAAkkkkkkkkkkkkk'
clones = [a, b, c, d, e, f, g, aa, bb, cc, dd, ee, ff,gg, aaa, bbb, ccc, ddd, eee, fff, ggg, kkk]
clones_2 = clones
clones_2.extend(clones)
clones_2.extend(clones)
clones_2.extend(clones)
clones_2.extend(clones)
clones_2.extend(clones)
clones_2.extend(clones)
# clones_2.extend(clones)
# clones_2.extend(clones)
#print(clones_2, len(clones_2))
def check(clone):
# ATTENZIONE ALLUNGA TEMPO CPU vs I/O ##############################################################################################################
# time.sleep(1) ####################################################### !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
if len(set(clone)) > 4:
print(f"unclean: {clone}")
else:
for char in clone:
if char not in "ACTG":
print(f"unclean: {clone}")
break
else:
print(f"clean: {clone}")
begin = datetime.now()
number_of_processes = 4
p = Pool(number_of_processes)
list_a = []
cnt_atgc = 0
while True:
for i in clones_2 :
try:
list_a.append(i)
cnt_atgc += 1
if cnt_atgc == number_of_processes:
result = p.map(check, list_a)
p.close()
p.join()
p = Pool(number_of_processes)
cnt_atgc = 0
list_a = []
else:
continue
except:
print('SKIPPED !!!')
if len(list_a) > 0:
p = Pool(number_of_processes)
result = p.map(check, list_a)
p.close()
p.join()
break
else:
print('FINITO !!!!!!!!!!')
break
print('done')
print(datetime.now() - begin)
I have to pre load a list containing the orfs to be multiprocessed at each iteration, despite that can at least cut the execuition time by half on my machine, not sure how stdout influence the speed of the multiprocessing (and how to cope with result order see python multiprocess.Pool show results in order in stdout).

Optimize python pattern matching in nucleotide sequences

I'm currently working on a bioinformatic and modelling project where I need to do some pattern matching. Let's say I have a DNA fragment as follow 'atggcgtatagagc' and I split that fragment in micro-sequences of 8 nucleotides so that I have :
'atggcgta' 'tggcgtat' 'ggcgtata' 'gcgtatag' 'cgtataga' 'gtatagag' 'tatagagc'
And for each of these fragment I want to search in a whole genome and per chromosome the number of time they appear and the positions (starting positions) of the matches.
Here is how my code looks like :
you can download the genome fasta file here :
drive to the fasta file
import re
from Bio.SeqIO.FastaIO import FastaIterator
from Bio.Seq import Seq
def reverse_complement(sequence: str) -> str:
my_sequence = Seq(sequence)
return str(my_sequence.reverse_complement())
# you will need to unzip the file ant change the path below according to your working directory
path = '../data/Genome_S288c.fa'
genome = open(path, "r")
chr_sequences = {}
for record in FastaIterator(genome):
chr_id = record.id
seq = str(record.seq).lower()
rc_seq = reverse_complement(seq)
chr_sequences[chr_id] = {'5to3': seq, '3to5': rc_seq}
genome.close()
sequences = 'ATGACTAACGAAAAGGTCTGGATAGAGAAGTTGGATAATCCAACTCTTTCAGTGTTACCACATGACTTTTTACGCCCACAATCTTTAT'.lower()
micro_size = 8
micro_sequences = []
start = micro_size - 1
for i in range(start, len(sequences), 1):
current_micro_seq = sequences[i - start:i + 1]
micro_sequences.append(current_micro_seq)
genome_count = 0
chr_count = {}
chr_locations = {}
micro_fragment_stats = {}
for ii_micro, micro_seq in enumerate(micro_sequences):
for chr_idx in list(chr_sequences.keys()):
chr_counter = 0
seq = chr_sequences[chr_idx]['5to3']
pos = [m.start() for m in re.finditer(pattern=r'(?=(' + micro_seq + '))', string=seq)]
rc_seq = chr_sequences[chr_idx]['3to5']
rc_pos = [m.start() for m in re.finditer(pattern=r'(?=(' + micro_seq + '))', string=rc_seq)]
chr_locations[chr] = {'5to3': pos, '3to5': rc_pos}
chr_counter += len(pos) + len(rc_pos)
chr_count[chr_idx] = chr_counter
genome_count += chr_counter
micro_fragment_stats[ii_micro] = {'occurrences genome': genome_count,
'occurrences chromosomes': chr_count,
'locations chromosomes': chr_locations}
Actually my fragment is something like 2000bp long, so I took about 1 hour to compute all the micro-sequences. \
By the way, I use the r'(?=('+self.sequence+'))' to avoid the case of pattern that overlaps itself in the sequence, for instance :
pattern = 'aaggaaaaa'
string = 'aaggaaaaaggaaaaa'
expected output : (0, 7)
I am looking for a more efficient regex method that I can use for my case (in python if possible).
Thanks in advance

I would not recommend using regex for repetitive simple pattern matching. Outright comparison is expected to perform better. I did some basic testing and came up with the demo below.
import time
import re
import random
def compare(r1, r2, microseq_len, test_condition=1):
# condition 1: make microseqs/indexes from longer sequence and search against shorter
# condition 2: use regex to find position of microseq in reference sequence
# condition 3: use regex to find position of microseq in reference sequence after verifying if microseq in reference strain
start_time = time.time()
if test_condition == 1:
r1, r2 = r2, r1
# assemble dictionary containing microsequences and index positions
microseq_di = {}
for i in range(len(r1)-microseq_len):
microseq = r1[i:i+microseq_len]
if microseq not in microseq_di:
microseq_di[microseq] = []
microseq_di[microseq].append([i, i+microseq_len])
# mark for deletion
for microseq in microseq_di:
# condition 2
if test_condition == 2:
microseq_di[microseq] = [m.start() for m in re.finditer(pattern=r'(?=('+microseq+'))', string=r2)]
elif microseq not in r2:
microseq_di[microseq] = []
# condition 3
elif test_condition == 3:
microseq_di[microseq] = [m.start() for m in re.finditer(pattern=r'(?=('+microseq+'))', string=r2)]
print(time.time() - start_time) # run time
# delete and return
return({x:y for x, y in microseq_di.items() if y != []})
Input and Output:
r_short = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(500)])
r_long = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(100000)])
len(compare(r_short, r_long, 8, test_condition=1).keys())
0.19868111610412598
Out[1]: 400
len(compare(r_short, r_long, 8, test_condition=2).keys())
0.8831210136413574
Out[2]: 399
len(compare(r_short, r_long, 8, test_condition=3).keys())
0.7925639152526855
Out[3]: 399
Test condition 1 (microseqs from longer sequence) performed a lot better than the other two conditions using regex. Relative performance should improve with longer strings.
r_short = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(2000)])
r_long = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(1000000)])
len(compare3(r_short, r_long, 8, test_condition=1).keys())
2.2517480850219727
Out[4]: 1970
len(compare3(r_short, r_long, 8, test_condition=2).keys())
35.65084385871887
Out[5]: 1969
len(compare3(r_short, r_long, 8, test_condition=3).keys())
34.994577169418335
Out[6]: 1969
Note that condition 1 is not fully accommodating to your use-case since it doesn't exclude overlapping microseqs.

I've been playing with this question for a while and I end up with some ideas.
The algorithm is mainly divided in two parts: k-mer generation and k-mer searching in the reference.
For the k-mer generation part, I can see that your algorithm is quick, but it generates duplicates (that you have to filter afterwards when generating the dictionary). My approach has been to generate a deduplicated list directly. In my sample code I also modified your method to perform the deduplication at the same time, so you can avoid doing it later and, more important, allows for a fair time comparison with my approach.
You will see that using a set to keep the kmers offers us free deduplication, and is faster than using a list, as it has not to be traversed.
For the search of the kmer in the reference, given that you were doing exact searches, using a regex is overkill. It's far more cheaper to do a standard search. In this code, I used the methods provided by the Seq class: find and index. The idea is to find the first occurrence starting from the beginning, and the repeat the search starting with the next position after the last index found (if you want to avoid overlaps, then start after the last position found plus the k-mer size).
The code generated follows:
import re
from pathlib import Path
from timeit import timeit
from Bio.Seq import Seq
from Bio.SeqIO.FastaIO import FastaIterator
def reverse_complement(sequence: Seq) -> Seq:
return sequence.reverse_complement()
def generate_kmers(sequence: Seq, kmer_size: int) -> set[Seq]:
return {
Seq(sequence[i : i + kmer_size]) for i in range(len(sequence) - kmer_size + 1)
}
def generate_kmers_original(sequence: Seq, kmer_size: int) -> list[Seq]:
kmers: list[Seq] = []
start = kmer_size - 1
for i in range(start, len(sequence), 1):
current_micro_seq = Seq(sequence[i - start : i + 1])
# We had to add this check to avoid the duplication of k-mers
if current_micro_seq not in kmers:
kmers.append(current_micro_seq)
return kmers
def load_fasta(fasta_file: str) -> dict[str, dict[str, Seq]]:
fasta_dict: dict[str, dict[str, Seq]] = {}
with Path(fasta_file).open("r", encoding="UTF-8") as genome:
for record in FastaIterator(genome):
seq = record.seq.lower()
fasta_dict[record.id] = {"5to3": seq, "3to5": reverse_complement(seq)}
return fasta_dict
if __name__ == "__main__":
# Load the big fasta file
chr_sequences = load_fasta(
".../Saccharomyces_cerevisiae/S288c_R64/fasta/scerevisiae.S288c_R64.fasta"
)
# Generate the micro-sequences
micro_size = 8
sequences = Seq(
"ATGACTAACGAAAAGGTCTGGATAGAGAAGTTGGATAATCCAACTCTTTCAGTGTTACCACATGACTTTTTACGCCCACAATCTTTAT"
).lower()
micro_sequences = generate_kmers(sequences, micro_size)
# k-mer generation benchmark
test_size = 1000
kmer_generation_time = timeit(
"generate_kmers(sequences, micro_size)", number=test_size, globals=globals()
)
kmer_generation_original_time = timeit(
"generate_kmers_original(sequences, micro_size)",
number=test_size,
globals=globals(),
)
print(f"New k-mer generation time : {kmer_generation_time}")
print(f"Original k-mer generation time: {kmer_generation_original_time}")
print(f"There are {len(micro_sequences)} k-mers")
# Search for the kmers in the reference
def find_kmers_original(sequence: Seq, kmer: Seq) -> list[int]:
positions = [
m.start()
for m in re.finditer(
pattern=r"(?=(" + str(kmer) + "))", string=str(sequence)
)
]
return positions
def find_kmers_find(sequence: Seq, kmer: Seq) -> list[int]:
current = 0
positions: list[int] = []
while current < len(sequence):
index = sequence.find(kmer, current)
if index == -1:
break
positions.append(index)
current = index + 1
return positions
def find_kmers_index(sequence: Seq, kmer: Seq) -> list[int]:
positions: list[int] = []
current = 0
try:
while True:
index = sequence.index(kmer, current)
positions.append(index)
current = index + 1
except ValueError:
# Exception thrown when the kmer is not found
# This is our exit condition
pass
return positions
# k-mer search benchmark
test_size = 1000
haystack = next(iter(chr_sequences.values()))["5to3"]
needle = next(iter(micro_sequences))
search_original_time = timeit(
"find_kmers_original(haystack, needle)",
number=test_size,
globals=globals(),
)
search_find_time = timeit(
"find_kmers_find(haystack, needle)",
number=test_size,
globals=globals(),
)
search_index_time = timeit(
"find_kmers_index(haystack, needle)",
number=test_size,
globals=globals(),
)
print(f"Search with original time: {search_original_time}")
print(f"Search with find time : {search_find_time}")
print(f"Search with index time : {search_index_time}")
# Actual calculus
genome_count = 0
chr_count: dict[str, int] = {}
chr_locations: dict[str, dict[str, list[int]]] = {}
micro_fragment_stats: dict[
int, dict[str, int | dict[str, int] | dict[str, dict[str, list[int]]]]
] = {}
for ii_micro, micro_seq in enumerate(micro_sequences):
for chr_counter, (chromosome, contents) in enumerate(chr_sequences.items()):
pos = find_kmers_find(contents["5to3"], micro_seq)
rc_pos = find_kmers_find(contents["3to5"], micro_seq)
chr_locations[chromosome] = {"5to3": pos, "3to5": rc_pos}
chr_counter += len(pos) + len(rc_pos)
chr_count[chromosome] = chr_counter
genome_count += chr_counter
micro_fragment_stats[ii_micro] = {
"occurrences genome": genome_count,
"occurrences chromosomes": chr_count,
"locations chromosomes": chr_locations,
}
The output of this toy example is:
New k-mer generation time : 0.6696164240129292
Original k-mer generation time: 5.967410315992311
There are 81 k-mers
Search with original time: 3.1360475399997085
Search with find time : 0.5738343889825046
Search with index time : 0.5662875371053815
You can see that the k-mer generation is 9x faster and the search without the regex is around 5.5x faster.
In general, you will be better taking advantage of comprehensions and built-in data types (like the sets used here). And using the more simple approach also helps with performance. Regexes are powerful, but they need their time; if they are not required, better to avoid them. Specially in loops, where every small performance change is amplified.
Besides all of this benchmarking, you can also try to add the approach introduced by #Ghothi where the long and short sequences are exchanged. Maybe it could lead to some further improvement.
As a side note, Seq.find and Seq.index seems to offer the same performance, but I find it cleaner and more elegant the Seq.index version: you don't need a weird value to test against and the code intent is clearer. Also, the performance is slightly better, as it is avoiding a comparison in the loop, but this is a very minor improvement.

Somewhere inside my loop it's not appending results to a list. Why?

So I have two files/dictionaries I want to compare, using a binary search implementation (yes, this is very obviously homework).
One file is
american-english
Amazon
Americana
Americanization
Civilization
And the other file is
british-english
Amazon
Americana
Americanisation
Civilisation
The code below should be pretty straight forward. Import files, compare them, return differences. However, somewhere near the bottom, where it says entry == found_difference: I feel as if the debugger skips right over, even though I can see the two variables in memory being different, and I only get the final element returned in the end. Where am I going wrong?
# File importer
def wordfile_to_list(filename):
"""Converts a list of words to a Python list"""
wordlist = []
with open(filename) as f:
for line in f:
wordlist.append(line.rstrip("\n"))
return wordlist
# Binary search algorithm
def binary_search(sorted_list, element):
"""Search for element in list using binary search. Assumes sorted list"""
matches = []
index_start = 0
index_end = len(sorted_list)
while (index_end - index_start) > 0:
index_current = (index_end - index_start) // 2 + index_start
if element == sorted_list[index_current]:
return True
elif element < sorted_list[index_current]:
index_end = index_current
elif element > sorted_list[index_current]:
index_start = index_current + 1
return element
# Check file differences using the binary search algorithm
def wordfile_differences_binarysearch(file_1, file_2):
"""Finds the differences between two plaintext lists,
using binary search algorithm, and returns them in a new list"""
wordlist_1 = wordfile_to_list(file_1)
wordlist_2 = wordfile_to_list(file_2)
matches = []
for entry in wordlist_1:
found_difference = binary_search(sorted_list=wordlist_2, element=entry)
if entry == found_difference:
pass
else:
matches.append(found_difference)
return matches
# Check if it works
differences = wordfile_differences_binarysearch(file_1="british-english", file_2="american-english")
print(differences)

You don't have an else suite for your if statement. Your if statement does nothing (it uses pass when the test is true, skipped otherwise).
You do have an else suite for the for loop:
for entry in wordlist_1:
# ...
else:
matches.append(found_difference)
A for loop can have an else suite as well; it is executed when a loop completes without a break statement. So when your for loop completes, the current value for found_difference is appended; so whatever was assigned last to that name.
Fix your indentation if the else suite was meant to be part of the if test:
for entry in wordlist_1:
found_difference = binary_search(sorted_list=wordlist_2, element=entry)
if entry == found_difference:
pass
else:
matches.append(found_difference)
However, you shouldn't use a pass statement there, just invert the test:
matches = []
for entry in wordlist_1:
found_difference = binary_search(sorted_list=wordlist_2, element=entry)
if entry != found_difference:
matches.append(found_difference)
Note that the variable name matches feels off here; you are appending words that are missing in the other list, not words that match. Perhaps missing is a better variable name here.
Note that your binary_search() function always returns element, the word you searched on. That'll always be equal to the element you passed in, so you can't use that to detect if a word differed! You need to unindent that last return line and return False instead:
def binary_search(sorted_list, element):
"""Search for element in list using binary search. Assumes sorted list"""
matches = []
index_start = 0
index_end = len(sorted_list)
while (index_end - index_start) > 0:
index_current = (index_end - index_start) // 2 + index_start
if element == sorted_list[index_current]:
return True
elif element < sorted_list[index_current]:
index_end = index_current
elif element > sorted_list[index_current]:
index_start = index_current + 1
return False
Now you can use a list comprehension in your wordfile_differences_binarysearch() loop:
[entry for entry in wordlist_1 if not binary_search(wordlist_2, entry)]
Last but not least, you don't have to re-invent the binary seach wheel, just use the bisect module:
from bisect import bisect_left
def binary_search(sorted_list, element):
return sorted_list[bisect(sorted_list, element)] == element

With sets
Binary search is used to improve efficiency of an algorithm, and decrease complexity from O(n) to O(log n).
Since the naive approach would be to check every word in wordlist1 for every word in wordlist2, the complexity would be O(n**2).
Using binary search would help to get O(n * log n), which is already much better.
Using sets, you could get O(n):
american = """Amazon
Americana
Americanization
Civilization"""
british = """Amazon
Americana
Americanisation
Civilisation"""
american = {line.strip() for line in american.split("\n")}
british = {line.strip() for line in british.split("\n")}
You could get the american words not present in the british dictionary:
print(american - british)
# {'Civilization', 'Americanization'}
You could get the british words not present in the american dictionary:
print(british - american)
# {'Civilisation', 'Americanisation'}
You could get the union of the two last sets. I.e. words that are present in exactly one dictionary:
print(american ^ british)
# {'Americanisation', 'Civilisation', 'Americanization', 'Civilization'}
This approach is faster and more concise than any binary search implementation. But if you really want to use it, as usual, you cannot go wrong with #MartijnPieters' answer.
With two iterators
Since you know the two lists are sorted, you could simply iterate in parallel over the two sorted lists and look for any difference:
american = """Amazon
Americana
Americanism
Americanization
Civilization"""
british = """Amazon
Americana
Americanisation
Americanism
Civilisation"""
american = [line.strip() for line in american.split("\n")]
british = [line.strip() for line in british.split("\n")]
n1, n2 = len(american), len(british)
i, j = 0, 0
while True:
try:
w1 = american[i]
w2 = british[j]
if w1 == w2:
i += 1
j += 1
elif w1 < w2:
print('%s is in american dict only' % w1)
i += 1
else:
print('%s is in british dict only' % w2)
j += 1
except IndexError:
break
for w1 in american[i:]:
print('%s is in american dict only' % w1)
for w2 in british[j:]:
print('%s is in british dict only' % w2)
It outputs:
Americanisation is in british dict only
Americanization is in american dict only
Civilisation is in british dict only
Civilization is in american dict only
It's O(n) as well.

Python Unknown pattern finding

Okay, basically what I want is to compress a file by reusing code and then at runtime replace missing code. What I've come up with is really ugly and slow, at least it works. The problem is that the file has no specific structure, for example 'aGVsbG8=\n', as you can see it's base64 encoding. My function is really slow because the length of the file is 1700+ and it checks for patterns 1 character at the time. Please help me with new better code or at least help me with optimizing what I got :). Anything that helps is welcome! BTW i have already tried compression libraries but they didn't compress as good as my ugly function.
def c_long(inp, cap=False, b=5):
import re,string
if cap is False: cap = len(inp)
es = re.escape; le=len; ref = re.findall; ran = range; fi = string.find
c = b;inpc = inp;pattern = inpc[:b]; l=[]
rep = string.replace; ins = list.insert
while True:
if c == le(inpc) and le(inpc) > b+1: c = b; inpc = inpc[1:]; pattern = inpc[:b]
elif le(inpc) <= b: break
if c == cap: c = b; inpc = inpc[1:]; pattern = inpc[:b]
p = ref(es(pattern),inp)
pattern += inpc[c]
if le(p) > 1 and le(pattern) >= b+1:
if l == []: l = [[pattern,le(p)+le(pattern)]]
elif le(ref(es(inpc[:c+2]),inp))+le(inpc[:c+2]) < le(p)+le(pattern):
x = [pattern,le(p)+le(inpc[:c+1])]
for i in ran(le(l)):
if x[1] >= l[i][1] and x[0][:-1] not in l[i][0]: ins(l,i,x); break
elif x[1] >= l[i][1] and x[0][:-1] in l[i][0]: l[i] = x; break
inpc = inpc[:fi(inpc,x[0])] + inpc[le(x[0]):]
pattern = inpc[:b]
c = b-1
c += 1
d = {}; c = 0
s = ran(le(l))
for x in l: inp = rep(inp,x[0],'{%d}' % s[c]); d[str(s[c])] = x[0]; c += 1
return [inp,d]
def decompress(inp,l): return apply(inp.format, [l[str(x)] for x in sorted([int(x) for x in l.keys()])])

The easiest way to compress base64-encoded data is to first convert it to binary data -- this will already save 25 percent of the storage space:
>>> s = "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=\n"
>>> t = s.decode("base64")
>>> len(s)
37
>>> len(t)
26
In most cases, you can compress the string even further using some compression algorithm, like t.encode("bz2") or t.encode("zlib").
A few remarks on your code: There are lots of factors that make the code hard to read: inconsistent spacing, overly long lines, meaningless variable names, unidiomatic code, etc. An example: Your decompress() function could be equivalently written as
def decompress(compressed_string, substitutions):
subst_list = [substitutions[k] for k in sorted(substitutions, key=int)]
return compressed_string.format(*subst_list)
Now it's already much more obvious what it does. You could go one step further: Why is substitutions a dictionary with the string keys "0", "1" etc.? Not only is it strange to use strings instead of integers -- you don't need the keys at all! A simple list will do, and decompress() will simplify to
def decompress(compressed_string, substitutions):
return compressed_string.format(*substitutions)
You might think all this is secondary, but if you make the rest of your code equally readable, you will find the bugs in your code yourself. (There are bugs -- it crashes for "abcdefgabcdefg" and many other strings.)

Typically one would pump the program through a compression algorithm optimized for text, then run that through exec, e.g.
code="""..."""
exec(somelib.decompress(code), globals=???, locals=???)
It may be the case that .pyc/.pyo files are compressed already, and one could check by creating one with x="""aaaaaaaa""", then increasing the length to x="""aaaaaaaaaaaaaaaaaaaaaaa...aaaa""" and seeing if the size changes appreciably.

Finding combinations of stems and endings

I have mappings of "stems" and "endings" (may not be the correct words) that look like so:
all_endings = {
'birth': set(['place', 'day', 'mark']),
'snow': set(['plow', 'storm', 'flake', 'man']),
'shoe': set(['lace', 'string', 'maker']),
'lock': set(['down', 'up', 'smith']),
'crack': set(['down', 'up',]),
'arm': set(['chair']),
'high': set(['chair']),
'over': set(['charge']),
'under': set(['charge']),
}
But much longer, of course. I also made the corresponding dictionary the other way around:
all_stems = {
'chair': set(['high', 'arm']),
'charge': set(['over', 'under']),
'up': set(['lock', 'crack', 'vote']),
'down': set(['lock', 'crack', 'fall']),
'smith': set(['lock']),
'place': set(['birth']),
'day': set(['birth']),
'mark': set(['birth']),
'plow': set(['snow']),
'storm': set(['snow']),
'flake': set(['snow']),
'man': set(['snow']),
'lace': set(['shoe']),
'string': set(['shoe']),
'maker': set(['shoe']),
}
I've now tried to come up with an algorithm to find any match of two or more "stems" that match two or more "endings". Above, for example, it would match down and up with lock and crack, resulting in
lockdown
lockup
crackdown
crackup
But not including 'upvote', 'downfall' or 'locksmith' (and it's this that causes me the biggest problems). I get false positives like:
pancake
cupcake
cupboard
But I'm just going round in "loops". (Pun intended) and I don't seem to get anywhere. I'd appreciate any kick in the right direction.
Confused and useless code so far, which you probably should just ignore:
findings = defaultdict(set)
for stem, endings in all_endings.items():
# What stems have matching endings:
for ending in endings:
otherstems = all_stems[ending]
if not otherstems:
continue
for otherstem in otherstems:
# Find endings that also exist for other stems
otherendings = all_endings[otherstem].intersection(endings)
if otherendings:
# Some kind of match
findings[stem].add(otherstem)
# Go through this in order of what is the most stems that match:
MINMATCH = 2
for match in sorted(findings.values(), key=len, reverse=True):
for this_stem in match:
other_stems = set() # Stems that have endings in common with this_stem
other_endings = set() # Endings this stem have in common with other stems
this_endings = all_endings[this_stem]
for this_ending in this_endings:
for other_stem in all_stems[this_ending] - set([this_stem]):
matching_endings = this_endings.intersection(all_endings[other_stem])
if matching_endings:
other_endings.add(this_ending)
other_stems.add(other_stem)
stem_matches = all_stems[other_endings.pop()]
for other in other_endings:
stem_matches = stem_matches.intersection(all_stems[other])
if len(stem_matches) >= MINMATCH:
for m in stem_matches:
for e in all_endings[m]:
print(m+e)

It's not particularly pretty, but this is quite straightforward if you break your dictionary down into two lists, and use explicit indices:
all_stems = {
'chair' : set(['high', 'arm']),
'charge': set(['over', 'under']),
'fall' : set(['down', 'water', 'night']),
'up' : set(['lock', 'crack', 'vote']),
'down' : set(['lock', 'crack', 'fall']),
}
endings = all_stems.keys()
stem_sets = all_stems.values()
i = 0
for target_stem_set in stem_sets:
i += 1
j = 0
remaining_stems = stem_sets[i:]
for remaining_stem_set in remaining_stems:
j += 1
union = target_stem_set & remaining_stem_set
if len(union) > 1:
print "%d matches found" % len(union)
for stem in union:
print "%s%s" % (stem, endings[i-1])
print "%s%s" % (stem, endings[j+i-1])
Output:
$ python stems_and_endings.py
2 matches found
lockdown
lockup
crackdown
crackup
Basically all we're doing is iterating through each set in turn, and comparing it with every remaining set to see if there are more than two matches. We never have to try sets that fall earlier than the current set, because they've already been compared in a prior iteration. The rest (indexing, etc.) is just book-keeping.

I think that the way I avoid those false positives is by removing candidates with no words in the intersection of stems - If this make sense :(
Please have a look and please let me know if I am missing something.
#using all_stems and all_endings from the question
#this function is declared at the end of this answer
two_or_more_stem_combinations = get_stem_combinations(all_stems)
print "two_or_more_stem_combinations", two_or_more_stem_combinations
#this print shows ... [set(['lock', 'crack'])]
for request in two_or_more_stem_combinations:
#we filter the initial index to only look for sets or words in the request
candidates = filter(lambda x: x[0] in request, all_endings.items())
#intersection of the words for the request
words = candidates[0][1]
for c in candidates[1:]:
words=words.intersection(c[1])
#it's handy to have it in a dict
candidates = dict(candidates)
#we need to remove those that do not contain
#any words after the intersection of stems of all the candidates
candidates_to_remove = set()
for c in candidates.items():
if len(c[1].intersection(words)) == 0:
candidates_to_remove.add(c[0])
for key in candidates_to_remove:
del candidates[key]
#now we know what to combine
for c in candidates.keys():
print "combine", c , "with", words
Output :
combine lock with set(['down', 'up'])
combine crack with set(['down', 'up'])
As you can see this solution doesn't contain those false positives.
Edit: complexity
And the complexity of this solution doesn't get worst than O(3n) in the worst scenario - without taking into account accessing dictionaries. And
for most executions the first filter narrows down quite a lot the solution space.
Edit: getting the stems
This function basically explores recursively the dictionary all_stems and finds the combinations of two or more endings for which two or more stems coincide.
def get_stems_recursive(stems,partial,result,at_least=2):
if len(partial) >= at_least:
stem_intersect=all_stems[partial[0]]
for x in partial[1:]:
stem_intersect = stem_intersect.intersection(all_stems[x])
if len(stem_intersect) < 2:
return
result.append(stem_intersect)
for i in range(len(stems)):
remaining = stems[i+1:]
get_stems_recursive(remaining,partial + [stems[i][0]],result)
def get_stem_combinations(all_stems,at_least=2):
result = []
get_stems_recursive(all_stems.items(),list(),result)
return result
two_or_more_stem_combinations = get_stem_combinations(all_stems)

== Edited answer: ==
Well, here's another iteration for your consideration with the mistakes I made the first time addressed. Actually the result is code that is even shorter and simpler. The doc for combinations says that "if the input elements are unique, there will be no repeat values in each combination", so it should only be forming and testing the minimum number of intersections. It also appears that determining endings_by_stems isn't necessary.
from itertools import combinations
MINMATCH = 2
print 'all words with at least', MINMATCH, 'endings in common:'
for (word0,word1) in combinations(stems_by_endings, 2):
ending_words0 = stems_by_endings[word0]
ending_words1 = stems_by_endings[word1]
common_endings = ending_words0 & ending_words1
if len(common_endings) >= MINMATCH:
for stem in common_endings:
print ' ', stem+word0
print ' ', stem+word1
# all words with at least 2 endings in common:
# lockdown
# lockup
# falldown
# fallup
# crackdown
# crackup
== Previous answer ==
I haven't attempted much optimizing, but here's a somewhat brute-force -- but short -- approach that first calculates 'ending_sets' for each stem word, and then finds all the stem words that have common ending_sets with at least the specified minimum number of common endings.
In the final phase it prints out all the possible combinations of these stem + ending words it has detected that have meet the criteria. I tried to make all variable names as descriptive as possible to make it easy to follow. ;-) I've also left out the definitions of all_endings' and 'all+stems.
from collections import defaultdict
from itertools import combinations
ending_sets = defaultdict(set)
for stem in all_stems:
# create a set of all endings that have this as stem
for ending in all_endings:
if stem in all_endings[ending]:
ending_sets[stem].add(ending)
MINMATCH = 2
print 'all words with at least', MINMATCH, 'endings in common:'
for (word0,word1) in combinations(ending_sets, 2):
ending_words0 = ending_sets[word0]
ending_words1 = ending_sets[word1]
if len(ending_words0) >= MINMATCH and ending_words0 == ending_words1:
for stem in ending_words0:
print ' ', stem+word0
print ' ', stem+word1
# output
# all words with at least 2 endings in common:
# lockup
# lockdown
# crackup
# crackdown

If you represent your stemming relationships in a square binary arrays (where 1 means "x can follow y", for instance, and where other elements are set to 0), what you are trying to do is equivalent to looking for "broken rectangles" filled with ones:
... lock **0 crack **1 ...
... ...
down ... 1 0 1 1
up ... 1 1 1 1
... ...
Here, lock, crack, and **1 (example word) can be matched with down and up (but not word **0). The stemming relationships draw a 2x3 rectangle filled with ones.
Hope this helps!