I have mappings of "stems" and "endings" (may not be the correct words) that look like so:
all_endings = {
'birth': set(['place', 'day', 'mark']),
'snow': set(['plow', 'storm', 'flake', 'man']),
'shoe': set(['lace', 'string', 'maker']),
'lock': set(['down', 'up', 'smith']),
'crack': set(['down', 'up',]),
'arm': set(['chair']),
'high': set(['chair']),
'over': set(['charge']),
'under': set(['charge']),
}
But much longer, of course. I also made the corresponding dictionary the other way around:
all_stems = {
'chair': set(['high', 'arm']),
'charge': set(['over', 'under']),
'up': set(['lock', 'crack', 'vote']),
'down': set(['lock', 'crack', 'fall']),
'smith': set(['lock']),
'place': set(['birth']),
'day': set(['birth']),
'mark': set(['birth']),
'plow': set(['snow']),
'storm': set(['snow']),
'flake': set(['snow']),
'man': set(['snow']),
'lace': set(['shoe']),
'string': set(['shoe']),
'maker': set(['shoe']),
}
I've now tried to come up with an algorithm to find any match of two or more "stems" that match two or more "endings". Above, for example, it would match down and up with lock and crack, resulting in
lockdown
lockup
crackdown
crackup
But not including 'upvote', 'downfall' or 'locksmith' (and it's this that causes me the biggest problems). I get false positives like:
pancake
cupcake
cupboard
But I'm just going round in "loops". (Pun intended) and I don't seem to get anywhere. I'd appreciate any kick in the right direction.
Confused and useless code so far, which you probably should just ignore:
findings = defaultdict(set)
for stem, endings in all_endings.items():
# What stems have matching endings:
for ending in endings:
otherstems = all_stems[ending]
if not otherstems:
continue
for otherstem in otherstems:
# Find endings that also exist for other stems
otherendings = all_endings[otherstem].intersection(endings)
if otherendings:
# Some kind of match
findings[stem].add(otherstem)
# Go through this in order of what is the most stems that match:
MINMATCH = 2
for match in sorted(findings.values(), key=len, reverse=True):
for this_stem in match:
other_stems = set() # Stems that have endings in common with this_stem
other_endings = set() # Endings this stem have in common with other stems
this_endings = all_endings[this_stem]
for this_ending in this_endings:
for other_stem in all_stems[this_ending] - set([this_stem]):
matching_endings = this_endings.intersection(all_endings[other_stem])
if matching_endings:
other_endings.add(this_ending)
other_stems.add(other_stem)
stem_matches = all_stems[other_endings.pop()]
for other in other_endings:
stem_matches = stem_matches.intersection(all_stems[other])
if len(stem_matches) >= MINMATCH:
for m in stem_matches:
for e in all_endings[m]:
print(m+e)
It's not particularly pretty, but this is quite straightforward if you break your dictionary down into two lists, and use explicit indices:
all_stems = {
'chair' : set(['high', 'arm']),
'charge': set(['over', 'under']),
'fall' : set(['down', 'water', 'night']),
'up' : set(['lock', 'crack', 'vote']),
'down' : set(['lock', 'crack', 'fall']),
}
endings = all_stems.keys()
stem_sets = all_stems.values()
i = 0
for target_stem_set in stem_sets:
i += 1
j = 0
remaining_stems = stem_sets[i:]
for remaining_stem_set in remaining_stems:
j += 1
union = target_stem_set & remaining_stem_set
if len(union) > 1:
print "%d matches found" % len(union)
for stem in union:
print "%s%s" % (stem, endings[i-1])
print "%s%s" % (stem, endings[j+i-1])
Output:
$ python stems_and_endings.py
2 matches found
lockdown
lockup
crackdown
crackup
Basically all we're doing is iterating through each set in turn, and comparing it with every remaining set to see if there are more than two matches. We never have to try sets that fall earlier than the current set, because they've already been compared in a prior iteration. The rest (indexing, etc.) is just book-keeping.
I think that the way I avoid those false positives is by removing candidates with no words in the intersection of stems - If this make sense :(
Please have a look and please let me know if I am missing something.
#using all_stems and all_endings from the question
#this function is declared at the end of this answer
two_or_more_stem_combinations = get_stem_combinations(all_stems)
print "two_or_more_stem_combinations", two_or_more_stem_combinations
#this print shows ... [set(['lock', 'crack'])]
for request in two_or_more_stem_combinations:
#we filter the initial index to only look for sets or words in the request
candidates = filter(lambda x: x[0] in request, all_endings.items())
#intersection of the words for the request
words = candidates[0][1]
for c in candidates[1:]:
words=words.intersection(c[1])
#it's handy to have it in a dict
candidates = dict(candidates)
#we need to remove those that do not contain
#any words after the intersection of stems of all the candidates
candidates_to_remove = set()
for c in candidates.items():
if len(c[1].intersection(words)) == 0:
candidates_to_remove.add(c[0])
for key in candidates_to_remove:
del candidates[key]
#now we know what to combine
for c in candidates.keys():
print "combine", c , "with", words
Output :
combine lock with set(['down', 'up'])
combine crack with set(['down', 'up'])
As you can see this solution doesn't contain those false positives.
Edit: complexity
And the complexity of this solution doesn't get worst than O(3n) in the worst scenario - without taking into account accessing dictionaries. And
for most executions the first filter narrows down quite a lot the solution space.
Edit: getting the stems
This function basically explores recursively the dictionary all_stems and finds the combinations of two or more endings for which two or more stems coincide.
def get_stems_recursive(stems,partial,result,at_least=2):
if len(partial) >= at_least:
stem_intersect=all_stems[partial[0]]
for x in partial[1:]:
stem_intersect = stem_intersect.intersection(all_stems[x])
if len(stem_intersect) < 2:
return
result.append(stem_intersect)
for i in range(len(stems)):
remaining = stems[i+1:]
get_stems_recursive(remaining,partial + [stems[i][0]],result)
def get_stem_combinations(all_stems,at_least=2):
result = []
get_stems_recursive(all_stems.items(),list(),result)
return result
two_or_more_stem_combinations = get_stem_combinations(all_stems)
== Edited answer: ==
Well, here's another iteration for your consideration with the mistakes I made the first time addressed. Actually the result is code that is even shorter and simpler. The doc for combinations says that "if the input elements are unique, there will be no repeat values in each combination", so it should only be forming and testing the minimum number of intersections. It also appears that determining endings_by_stems isn't necessary.
from itertools import combinations
MINMATCH = 2
print 'all words with at least', MINMATCH, 'endings in common:'
for (word0,word1) in combinations(stems_by_endings, 2):
ending_words0 = stems_by_endings[word0]
ending_words1 = stems_by_endings[word1]
common_endings = ending_words0 & ending_words1
if len(common_endings) >= MINMATCH:
for stem in common_endings:
print ' ', stem+word0
print ' ', stem+word1
# all words with at least 2 endings in common:
# lockdown
# lockup
# falldown
# fallup
# crackdown
# crackup
== Previous answer ==
I haven't attempted much optimizing, but here's a somewhat brute-force -- but short -- approach that first calculates 'ending_sets' for each stem word, and then finds all the stem words that have common ending_sets with at least the specified minimum number of common endings.
In the final phase it prints out all the possible combinations of these stem + ending words it has detected that have meet the criteria. I tried to make all variable names as descriptive as possible to make it easy to follow. ;-) I've also left out the definitions of all_endings' and 'all+stems.
from collections import defaultdict
from itertools import combinations
ending_sets = defaultdict(set)
for stem in all_stems:
# create a set of all endings that have this as stem
for ending in all_endings:
if stem in all_endings[ending]:
ending_sets[stem].add(ending)
MINMATCH = 2
print 'all words with at least', MINMATCH, 'endings in common:'
for (word0,word1) in combinations(ending_sets, 2):
ending_words0 = ending_sets[word0]
ending_words1 = ending_sets[word1]
if len(ending_words0) >= MINMATCH and ending_words0 == ending_words1:
for stem in ending_words0:
print ' ', stem+word0
print ' ', stem+word1
# output
# all words with at least 2 endings in common:
# lockup
# lockdown
# crackup
# crackdown
If you represent your stemming relationships in a square binary arrays (where 1 means "x can follow y", for instance, and where other elements are set to 0), what you are trying to do is equivalent to looking for "broken rectangles" filled with ones:
... lock **0 crack **1 ...
... ...
down ... 1 0 1 1
up ... 1 1 1 1
... ...
Here, lock, crack, and **1 (example word) can be matched with down and up (but not word **0). The stemming relationships draw a 2x3 rectangle filled with ones.
Hope this helps!
Related
I am trying to create a name matcher to compare say, 'JOHN LEWIS' to 'JOHN SMITH LEWIS'. They are clearly the same person and I want to create a function where when you enter those names, it turns it into a list then gives you the matching names.
The problem is that my loop is returning that 'LEWIS' matches with 'LEWIS' and 'SMITH' matches with 'LEWIS' because of the order that it is in.
from pyjarowinkler import distance
entered_name = 'JOHN LEWIS'.split(' ') # equals ['JOHN','LEWIS']
system_name = 'JOHN SMITH LEWIS'.split(' ') # equals ['JOHN','SMITH','LEWIS']
ratio = []
for i in entered_name:
maximum = 0
for j in system_name:
score = distance.get_jaro_distance(i, j, winkler=True,
scaling=0.1)
while score > maximum:
maximum = score
new = (i, j, maximum)
system_name.remove(i)
#removes that name from the original list
ratio.append(new)
would return something like: [('JOHN', 'JOHN', 1.0), ('LEWIS', 'SMITH', 0.47)]
and not: [('JOHN', 'JOHN', 1.0), ('LEWIS', 'LEWIS', 1.0)] <- this is what I want.
Also, if you try something like 'ALLY A ARM' with 'ALLY ARIANA ARMANI', it matches 'ALLY' twice if you don't do that remove(i) line. This is why I only want unique matches!
I just keep getting errors or the answers that I am not looking for.
The issue is with your system_name.remove(i) line. First of all, it's usually a bad idea to modify a list while you're iterating through that list. This can lead to unexpected behavior. In your case, here's what your code is doing:
First time through, matches 'JOHN', and 'JOHN'. No problem.
Removes 'JOHN' from system_name. Now system_name = ['SMITH', 'LEWIS'].
Second time through, i = 'LEWIS', j = 'SMITH', score = .47 which is greater than 0, so your check score > maximum passes
We set maximum = score
We set new = ('LEWIS', 'SMITH', 0.47)
We remove 'LEWIS' from system_name. Now system_name = ['SMITH']. Uh oh...
Simple rewrite below, using an if instead of a while loop because the while loop is totally unnecessary:
for i in entered_name:
maximum = 0
for j in system_name:
score = distance.get_jaro_distance(i, j, winkler=True,
scaling=0.1)
if score > maximum:
maximum = score
new = (i, j, maximum)
system_name.remove(new[1]) # want to remove 'SMITH' in the example, not 'LEWIS'
ratio.append(new)
All I did was move the system_name.remove() call outside of the loop over system_name, and replace i with j (using new[1] since I'm outside of the j loop).
Jaro-Winkler distance is for comparison of sequences, there is no need to compare individual elements as if you were trying to find an edit distance between individual characters rather than whole words.
With that in mind, one should probably treat parts of a name as individual letters, and the whole name as a word, comparing, say, "JL" vs. "JSL" instead of "JOHN LEWIS" and "JOHN SMITH LEWIS":
import string
import itertools
from pyjarowinkler import distance
WORDS_CACHE = {}
def next_letter():
base = ""
while True:
for ch in string.ascii_lowercase:
yield base + ch
base += ch
GENERATOR = next_letter()
def encode(word):
if word not in WORDS_CACHE:
WORDS_CACHE[word] = GENERATOR.next()
return WORDS_CACHE[word]
def score(first_name, second_name):
return distance.get_jaro_distance(
"".join(map(encode, first_name.split())),
"".join(map(encode, second_name.split())),
)
I need to loop trough n lines of a file and for any i between 1 and n-1 to get the difference between words of line(n-1) - line(n) (eg. line[i]word[j] - line[i+1]word[j] etc .. )
Input :
Hey there !
Hey thre !
What a suprise.
What a uprise.
I don't know what to do.
I don't know wt to do.
Output:
e
s
ha
The goal is to extract the missing character(s) between two consecutive line words only.
I'm new to python so if you can guide me through writing the code, I would be more than thankful.
Without any lib :
def extract_missing_chars(s1, s2):
if len(s1) < len(s2):
return extract_missing_chars(s2, s1)
i = 0
to_return = []
for c in s1:
if s2[i] != c:
to_return.append(c)
else:
i += 1
return to_return
f = open('testfile')
l1 = f.readline()
while l1:
l2 = f.readline()
print(''.join(extract_missing_chars(l1, l2)))
l1 = f.readline()
Your example indicates that you want the comparisons between pairs of lines. This is different from defining it as line(n-1)-line(n) which would give you 5 results, not 3.
The result also depends on what you consider to be differences. Is it positional, is it simply based on missing letters from the odd lines or are the differences applicable in both directions.
(e.g. "boat"-"tub" = "boat", "oa" or "oau" ?).
You also have to decide if you want the differences to be case sensitive or not.
Here's an example where computation of the differences is centralized in a function so that you can change the rules more easily. It assumes that "boat"-"tub" = "oau".
lines = """Hey there !
Hey thre !
What a suprise.
What a uprise.
I don't know what to do.
I don't know wt to do.
""".split('\n')
def differences(word1,word2):
if isinstance(word1,list):
return "".join( differences(w1,w2) for w1,w2 in zip(word1+[""]*len(word2),word2+[""]*len(word1)) )
return "".join( c*abs(word1.count(c)-word2.count(c)) for c in set(word1+word2) )
result = [ differences(line1.split(),line2.split()) for line1,line2 in zip(lines[::2],lines[1::2]) ]
# ['e', 's', 'ha']
Note that line processing for result is based on your example (not on your definition).
i ve got a dataframe series which contain sentences. (some are kind of long)
i ve also got 2 dictionaries which contain words as keys and ints as count.
Not all words from strings are present in both dictionaries. Some are in only one, some are in neither.
Dataframe is 124011 units long. function is taking me about 0.4 per string. which is waaaay to long.
W is just a reference value for the dictionary (weights = {}, weights[W] = {})
here is the function:
def match_share(string, W, weights, rel_weight):
words = string.split()
words_counts = Counter(words)
ratios = []
for word in words:
if ((word in weights[W].keys())&(word in rel_weight[W].keys())):
if (weights[W][word]!=0):
ratios.append(words_counts[word]*rel_weight[W][word]/weights[W][word])
else:
ratios.append(0)
if len(words)>0:
ratios = np.divide(ratios, float(len(words)))
ratio = np.sum(ratios)
return ratio
thx
Let's clean it up a bit:
def match_share(string, W, weights, rel_weight):
words = string.split()
words_counts = Counter(words)
words = string.split()
words_counts = Counter(words)
That's redundant! Replace 4 statements with 2:
def match_share(string, W, weights, rel_weight):
words = string.split()
words_counts = Counter(words)
Next:
ratios = []
for word in words:
if ((word in weights[W].keys())&(word in rel_weight[W].keys())):
if (weights[W][word]!=0):
ratios.append(words_counts[word]*rel_weight[W][word]/weights[W][word])
else:
ratios.append(0)
I don't know what you think that code does. I hope you're not being tricky. But .keys returns an iterable, and X in <iterable> is WAY slower than X in <dict>. Also, note: you don't append anything if the innermost (weights[W][word] != 0) condition fails. That might be a bug, since you try to append 0 in another else condition. (I don't know what you're doing, so I'm just pointing it out.) And this is Python, not Perl or C or Java. So no parens required around if <test>:
Let's go with that:
ratios = []
for word in words:
if word in weights[W] and word in rel_weight[W]:
if weights[W][word] != 0:
ratios.append(words_counts[word] * rel_weight[W][word] / weights[W][word])
else:
ratios.append(0)
Next:
if len(words)>0:
ratios = np.divide(ratios, float(len(words)))
You're trying to prevent dividing by zero. But you can use the truthiness of a list to check this, and avoid the comparison:
if words:
ratios = np.divide(ratios, float(len(words)))
The rest is fine, but you don't need the variable.
ratio = np.sum(ratios)
return ratio
With those mods applied, your function looks like this:
def match_share(string, W, weights, rel_weight):
words = string.split()
words_counts = Counter(words)
ratios = []
for word in words:
if word in weights[W] and word in rel_weight[W]:
if weights[W][word] != 0:
ratios.append(words_counts[word] * rel_weight[W][word] / weights[W][word])
else:
ratios.append(0)
if words:
ratios = np.divide(ratios, float(len(words)))
ratio = np.sum(ratios)
return ratio
Looking at it at little harder, I see you're doing this:
word_counts = Counter(words)
for word in words:
append( word_counts[word] * ...)
According to me, that means if "apple" appears 6 times, you are going to append 6*... to the list, once for each word. So you'll have 6 different occurrences of 6*... in your list. Are you sure that's what you want? Or should it be for word in word_counts to just iterate over the distinct words?
Another optimization is to remove lookups from inside your loop. You keep looking up weights[W] and rel_weight[W], even though the value of W never changes. Let's cache those values outside the loop. Also, let's cache a pointer to the ratios.append method.
def match_share(string, W, weights, rel_weight):
words = string.split()
words_counts = Counter(words)
ratios = []
# Cache these values for speed in loop
ratios_append = ratios.append
weights_W = weights[W]
rel_W = rel_weight[W]
for word in words:
if word in weights_W and word in rel_W:
if weights_W[word] != 0:
ratios_append(words_counts[word] * rel_W[word] / weights_W[word])
else:
ratios_append(0)
if words:
ratios = np.divide(ratios, float(len(words)))
ratio = np.sum(ratios)
return ratio
Try that, see how it works. Please look at the bold note above, and the questions. There might be bugs, there might be more ways to speed up.
I think that your time inefficiency may be coming from the fact you are using Counter instead of the dict. Some discussion here suggests that the dict class has parts written in pure c, while counter is written in python.
I suggest altering your code to using a dict and test to see if that provides a faster time
Also why is this code duplicated?:
words = string.split()
words_counts = Counter(words)
words = string.split()
words_counts = Counter(words)
ratios = []
It would be good if you had a profile of that function execution, but here are some generic ideas:
You needlessly get some elements on every iteration. You can extract these before the loop
For example
weights_W = weights[W]
rel_weights_W = rel_weights[W]
You don't need to call .keys() on dicts.
These are equivalent:
word in weights_W.keys()
word in weights_W
Attempt to get values without looking them up first. This will save you one lookup.
For example instead of:
if ((word in weights[W].keys())&(word in rel_weight[W].keys())):
if (weights[W][word]!=0):
you can do:
word_weight = weights_W.get(word)
if word_weight is not None:
word_rel_weight = rel_weights_W.get(word)
if word_rel_weight is not None:
if word_weight != 0: # lookup saved here
I’m trying to replicate the methodology from this article, 538 Post about Most Repetitive Phrases, in which the author mined US presidential debate transcripts to determine the most repetitive phrases for each candidate.
I'm trying to implement this methodology with another dataset in R with the tm package.
Most of the code (GitHub repository) concerns mining the transcripts and assembling counts of each ngram, but I get lost at the prune_substrings() function code below:
def prune_substrings(tfidf_dicts, prune_thru=1000):
pruned = tfidf_dicts
for candidate in range(len(candidates)):
# growing list of n-grams in list form
so_far = []
ngrams_sorted = sorted(tfidf_dicts[candidate].items(), key=operator.itemgetter(1), reverse=True)[:prune_thru]
for ngram in ngrams_sorted:
# contained in a previous aka 'better' phrase
for better_ngram in so_far:
if overlap(list(better_ngram), list(ngram[0])):
#print "PRUNING!! "
#print list(better_ngram)
#print list(ngram[0])
pruned[candidate][ngram[0]] = 0
# not contained, so add to so_far to prevent future subphrases
else:
so_far += [list(ngram[0])]
return pruned
The input of the function, tfidf_dicts, is an array of dictionaries (one for each candidate) with ngrams as keys and tf-idf scores as values. For example, Trump's tf-idf dict begins like this:
trump.tfidf.dict = {'we don't win': 83.2, 'you have to': 72.8, ... }
so the structure of the input is like this:
tfidf_dicts = {trump.tfidf.dict, rubio.tfidf.dict, etc }
MY understanding is that prune_substrings does the following things, but I'm stuck on the else if clause, which is a pythonic thing I don't understand yet.
A. create list : pruned as tfidf_dicts; a list of tfidf dicts for each candidate
B loop through each candidate:
so_far = start an empty list of ngrams gone through so so_far
ngrams_sorted = sorted member's tf-idf dict from smallest to biggest
loop through each ngram in sorted
loop through each better_ngram in so_far
IF overlap b/w (below) == TRUE:
better_ngram (from so_far) and
ngram (from ngrams_sorted)
THEN zero out tf-idf for ngram
ELSE if (WHAT?!?)
add ngram to list, so_far
C. return pruned, i.e. list of unique ngrams sorted in order
Any help at all is much appreciated!
Note the indentation in your code... The else is lined up with the second for, not the if. This is a for-else construct, not an if-else.
In that case, the else is being used to initialize the inner loop, because it will be executed when so_far is empty the first time through, and each time the inner loop runs out of items to iterate through...
I am not sure that this is the most efficient way to achieve these comparisons, but conceptually you can get a sense of the flow with this snippet:
s=[]
for j in "ABCD":
for i in s:
print i,
else:
print "\nelse"
s.append(j)
Output:
else
A
else
A B
else
A B C
else
I would think that in R there is a much better way to do this than nested loops....
4 months later but here's my solution. I'm sure there is a more efficient solution, but for my purposes, it worked. The pythonic for-else doesn't translate to R. So the steps are different.
Take top n ngrams.
Create a list, t, where each element of the list is a logical vector of length n that says whether ngram in question overlaps all other ngrams (but fix 1:x to be false automatically)
Cbind together every element of t into a table, t2
Return only elements of t2 row sum is zero
set elements 1:n to FALSE (i.e. no overlap)
Ouala!
PrunedList Function
#' GetPrunedList
#'
#' takes a word freq df with columns Words and LenNorm, returns df of nonoverlapping strings
GetPrunedList <- function(wordfreqdf, prune_thru = 100) {
#take only first n items in list
tmp <- head(wordfreqdf, n = prune_thru) %>%
select(ngrams = Words, tfidfXlength = LenNorm)
#for each ngram in list:
t <- (lapply(1:nrow(tmp), function(x) {
#find overlap between ngram and all items in list (overlap = TRUE)
idx <- overlap(tmp[x, "ngrams"], tmp$ngrams)
#set overlap as false for itself and higher-scoring ngrams
idx[1:x] <- FALSE
idx
}))
#bind each ngram's overlap vector together to make a matrix
t2 <- do.call(cbind, t)
#find rows(i.e. ngrams) that do not overlap with those below
idx <- rowSums(t2) == 0
pruned <- tmp[idx,]
rownames(pruned) <- NULL
pruned
}
Overlap function
#' overlap
#' OBJ: takes two ngrams (as strings) and to see if they overlap
#' INPUT: a,b ngrams as strings
#' OUTPUT: TRUE if overlap
overlap <- function(a, b) {
max_overlap <- min(3, CountWords(a), CountWords(b))
a.beg <- word(a, start = 1L, end = max_overlap)
a.end <- word(a, start = -max_overlap, end = -1L)
b.beg <- word(b, start = 1L, end = max_overlap)
b.end <- word(b, start = -max_overlap, end = -1L)
# b contains a's beginning
w <- str_detect(b, coll(a.beg, TRUE))
# b contains a's end
x <- str_detect(b, coll(a.end, TRUE))
# a contains b's beginning
y <- str_detect(a, coll(b.beg, TRUE))
# a contains b's end
z <- str_detect(a, coll(b.end, TRUE))
#return TRUE if any of above are true
(w | x | y | z)
}
I'm trying to write a program that mimics a word game where, from a given set of words, it will find the longest possible sequence of words. No word can be used twice.
I can do the matching letters and words up, and storing them into lists, but I'm having trouble getting my head around how to handle the potentially exponential number of possibilities of words in lists. If word 1 matches word 2 and then I go down that route, how do I then back up to see if words 3 or 4 match up with word one and then start their own routes, all stemming from the first word?
I was thinking some way of calling the function inside itself maybe?
I know it's nowhere near doing what I need it to do, but it's a start. Thanks in advance for any help!
g = "audino bagon baltoy banette bidoof braviary bronzor carracosta charmeleon cresselia croagunk darmanitan deino emboar emolga exeggcute gabite girafarig gulpin haxorus"
def pokemon():
count = 1
names = g.split()
first = names[count]
master = []
for i in names:
print (i, first, i[0], first[-1])
if i[0] == first[-1] and i not in master:
master.append(i)
count += 1
first = i
print ("success", master)
if len(master) == 0:
return "Pokemon", first, "does not work"
count += 1
first = names[count]
pokemon()
Your idea of calling a function inside of itself is a good one. We can solve this with recursion:
def get_neighbors(word, choices):
return set(x for x in choices if x[0] == word[-1])
def longest_path_from(word, choices):
choices = choices - set([word])
neighbors = get_neighbors(word, choices)
if neighbors:
paths = (longest_path_from(w, choices) for w in neighbors)
max_path = max(paths, key=len)
else:
max_path = []
return [word] + max_path
def longest_path(choices):
return max((longest_path_from(w, choices) for w in choices), key=len)
Now we just define our word list:
words = ("audino bagon baltoy banette bidoof braviary bronzor carracosta "
"charmeleon cresselia croagunk darmanitan deino emboar emolga "
"exeggcute gabite girafarig gulpin haxorus")
words = frozenset(words.split())
Call longest_path with a set of words:
>>> longest_path(words)
['girafarig', 'gabite', 'exeggcute', 'emolga', 'audino']
A couple of things to know: as you point out, this has exponential complexity, so beware! Also, know that python has a recursion limit!
Using some black magic and graph theory I found a partial solution that might be good (not thoroughly tested).
The idea is to map your problem into a graph problem rather than a simple iterative problem (although it might work too!). So I defined the nodes of the graph to be the first letters and last letters of your words. I can only create edges between nodes of type first and last. I cannot map node first number X to node last number X (a word cannot be followed by it self). And from that your problem is just the same as the Longest path problem which tends to be NP-hard for general case :)
By taking some information here: stackoverflow-17985202 I managed to write this:
g = "audino bagon baltoy banette bidoof braviary bronzor carracosta charmeleon cresselia croagunk darmanitan deino emboar emolga exeggcute gabite girafarig gulpin haxorus"
words = g.split()
begin = [w[0] for w in words] # Nodes first
end = [w[-1] for w in words] # Nodes last
links = []
for i, l in enumerate(end): # Construct edges
ok = True
offset = 0
while ok:
try:
bl = begin.index(l, offset)
if i != bl: # Cannot map to self
links.append((i, bl))
offset = bl + 1 # next possible edge
except ValueError: # no more possible edge for this last node, Next!
ok = False
# Great function shamelessly taken from stackoverflow (link provided above)
import networkx as nx
def longest_path(G):
dist = {} # stores [node, distance] pair
for node in nx.topological_sort(G):
# pairs of dist,node for all incoming edges
pairs = [(dist[v][0]+1,v) for v in G.pred[node]]
if pairs:
dist[node] = max(pairs)
else:
dist[node] = (0, node)
node,(length,_) = max(dist.items(), key=lambda x:x[1])
path = []
while length > 0:
path.append(node)
length,node = dist[node]
return list(reversed(path))
# Construct graph
G = nx.DiGraph()
G.add_edges_from(links)
# TADAAAA!
print(longest_path(G))
Although it looks nice, there is a big drawback. You example works because there is no cycle in the resulting graph of input words, however, this solution fails on cyclic graphs.
A way around that is to detect cycles and break them. Detection can be done this way:
if nx.recursive_simple_cycles(G):
print("CYCLES!!! /o\")
Breaking the cycle can be done by just dropping a random edge in the cycle and then you will randomly find the optimal solution for your problem (imagine a cycle with a tail, you should cut the cycle on the node having 3 edges), thus I suggest brute-forcing this part by trying all possible cycle breaks, computing longest path and taking the longest of the longest path. If you have multiple cycles it becomes a bit more explosive in number of possibilities... but hey it's NP-hard, at least the way I see it and I didn't plan to solve that now :)
Hope it helps
Here's a solution that doesn't require recursion. It uses the itertools permutation function to look at all possible orderings of the words, and find the one with the longest length. To save time, as soon as an ordering hits a word that doesn't work, it stops checking that ordering and moves on.
>>> g = 'girafarig eudino exeggcute omolga gabite'
... p = itertools.permutations(g.split())
... longestword = ""
... for words in p:
... thistry = words[0]
... # Concatenates words until the next word doesn't link with this one.
... for i in range(len(words) - 1):
... if words[i][-1] != words[i+1][0]:
... break
... thistry += words[i+1]
... i += 1
... if len(thistry) > len(longestword):
... longestword = thistry
... print(longestword)
... print("Final answer is {}".format(longestword))
girafarig
girafariggabiteeudino
girafariggabiteeudinoomolga
girafariggabiteexeggcuteeudinoomolga
Final answer is girafariggabiteexeggcuteeudinoomolga
First, let's see what the problem looks like:
from collections import defaultdict
import pydot
words = (
"audino bagon baltoy banette bidoof braviary bronzor carracosta "
"charmeleon cresselia croagunk darmanitan deino emboar emolga "
"exeggcute gabite girafarig gulpin haxorus"
).split()
def main():
# get first -> last letter transitions
nodes = set()
arcs = defaultdict(lambda: defaultdict(list))
for word in words:
first = word[0]
last = word[-1]
nodes.add(first)
nodes.add(last)
arcs[first][last].append(word)
# create a graph
graph = pydot.Dot("Word_combinations", graph_type="digraph")
# use letters as nodes
for node in sorted(nodes):
n = pydot.Node(node, shape="circle")
graph.add_node(n)
# use first-last as directed edges
for first, sub in arcs.items():
for last, wordlist in sub.items():
count = len(wordlist)
label = str(count) if count > 1 else ""
e = pydot.Edge(first, last, label=label)
graph.add_edge(e)
# save result
graph.write_jpg("g:/temp/wordgraph.png", prog="dot")
if __name__=="__main__":
main()
results in
which makes the solution fairly obvious (path shown in red), but only because the graph is acyclic (with the exception of two trivial self-loops).