Develop Reference

MapReduce Vowel Cont

I am looking to change my MapReduce code that finds words in a text with the same vowels. For example:
hEllo’ and ‘pOle’ both contain exactly 1 e and exactly 1 o. The order of the vowels and the case from the original input word does not matter.
Imagine the following example:
hEllo moose
pOle cccttt.ggg
We would end up with the following output:
:1
eo:2
eoo:1
The map code that I have so far is:
import sys
import re
line = sys.stdin.readline()
pattern = re.compile("[a,e,i,o,u]+")
while line:
for char in pattern.findall(line):
print(char+"\t"+"1")
line = sys.stdin.readline()
and the reducer code:
import sys
current_word = None
current_count = 0
word = None
for line in sys.stdin:
line = line.strip()
word, count = line.split('\t', 1)
count = int(count)
if current_word == word:
current_count += count
else:
if current_word:
print('%s\t%s' % (current_word, current_count))
current_count = count
current_word = word
if current_word == word:
print('%s\t%s' % (current_word, current_count))
When I run this MapReduce code in Hadoop I get the following output:
a 1
e 4
o 1

Standard version
Not sure if there's a way using regex (there probably is), but here's a non-regex approach that at least gets you the desired result. This approach could likely be improved upon, right now it's a sort of rough draft that seems to work for the case outlined in the original question. Let me know if I need to clarify anything in particular.
from collections import defaultdict
def vowel_count_map(sentence: str):
"""
Return a map of vowel sequence to frequency per word in sentence.
"""
lowercased = sentence.lower()
count = defaultdict(int)
for word in lowercased.split(' '):
vowel_seq = _count_vowels(word)
count[vowel_seq] += 1
return count
def _count_vowels(word, vowels='aeiou') -> str:
"""Return an alphabetized sequence of vowels found in word."""
count = defaultdict(int) # init counter
for char in word:
# Technically this `if` condition is not needed, and can be
# omitted (not sure how it would affect performance?)
if char in vowels:
count[char] += 1
# For example, this uses the same logic as below:
# 'a' * 3 = 'aaa'
return ''.join([vowel * count[vowel] for vowel in vowels])
Shorter version
Here's a one-liner version of the same function that I think is pretty cool, at the cost of being a little harder to understand:
from collections import Counter
vowel_count_map = lambda sentence: Counter([''.join([v * word.count(v) for v in 'aeiou']) for word in sentence.lower().split(' ')])
Usage
I put together some sample strings we can use for test data, and pass as inputs to either version of the vowel_count_map function above. The first approach above returns a defaultdict, the second one returns a Counter. They are essentially both dict-like objects, so you can iterate over their key-value pairs as usual.
a_string = "hEllo moose pOle cccttt.ggg"
b_string = "testuueaaxyzioabceezu actionable conciliatory"
print(vowel_count_map(a_string))
# defaultdict(<class 'int'>, {'eo': 2, 'eoo': 1, '': 1})
print(vowel_count_map(b_string))
# defaultdict(<class 'int'>, {'aaaeeeeiouuu': 1, 'aaeio': 1, 'aiioo': 1})

How to extract words from repeating strings

Here I have a string in a list:
['aaaaaaappppppprrrrrriiiiiilll']
I want to get the word 'april' in the list, but not just one of them, instead how many times the word 'april' actually occurs the string.
The output should be something like:
['aprilaprilapril']
Because the word 'april' occurred three times in that string.
Well the word actually didn't occurred three times, all the characters did. So I want to order these characters to 'april' for how many times did they appeared in the string.
My idea is basically to extract words from some random strings, but not just extracting the word, instead to extract all of the word that appears in the string. Each word should be extracted and the word (characters) should be ordered the way I wanted to.
But here I have some annoying conditions; you can't delete all the elements in the list and then just replace them with the word 'april'(you can't replace the whole string with the word 'april'); you can only extract 'april' from the string, not replacing them. You can't also delete the list with the string. Just think of all the string there being very important data, we just want some data, but these data must be ordered, and we need to delete all other data that doesn't match our "data chain" (the word 'april'). But once you delete the whole string you will lose all the important data. You don't know how to make another one of these "data chains", so we can't just put the word 'april' back in the list.
If anyone know how to solve my weird problem, please help me out, I am a beginner python programmer. Thank you!

One way is to use itertools.groupby which will group the characters individually and unpack and iterate them using zip which will iterate n times given n is the number of characters in the smallest group (i.e. the group having lowest number of characters)
from itertools import groupby
'aaaaaaappppppprrrrrriiiiiilll'
result = ''
for each in zip(*[list(g) for k, g in groupby('aaaaaaappppppprrrrrriiiiiilll')]):
result += ''.join(each)
# result = 'aprilaprilapril'
Another possible solution is to create a custom counter that will count each unique sequence of characters (Please be noted that this method will work only for Python 3.6+, for lower version of Python, order of dictionaries is not guaranteed):
def getCounts(strng):
if not strng:
return [], 0
counts = {}
current = strng[0]
for c in strng:
if c in counts.keys():
if current==c:
counts[c] += 1
else:
current = c
counts[c] = 1
return counts.keys(), min(counts.values())
result = ''
counts=getCounts('aaaaaaappppppprrrrrriiiiiilll')
for i in range(counts[1]):
result += ''.join(counts[0])
# result = 'aprilaprilapril'

How about using regex?
import re
word = 'april'
text = 'aaaaaaappppppprrrrrriiiiiilll'
regex = "".join(f"({c}+)" for c in word)
match = re.match(regex, text)
if match:
# Find the lowest amount of character repeats
lowest_amount = min(len(g) for g in match.groups())
print(word * lowest_amount)
else:
print("no match")
Outputs:
aprilaprilapril
Works like a charm

Here is a more native approach, with plain iteration.
It has a time complexity of O(n).
It uses an outer loop to iterate over the character in the search key, then an inner while loop that consumes all occurrences of that character in the search string while maintaining a counter. Once all consecutive occurrences of the current letter have been consumes, it updates a the minLetterCount to be the minimum of its previous value or this new count. Once we have iterated over all letters in the key, we return this accumulated minimum.
def countCompleteSequenceOccurences(searchString, key):
left = 0
minLetterCount = 0
letterCount = 0
for i, searchChar in enumerate(key):
while left < len(searchString) and searchString[left] == searchChar:
letterCount += 1
left += 1
minLetterCount = letterCount if i == 0 else min(minLetterCount, letterCount)
letterCount = 0
return minLetterCount
Testing:
testCasesToOracles = {
"aaaaaaappppppprrrrrriiiiiilll": 3,
"ppppppprrrrrriiiiiilll": 0,
"aaaaaaappppppprrrrrriiiiii": 0,
"aaaaaaapppppppzzzrrrrrriiiiiilll": 0,
"pppppppaaaaaaarrrrrriiiiiilll": 0,
"zaaaaaaappppppprrrrrriiiiiilll": 3,
"zzzaaaaaaappppppprrrrrriiiiiilll": 3,
"aaaaaaappppppprrrrrriiiiiilllzzz": 3,
"zzzaaaaaaappppppprrrrrriiiiiilllzzz": 3,
}
key = "april"
for case, oracle in testCasesToOracles.items():
result = countCompleteSequenceOccurences(case, key)
assert result == oracle
Usage:
key = "april"
result = countCompleteSequenceOccurences("aaaaaaappppppprrrrrriiiiiilll", key)
print(result * key)
Output:
aprilaprilapril

A word will only occur as many times as the minimum letter recurrence. To account for the possibility of having repeated letters in the word (for example, appril, you need to factor this count out. Here is one way of doing this using collections.Counter:
from collections import Counter
def count_recurrence(kernel, string):
# we need to count both strings
kernel_counter = Counter(kernel)
string_counter = Counter(string)
# now get effective count by dividing the occurence in string by occurrence
# in kernel
effective_counter = {
k: int(string_counter.get(k, 0)/v)
for k, v in kernel_counter.items()
}
# min occurence of kernel is min of effective counter
min_recurring_count = min(effective_counter.values())
return kernel * min_recurring_count

finding the word with most repeated letters from a string containing a sentence in python

I want to find a word with the most repeated letters given an input a sentence.
I know how to find the most repeated letters given the sentence but I'm not able how to print the word.
For example:
this is an elementary test example
should print
elementary
def most_repeating_word(strg):
words =strg.split()
for words1 in words:
dict1 = {}
max_repeat_count = 0
for letter in words1:
if letter not in dict1:
dict1[letter] = 1
else:
dict1[letter] += 1
if dict1[letter]> max_repeat_count:
max_repeat_count = dict1[letter]
most_repeated_char = letter
result=words1
return result

You are resetting the most_repeat_count variable for each word to 0. You should move that upper in you code, above first for loop, like this:
def most_repeating_word(strg):
words =strg.split()
max_repeat_count = 0
for words1 in words:
dict1 = {}
for letter in words1:
if letter not in dict1:
dict1[letter] = 1
else:
dict1[letter] += 1
if dict1[letter]> max_repeat_count:
max_repeat_count = dict1[letter]
most_repeated_char = letter
result=words1
return result
Hope this helps

Use a regex instead. It is simple and easy. Iteration is an expensive operation compared to regular expressions.
Please refer to the solution for your problem in this post:
Count repeated letters in a string

Interesting exercise! +1 for using Counter(). Here's my suggestion also making use of max() and its key argument, and the * unpacking operator.
For a final solution note that this (and the other proposed solutions to the question) don't currently consider case, other possible characters (digits, symbols etc) or whether more than one word will have the maximum letter count, or if a word will have more than one letter with the maximum letter count.
from collections import Counter
def most_repeating_word(strg):
# Create list of word tuples: (word, max_letter, max_count)
counters = [ (word, *max(Counter(word).items(), key=lambda item: item[1]))
for word in strg.split() ]
max_word, max_letter, max_count = max(counters, key=lambda item: item[2])
return max_word

word="SBDDUKRWZHUYLRVLIPVVFYFKMSVLVEQTHRUOFHPOALGXCNLXXGUQHQVXMRGVQTBEYVEGMFD"
def most_repeating_word(strg):
dict={}
max_repeat_count = 0
for word in strg:
if word not in dict:
dict[word] = 1
else:
dict[word] += 1
if dict[word]> max_repeat_count:
max_repeat_count = dict[word]
result={}
for word, value in dict.items():
if value==max_repeat_count:
result[word]=value
return result
print(most_repeating_word(word))

Python 3 counter that is ignoring strings with less than x characters

i have a programm that counts words of a text file. Now i want to restrict the counter to strings with more than x characters
from collections import Counter
input = 'C:/Users/micha/Dropbox/IPCC_Boox/FOD_v1_ch15.txt'
Counter = {}
words = {}
with open(input,'r', encoding='utf-8-sig') as fh:
for line in fh:
word_list = line.replace(',','').replace('\'','').replace('.','').lower().split()
for word in word_list:
if word not in Counter:
Counter[word] = 1
else:
Counter[word] = Counter[word] + 1
N = 20
top_words = Counter(Counter).most_common(N)
for word, frequency in top_words:
print("%s %d" % (word, frequency))
I tried the re code, but it did not work.
re.sub(r'\b\w{1,3}\b')
I dont know how to implement it...
At the end I would like to have an output that ignores all the short words like and, you, be etc.

You could do this more simply with:
for word in word_list:
if len(word) < 5: # check the length of each word is less than 5 for example
continue # this skips the counter portion and jumps to next word in word_list
elif word not in Counter:
Counter[word] = 1
else:
Counter[word] = Counter[word] + 1

Few notes.
1) You import a Counter but don't use it properly (you do a Counter = {} thus overwriting the import).
from collections import Counter
2) Instead of doing several replaces use list comprehension with a set, its faster and only does one (two with the join) iterations instead of several:
sentence = ''.join([char for char in line if char not in {'.', ',', "'"}])
word_list = sentence.split()
3) Use the counter and list comp for length:
c = Counter(word for word in word_list if len(word) > 3)
Thats it.

Counter already does what you want. You can "feed" it wiht an iterable and this will work.
https://docs.python.org/2/library/collections.html#counter-objects
You can use the filter function too https://docs.python.org/3.7/library/functions.html#filter
The could look alike:
counted = Counter(filter(lambda x: len(x) >= 5, words))

How do you count a negative or positive word prior to a specific word - Sentiment Analysis in Python?

I'm trying to count how many times a negative word from a list appears before a specific word. For example, "This terrible laptop." The specified word being "laptop", I want the output to have "Terrible 1" in Python.
def run(path):
negWords={} #dictionary to return the count
#load the negative lexicon
negLex=loadLexicon('negative-words.txt')
fin=open(path)
for line in fin: #for every line in the file (1 review per line)
line=line.lower().strip().split(' ')
review_set=set() #Adding all the words in the review to a set
for word in line: #Check if the word is present in the line
review_set.add(word) #As it is a set, only adds one time
for word in review_set:
if word in negLex:
if word in negWords:
negWords[word]=negWords[word]+1
else:
negWords[word] = 1
fin.close()
return negWords
if __name__ == "__main__":
print(run('textfile'))

This should do what you're looking for, it uses set & intersection to avoid some of the looping. The steps are —
get the negative words in the line
check the location of each word
if the word after that location is 'laptop' record it
Note that this will only identify the first occurrence of a negative word in a line, so "terrible terrible laptop" will not be a match.
from collections import defaultdict
def run(path):
negWords=defaultdict(int) # A defaultdict(int) will start at 0, can just add.
#load the negative lexicon
negLex=loadLexicon('negative-words.txt')
# ?? Is the above a list or a set, if it's a list convert to set
negLex = set(negLex)
fin=open(path)
for line in fin: #for every line in the file (1 review per line)
line=line.lower().strip().split(' ')
# Can just pass a list to set to make a set of it's items.
review_set = set(line)
# Compare the review set against the neglex set. We want words that are in
# *both* sets, so we can use intersection.
neg_words_used = review_set & negLex
# Is the bad word followed by the word laptop?
for word in neg_words_used:
# Find the word in the line list
ix = line.index(word)
if ix > len(line) - 2:
# Can't have laptop after it, it's the last word.
continue
# The word after this index in the line is laptop.
if line[ix+1] == 'laptop':
negWords[word] += 1
fin.close()
return negWords
If you're only interested in words preceding the word 'laptop', a far more sensible approach would be to look for the word 'laptop', then check the word prior to that to see if it is a negative word. The following example does that.
find laptop in the current line
if laptop isn't in the line, or is the first word, skip the line
get the word before laptop, check against the negative words
if you have a match add it to our result
This avoids doing lookups for words which are not related to laptops.
from collections import defaultdict
def run(path):
negWords=defaultdict(int) # A defaultdict(int) will start at 0, can just add.
#load the negative lexicon
negLex=loadLexicon('negative-words.txt')
# ?? Is the above a list or a set, if it's a list convert to set
negLex = set(negLex)
fin=open(path)
for line in fin: #for every line in the file (1 review per line)
line=line.lower().strip().split(' ')
try:
ix = line.index('laptop')
except ValueError:
# If we dont' find laptop, continue to next line.
continue
if ix == 0:
# Laptop is the first word of the line, can't check prior word.
continue
previous_word = line[ix-1]
if previous_word in negLex:
# Negative word before the current one.
negWords[previous_word] += 1
fin.close()
return negWords

It looks like you want to check a function against consecutive words, here is one way to do it, condition will be checked against every consecutive words.
text = 'Do you like bananas? Not only do I like bananas, I love bananas!'
trigger_words = {'bananas'}
positive_words = {'like', 'love'}
def condition(w):
return w[0] in positive_words and w[1] in trigger_words
for c in '.,?!':
text = text.replace(c, '')
words = text.lower().split()
matches = filter(condition, zip(words, words[1:]))
n_positives = 0
for w1, w2 in matches:
print(f'{w1.upper()} {w2} => That\'s positive !')
n_positives += 1
print(f'This text had a score of {n_positives}')
Output:
LIKE bananas => That's positive !
LIKE bananas => That's positive !
LOVE bananas => That's positive !
3
Bonus:
You can search for 3 consecutive words by just changing zip(w, w[1:]) to zip(w, w[1:], w[2:]) with a condition that checks for 3 words.
You can get a counter dictionary by doing this:
from collections import Counter
counter = Counter((i[0] for i in matches)) # counter = {'like': 2, 'love': 1}