The output is unsorted, and sorting on the second column is not possible. Is there special method to sort on the second value.
This program takes a text and counts how many times a word is in a text
import string
with open("romeo.txt") as file: # opens the file with text
lst = []
d = dict ()
uniquewords = open('romeo_unique.txt', 'w')
for line in file:
words = line.split()
for word in words: # loops through all words
word = word.translate(str.maketrans('', '', string.punctuation)).upper() #removes the punctuations
if word not in d:
d[word] =1
else:
d[word] = d[word] +1
if word not in lst:
lst.append(word) # append only this unique word to the list
uniquewords.write(str(word) + '\n') # write the unique word to the file
print(d)
Dictionaries with default value
The code snippet:
d = dict()
...
if word not in d:
d[word] =1
else:
d[word] = d[word] +1
has become so common in python that a subclass of dict has been created to get rid of it. It goes by the name defaultdict and can be found in module collections.
Thus we can simplify your code snippet to:
from collections import defaultdict
d = defaultdict(int)
...
d[word] = d[word] + 1
No need for this manual if/else test; if word is not in the defaultdict, it will be added automatically with initial value 0.
Counters
Counting occurrences is also something that is frequently useful; so much so that there exists a subclass of dict called Counter in module collections. It will do all the hard work for you.
from collections import Counter
import string
with open('romeo.txt') as input_file:
counts = Counter(word.translate(str.maketrans('', '', string.punctuation)).upper() for line in input_file for word in line.split())
with open('romeo_unique.txt', 'w') as output_file:
for word in counts:
output_file.write(word + '\n')
As far as I can tell from the documentation, Counters are not guaranteed to be ordered by number of occurrences by default; however:
When I use them in the interactive python interpreter they are always printed in decreasing number of occurrences;
they provide a method .most_common() which is guaranteed to return in decreasing number of occurrences.
In Python, standard dictionaries are an unsorted data type, but you can look here, assuming that with sorting your output you mean d
A couple of remarks first:
You are not sorting explicitly (e.g. by using sorted) by a given property. Dictionaries might be considered to have a "natural" order by the alphanumeric value of the value part of each key-value pair and they might sort correctly when iterated (e.g. for printing), but it is better to explicitly sort a dict.
You check the existence of a word in the lst variable, which is very slow since checking a list requires checking all entries until something is found (or not). It would be much better to check for existence in a dict.
I'm assuming by "the second column" you mean the information for each word that counts the order in which the word first appeared.
With that I'd change the code to also record the word index of the first occurence of each word with, which then allows for sorting on exactly that.
Edit: Fixed the code. The sorting yielded by sorted sorts by key, not value. That's what I get for not testing code before posting an answer.
import string
from operator import itemgetter
with open("romeo.txt") as file: # opens the file with text
first_occurence = {}
uniqueness = {}
word_index = 1
uniquewords = open('romeo_unique.txt', 'w')
for line in file:
words = line.split()
for word in words: # loops through all words
word = word.translate(str.maketrans('', '', string.punctuation)).upper() #removes the punctuations
if word not in uniqueness:
uniqueness[word] = 1
else:
uniqueness[word] += 1
if word not in first_occurence:
first_occurence[word] = word_index
uniquewords.write(str(word) + '\n') # write the unique word to the file
word_index += 1
print(sorted(uniqueness.items(), key=itemgetter(1)))
print(sorted(first_occurence.items(), key=itemgetter(1)))
def doc_read_alpha():
with open('input.txt', 'r') as file:
for line in file:
f_contents = file.read()
lines = line.split()
lines = sorted(lines)
The above is an algorithm used to iterate through my file contents and separate each word into an element of an array 'lines'. I am having trouble adding a filter function which would remove duplicates from my array.
input_file_string = " ".join(lines)
return lines
def main():
print(doc_read_alpha())
if __name__ == '__main__':
main()
If I understand you correctly, you want to have a list of unique words/tokens extracted from a text. You can achieve that through a "set" instead of a list, which behaves almost identically as a list, but does not allow for duplicate entries.
It is as simple as:
lines = set(line.split())
If you want to avoid duplicates in your list. A better solution would be to use a set. Each element in the set is unique and does not repeat.
you can convert your list into a set using
s = set(lines)
However, sets are unordered and while it makes it faster to check if a value belongs in the set. It means you cannot find an element using an index
def doc_read_alpha():
with open('text.txt', 'r') as file:
s = set()
for line in file.readlines():
s.update(line.split())
return s
I dont know if this is the best solution but it works. However this method does count "word" and "word." as two different words. To avoid this you have do strip all non letter characters.
I have to open a file read it line by line. For each line, I have to split the line into a list of words using the split() method. The program should build a list of words. For each word on each line check to see if the word is already on the list and if not append it to the list. When the program completes, print the resulting words in alphabetical order.
fname = input("Enter file name: ")
fh = open(fname)
lst=list()
for line in fh:
for each in line:
word = line.split()
if word not in lst:
lst.append(word)
print(lst)
after this, I am getting 4 different lists but I am required to get a single list and I am not able to get that
one line, create a set in a set comprehension using split on each line, and sort the set into a list, using ,key=str.casefold to sort case-insensitive/locale wise.
with open(fname) as f:
result = sorted({word for line in f for word in line.split()},key=str.casefold)
this is particularly efficient since you don't have to use in in your existing list, which performs a linear search, very slow if the list is big.
If the file contains punctuation, that won't work very well because split won't remove it. Use a regex in that case:
result = sorted({word for line in f for word in re.split("\W+",line) if word},key = str.casefold)
(you have to add an extra non-empty filter)
Try this
fname = input("Enter file name: ")
fh = open(fname)
lst=list()
for line in fh:
for each in line:
word = line.split()
for wrd in word:
if wrd not in lst:
lst.append(wrd)
print(lst)
There are several problems in the code
First, it's unclear what you want to do with for each in line. I simply removed it.
Second, with word = line.split(), you get word as a list of words. You need to iterate through the list of words and perform actions on individual words.
Then, use sorted() to sort the words.
The refined code looks like this:
fname = input("Enter file name: ")
fh = open(fname)
lst=list()
for line in fh:
words = line.split()
for word in words
if word not in lst:
lst.append(word)
fh.close()
print(sorted(lst))
Side note: You're not closing the file. Use fh.close() (as I added above) or use with open(fname) as fh: which will close the file for you after leaving the with block.
Same style as the answers before but rather than using a for loop to get all of the lines I use the readlines() functions
The line lines=fh.readlines() will return a full list of the lines.
Then you must look at each line with for line in lines to see each line. Then you look at each word in the line with for word in words.
fname = input("Enter file name: ")
fh = open(fname)
lst=list()
lines=fh.readlines()
for line in lines:
words = line.split()
for word in words:
if word not in lst:
lst.append(word)
You can do this like this:
with open(fname) as fh:
unique_words = set(f.read().split())
To move that set into list use:
unique_words = list(unique_words)
and to sort that list:
unique_words.sort()
You should be able to adapt this idea to your problem.
words.txt
In linguistics, a word is the smallest element
that can be uttered in isolation with objective
or practical meaning. This contrasts deeply with
a morpheme, which is the smallest unit of meaning
but will not necessarily stand on its own.
import re
with open('words.txt', 'r', encoding='utf8') as f:
words = f.read()
all_words = re.findall(r'\w+', words)
result = sorted(set(all_words), key=str.lower)
print(result)
['a', 'be', 'but', 'can', 'contrasts', 'deeply', 'element', 'in', 'In',
'is', 'isolation', 'its', 'linguistics', 'meaning', 'morpheme',
'necessarily', 'not', 'objective', 'of', 'on', 'or', 'own', 'practical',
'smallest', 'stand', 'that', 'the', 'This', 'unit', 'uttered', 'which',
'will', 'with', 'word']
Is this a interview question by chanse? I am asking because of this part:
For each word on each line check to see if the word is already in the list
If this is a interview question, it could very well be that your interviewer does not necessarily want you to use language features to solve this but rather implement a searching algorithm or structure, for example a BST.
Assuming this is not the case however, let's go over your code. First, I would recommend you switch from opening the file like you do, fh = open(fname) and switch to a context, using with. Or, at least, close the file handler.
word_dictionary = dict()
with open(file_name) as source:
for line in source:
for word in line.split():
if word_dictionary.get(word) is None:
word_dictionary[word] = True
word_list = [word for word, _ in word_dictionary.items()]
sorted(word_list)
print word_list
Let's go over the code together.
First, we define a dictionary, called word_dictionary. A dictionary or hash table is a data structure that allows you to make lookup operation in constant, O(1), time. That is to say, very fast.
Second, we open the file containing the text. This is with open(file_name) as source: We call this a context. It is a convenient way of dealing with files ( and not only ) that automatically takes care of resource management. I won't go into detail, but i will recommend this article.
We begin by reading each line, and for each line, we read each word.
for line in source:
for word in line.split():
For each word, we check to see if we have already encountered it. We do this by using the .get() method of dictionaries. This method will check to see if the argument exists as a key in the dictionary. If it is, it will return the value associated with the key. Otherwise, it will return None.
if word_dictionary.get(word) is None:
word_dictionary[word] = True
This says that if we encounter a word that we have not seen already, then we register seeing it. Note that it's not necessary to use True as a value. Anything different to None will work.
One we have seen every word in the text, we do this.
word_list = [word for word, _ in word_dictionary.items()]
Using .items() we iterate over the key, value pairs of the dictionary. That is to say, if we had a dictionary d = {0: "a", 1: "b", 2: "c"}, calling for key, value in d.items() will yield key = 0, value = "a" at first, key = 1, value = "b" second and finally key = 2, value = "c"
In our case, we are not interested in the value. That is why we use _. We are only interested in the word.
What the list comprehension will leave us with is a list of all the words you encountered, in a random order. This is because dictionaries make no guarantee on the order to the key, value pairs.
Therefore, we need to sort.
sorted(word_list)
Since the word_list variable is going to be a list of string and since strings are comparable by default, they will be sorted.
Now, there are several things to consider. First, .split() will consider 'this' and 'this!' to be different words. You may or may not want this.
If you do not want this, you can use the 'string' module to check against punctuation or you can use a regex to clean up your word.
The second thing to consider is capitalisation. You make no mention of this. Are you allowed to lower or capitalise your text? If you are, your life will be easier. You lowercase everything and your problems go away.
If you are not allowed to lower your text, you will have to change the sort call. this is because capital letters are "smaller" then lowercase ones.
>>> "A" < "a"
True
This will manifest in the following way.
>>> sorted(["b", "a", "C"])
['C', 'a', 'b']
Most likely you expect ["a", "b", "C"] here. In this case, I recommend you look into the cpm argument for sort.
How can I convert file's words to dictionary as it's keys, directly, without using the list? Here is my code:
def abc(f):
with open(f, "r")as f:
dict = {}
lst = []
for line in f:
lst += line.strip().split()
for item in lst:
dict[item] = ""
return dict
print(abc("file.txt"))
Example of input "file.txt":
abc def
ghi jkl mno
pqr
Output:
{"abc":"", "def":"", "ghi":"", "jkl":"", "mno":"", "pqr":""}
The output of split() is a list. So, by use of that I need to read the data from file, store it on a list and then define it as dictionary's keys. My question is how can we ignore the list, and after reading data from file, put them directly to the dictionary?
if you really want a dictionary, build it in one go using a dict comprehension on the splitted words:
def abc(f):
return {word:"" for line in f for word in line.split()}
but you probably want a set instead since there are no values to your dict:
def abc(f):
return {word for line in f for word in line.split()}
I suspect you want to count the words, in that case:
def abc(f):
return collections.Counter(word for line in f for word in line.split())
note that split doesn't split on punctuation, so if the text contains some, you'll have duplicate words unless you replace
for word in line.split()
by
for word in re.split("\W",line) if word
(using re package, which has the slight drawback to generate empty fields at the start / end which is easily fixed by filtering word)
I don't know what you're trying to do nor what the expected output really is, but the following is functionnaly equivalent to your code snippet (and a bit cleaner too)
def abc(f):
dic = {}
with open(f, "r")as f:
for line in f:
# using split() will already remove
# trainling whitespaces so you don't
# need strip() here
for item in line.split():
dic[item] = ""
return dic
I'm using python to convert the words in sentences in a text file to individual tokens in a list for the purpose of counting up word frequencies. I'm having trouble converting the different sentences into a single list. Here's what I do:
f = open('music.txt', 'r')
sent = [word.lower().split() for word in f]
That gives me the following list:
[['party', 'rock', 'is', 'in', 'the', 'house', 'tonight'],
['everybody', 'just', 'have', 'a', 'good', 'time'],...]
Since the sentences in the file were in separate lines, it returns this list of lists and defaultdict can't identify the individual tokens to count up.
It tried the following list comprehension to isolate the tokens in the different lists and return them to a single list, but it returns an empty list instead:
sent2 = [[w for w in word] for word in sent]
Is there a way to do this using list comprehensions? Or perhaps another easier way?
Just use a nested loop inside the list comprehension:
sent = [word for line in f for word in line.lower().split()]
There are some alternatives to this approach, for example using itertools.chain.from_iterable(), but I think the nested loop is much easier in this case.
Just read the entire file to memory,a s a single string, and apply splitonce tot hat string.
There is no need to read the file line by line in such a case.
Therefore your core can be as short as:
sent = open("music.txt").read().split()
(A few niceties like closing the file, checking for errors, turn the code a little larger, of course)
Since you want to be counting word frequencies, you can use the collections.Counter class for that:
from collections import Counter
counter = Counter()
for word in open("music.txt").read().split():
counter[word] += 1
List comprehensions can do the job but will accumulate everything in memory. For large inputs this could be an unacceptable cost. The below solution will not accumulate large amounts of data in memory, even for large files. The final product is a dictionary of the form {token: occurrences}.
import itertools
def distinct_tokens(filename):
tokendict = {}
f = open(filename, 'r')
tokens = itertools.imap(lambda L: iter(L.lower.split()), f)
for tok in itertools.chain.from_iterable(tokens):
if tok in tokendict:
tokendict[tok] += 1
else:
tokendict[tok] = 1
f.close()
return tokendict