I'm fairly new to python and I'm having an issue with my python script (split_fasta.py). Here is an example of my issue:
list = ["1.fasta", "2.fasta", "3.fasta"]
for file in list:
contents = open(file, "r")
for line in contents:
if line[0] == ">":
new_file = open(file + "_chromosome.fasta", "w")
new_file.write(line)
I've left the bottom part of the program out because it's not needed. My issue is that when I run this program in the same direcoty as my fasta123 files, it works great:
python split_fasta.py *.fasta
But if I'm in a different directory and I want the program to output the new files (eg. 1.fasta_chromsome.fasta) to my current directory...it doesn't:
python /home/bin/split_fasta.py /home/data/*.fasta
This still creates the new files in the same directory as the fasta files. The issue here I'm sure is with this line:
new_file = open(file + "_chromosome.fasta", "w")
Because if I change it to this:
new_file = open("seq" + "_chromosome.fasta", "w")
It creates an output file in my current directory.
I hope this makes sense to some of you and that I can get some suggestions.
You are giving the full path of the old file, plus a new name. So basically, if file == /home/data/something.fasta, the output file will be file + "_chromosome.fasta" which is /home/data/something.fasta_chromosome.fasta
If you use os.path.basename on file, you will get the name of the file (i.e. in my example, something.fasta)
From #Adam Smith
You can use os.path.splitext to get rid of the .fasta
basename, _ = os.path.splitext(os.path.basename(file))
Getting back to the code example, I saw many things not recommended in Python. I'll go in details.
Avoid shadowing builtin names, such as list, str, int... It is not explicit and can lead to potential issues later.
When opening a file for reading or writing, you should use the with syntax. This is highly recommended since it takes care to close the file.
with open(filename, "r") as f:
data = f.read()
with open(new_filename, "w") as f:
f.write(data)
If you have an empty line in your file, line[0] == ... will result in a IndexError exception. Use line.startswith(...) instead.
Final code :
files = ["1.fasta", "2.fasta", "3.fasta"]
for file in files:
with open(file, "r") as input:
for line in input:
if line.startswith(">"):
new_name = os.path.splitext(os.path.basename(file)) + "_chromosome.fasta"
with open(new_name, "w") as output:
output.write(line)
Often, people come at me and say "that's hugly". Not really :). The levels of indentation makes clear what is which context.
Related
All,
I'm trying to read text files that are being downloaded every 20 min in to a certain folder, check it, manipulate it and move it to another location for further processing. Basically, I want to check each file that comes in, check if a string contains a "0.00" value and if so, delete that particular string. There are two strings per file.
I managed to manipulate a file with a given name, but now I need to do the same for files with variable names (there is a timestamp included in the title). One file will need to be processed at a time.
This is what I got so far:
import os
path = r"C:\Users\r1.0"
dir = os.listdir(path)
def remove_line(line, stop):
return any([word in line for word in stop])
stop = ["0.00"]
for file in dir:
if file.lower().endswith('.txt'):
with open(file, "r") as f:
lines = f.readlines()
with open(file, "w") as f:
for line in lines:
if not remove_line(line, stop):
f.write(line)
What works are the def-function and the two "with open..." codes. What am I doing wrong here?
Also, can I write a file to another directory using the open() function?
Thanks in advance!
Your code looks mostly fine. I don't think your list comprehension method does remove the string though. You can write to a different folder with Open(). This should do the trick for you:
import os
path = r"C:\Users\r1.0"
dir = os.listdir(path)
stop = ["0.00"]
for file in dir:
if file.lower().endswith('.txt'):
with open(file, "r") as f:
lines = f.readlines()
# put the new file in a different location
newfile = os path.join("New", "directory", file)
with open(newfile, "w") as f:
for line in lines:
if stop in line: #check if we need to modify lime
#modify line here
#this will remove stop from the line
line.replace(stop, "")
# Regardless of whether the line has changed, we need to write it out.
f.write(line)
I am new to programming and got an issue with writing bytes. Here is what I wrote:
file = open('filePath/input.train', 'wb')
for i in range(len(myList)):
file.write(bytes((myList[i]),'UTF-8'));
If I print 'i' here, it is 629.
The '.train' suffix is required by the project. In order to check it, I read it and write to a txt file:
file = open('filePath/input.train', 'rb')
content = file.read()
testFile = open('filePath/test.txt', 'wb')
testFile.write(content)
Now, the problem is, len(list) = 629 while I got 591 lines in test.txt file. It brought me problems later.
Why did this happen and how should I solve it?
first, when you open and write a file, need remember close the file after the write.like this.
file = open('filePath/input.train', 'wb')
for i in range(len(myList)):
file.write(bytes((myList[i]),'UTF-8'));
file.close()
second, python code not must has ";"
third, file is python's keyword, so don't use file be your variable name. you can use f or my_file or anyone, but don't use python's keyword.
fourth, python has a iterator, use iterator is better than your for i in range(len(xxx)).
all of this, your code can look like this.
f = open('filePath/input.train', 'wb')
for line in myList:
f.write(bytes(line, 'UTF-8'))
f.close()
I am a newbie to python. I have a code in which I must write the contents again to my same file,but when I do it it clears my content.Please help to fix it.
How should I modify my code such that the contents will be written back on the same file?
My code:
import re
numbers = {}
with open('1.txt') as f,open('11.txt', 'w') as f1:
for line in f:
row = re.split(r'(\d+)', line.strip())
words = tuple(row[::2])
if words not in numbers:
numbers[words] = [int(n) for n in row[1::2]]
numbers[words] = [n+1 for n in numbers[words]]
row[1::2] = map(str, numbers[words])
indentation = (re.match(r"\s*", line).group())
print (indentation + ''.join(row))
f1.write(indentation + ''.join(row) + '\n')
In general, it's a bad idea to write over a file you're still processing (or change a data structure over which you are iterating). It can be done...but it requires much care, and there is little safety or restart-ability should something go wrong in the middle (an error, a power failure, etc.)
A better approach is to write a clean new file, then rename it to the old name. For example:
import re
import os
filename = '1.txt'
tempname = "temp{0}_{1}".format(os.getpid(), filename)
numbers = {}
with open(filename) as f, open(tempname, 'w') as f1:
# ... file processing as before
os.rename(tempname, filename)
Here I've dropped filenames (both original and temporary) into variables, so they can be easily referred to multiple times or changed. This also prepares for the moment when you hoist this code into a function (as part of a larger program), as opposed to making it the main line of your program.
You don't strictly need the temporary name to embed the process id, but it's a standard way of making sure the temp file is uniquely named (temp32939_1.txt vs temp_1.txt or tempfile.txt, say).
It may also be helpful to create backups of the files as they were before processing. In which case, before the os.rename(tempname, filename) you can drop in code to move the original data to a safer location or a backup name. E.g.:
backupname = filename + ".bak"
os.rename(filename, backupname)
os.rename(tempname, filename)
While beyond the scope of this question, if you used a read-process-overwrite strategy frequently, it would be possible to create a separate module that abstracted these file-handling details away from your processing code. Here is an example.
Use
open('11.txt', 'a')
To append to the file instead of w for writing (a new or overwriting a file).
If you want to read and modify file in one time use "r+' mode.
f=file('/path/to/file.txt', 'r+')
content=f.read()
content=content.replace('oldstring', 'newstring') #for example change some substring in whole file
f.seek(0) #move to beginning of file
f.write(content)
f.truncate() #clear file conent "tail" on disk if new content shorter then old
f.close()
I'm looking at how to do file input and output in Python. I've written the following code to read a list of names (one per line) from a file into another file while checking a name against the names in the file and appending text to the occurrences in the file. The code works. Could it be done better?
I'd wanted to use the with open(... statement for both input and output files but can't see how they could be in the same block meaning I'd need to store the names in a temporary location.
def filter(txt, oldfile, newfile):
'''\
Read a list of names from a file line by line into an output file.
If a line begins with a particular name, insert a string of text
after the name before appending the line to the output file.
'''
outfile = open(newfile, 'w')
with open(oldfile, 'r', encoding='utf-8') as infile:
for line in infile:
if line.startswith(txt):
line = line[0:len(txt)] + ' - Truly a great person!\n'
outfile.write(line)
outfile.close()
return # Do I gain anything by including this?
# input the name you want to check against
text = input('Please enter the name of a great person: ')
letsgo = filter(text,'Spanish', 'Spanish2')
Python allows putting multiple open() statements in a single with. You comma-separate them. Your code would then be:
def filter(txt, oldfile, newfile):
'''\
Read a list of names from a file line by line into an output file.
If a line begins with a particular name, insert a string of text
after the name before appending the line to the output file.
'''
with open(newfile, 'w') as outfile, open(oldfile, 'r', encoding='utf-8') as infile:
for line in infile:
if line.startswith(txt):
line = line[0:len(txt)] + ' - Truly a great person!\n'
outfile.write(line)
# input the name you want to check against
text = input('Please enter the name of a great person: ')
letsgo = filter(text,'Spanish', 'Spanish2')
And no, you don't gain anything by putting an explicit return at the end of your function. You can use return to exit early, but you had it at the end, and the function will exit without it. (Of course with functions that return a value, you use the return to specify the value to return.)
Using multiple open() items with with was not supported in Python 2.5 when the with statement was introduced, or in Python 2.6, but it is supported in Python 2.7 and Python 3.1 or newer.
http://docs.python.org/reference/compound_stmts.html#the-with-statement
http://docs.python.org/release/3.1/reference/compound_stmts.html#the-with-statement
If you are writing code that must run in Python 2.5, 2.6 or 3.0, nest the with statements as the other answers suggested or use contextlib.nested.
Use nested blocks like this,
with open(newfile, 'w') as outfile:
with open(oldfile, 'r', encoding='utf-8') as infile:
# your logic goes right here
You can nest your with blocks. Like this:
with open(newfile, 'w') as outfile:
with open(oldfile, 'r', encoding='utf-8') as infile:
for line in infile:
if line.startswith(txt):
line = line[0:len(txt)] + ' - Truly a great person!\n'
outfile.write(line)
This is better than your version because you guarantee that outfile will be closed even if your code encounters exceptions. Obviously you could do that with try/finally, but with is the right way to do this.
Or, as I have just learnt, you can have multiple context managers in a with statement as described by #steveha. That seems to me to be a better option than nesting.
And for your final minor question, the return serves no real purpose. I would remove it.
Sometimes, you might want to open a variable amount of files and treat each one the same, you can do this with contextlib
from contextlib import ExitStack
filenames = [file1.txt, file2.txt, file3.txt]
with open('outfile.txt', 'a') as outfile:
with ExitStack() as stack:
file_pointers = [stack.enter_context(open(file, 'r')) for file in filenames]
for fp in file_pointers:
outfile.write(fp.read())
I have a data file (unstructured, messy file) from which I have to scrub specific list of strings (delete strings).
Here is what I am doing but with no result:
infile = r"messy_data_file.txt"
outfile = r"cleaned_file.txt"
delete_list = ["firstname1 lastname1","firstname2 lastname2"....,"firstnamen lastnamen"]
fin=open(infile,"")
fout = open(outfile,"w+")
for line in fin:
for word in delete_list:
line = line.replace(word, "")
fout.write(line)
fin.close()
fout.close()
When I execute the file, I get the following error:
NameError: name 'word' is not defined
The readlines method returns a list of lines, not words, so your code would only work where one of your words is on a line by itself.
Since files are iterators over lines this can be done much easier:
infile = "messy_data_file.txt"
outfile = "cleaned_file.txt"
delete_list = ["word_1", "word_2", "word_n"]
with open(infile) as fin, open(outfile, "w+") as fout:
for line in fin:
for word in delete_list:
line = line.replace(word, "")
fout.write(line)
To remove the string within the same file, I used this code
f = open('./test.txt','r')
a = ['word1','word2','word3']
lst = []
for line in f:
for word in a:
if word in line:
line = line.replace(word,'')
lst.append(line)
f.close()
f = open('./test.txt','w')
for line in lst:
f.write(line)
f.close()
Based on your comment "I am double clicking the .py file. It seems to invoke the python application which disappears after a couple of seconds. I dont get any error thought" I believe your issue is the script is not finding the input file. That is also why you are not getting any output. When you double click on it... I actually can't recall where the interpreter is going to look but I think it's where the python.exe is installed.
Use a fully qualified path like so.
# Depends on your OS
infile = r"C:\tmp\messy_data_file.txt"
outfile = r"C:\tmp\cleaned_file.txt"
infile = r"/etc/tmp/messy_data_file.txt"
outfile = r"/etc/tmp/cleaned_file.txt"
Also, for your sanity, run it from the command-line instead of double clicking. It'll be much easier to catch errors/output.
To the OP,
Ross Patterson's method above works perfectly for me, i.e.
infile = "messy_data_file.txt"
outfile = "cleaned_file.txt"
delete_list = ["word_1", "word_2", "word_n"]
fin = open(infile)
fout = open(outfile, "w+")
for line in fin:
for word in delete_list:
line = line.replace(word, "")
fout.write(line)
fin.close()
fout.close()
Example:
I have a file named messy_data_file.txt that includes the following words (animals), not necessarily on the same line. Like this:
Goat
Elephant
Horse Donkey Giraffe
Lizard
Bird
Fish
When I modify the code to read (actually just adding the words to delete to the "delete_list" line):
infile = "messy_data_file.txt"
outfile = "cleaned_file.txt"
delete_list = ["Donkey", "Goat", "Fish"]
fin = open(infile)
fout = open(outfile, "w+")
for line in fin:
for word in delete_list:
line = line.replace(word, "")
fout.write(line)
fin.close()
fout.close()
The resulting "cleaned_file.txt" looks like this:
Elephant
Horse Giraffe
Lizard
Bird
There is a blank line where "Goat" used to be (where, oddly, removing "Donkey" did not) but for my purposes, this works fine.
I also add input("Press Enter to exit...") the the very end of the code to keep the command line window from opening and slamming shut on me when I'm double-clicking the remove_text.py file to run it, but take note that you'll catch no errors this way.
To do that I run it from the command line (where C:\Just_Testing is the directory where all my files are, i.e. remove_text.py and messy_text.txt)
like this:
C:\Just_Testing\>py remove_text.py
or
C:\Just_Testing>python remove_text.py
works exactly the same.
Of course, like when writing HTML, I guess it never hurts to use a fully qualified path when running py or python from somewhere other than the directory you happen to be sitting in, such as:
C:\Windows\System32\>python C:\Users\Me\Desktop\remove_text.py
Of course in the code it would be:
infile = "C:\Users\Me\Desktop\messy_data_file.txt"
outfile = "C:\Users\Me\Desktop\cleaned_file.txt"
Be careful to use the same fully qualified path to place your newly created cleaned_file.txt in or it will be created wherever you may be and that could cause confusion when looking for it.
Personally, I have the PATH in my Environment Variables set to point to all my Python installs i.e. C:\Python3.5.3, C:\Python2.7.13, etc. so I can run py or python from anywhere.
Anyway, I hope making fine-tuning adjustments to this code from Mr. Patterson can get you exactly what you need. :)
.
Maybe you can add encoding='utf-8' in your fin and fout variables.
Here is the modified one you may want to use:
fin=open(infile,"", encoding='utf-8')
fout = open(outfile,"w+", encoding='utf-8')
This(adding utf-8) mostly occurs on the OS Windows. Also for reading, writing, and appending the file, this usually isn't a problem but for advanced things to do a file like replacing text in there, etc then you should do this.
Hope this helps you.
The code below just gets the old data and checks if the string doesnt contain the string you doesnt want then continues. (this also works if you want to remove empty lines)
str = []
with open("file.txt", "r+") as f:
for i in f.readlines():
str.append(i)
with open("file.txt", "w") as f:
for i in str:
if i != "The string you want to remove":
f.write(i)