I have a noob python question... so bear with me.
Can I open multiple files before closing the previos.
So... can I run
import os
files=[]
for file in os.listdir(os.curdir):
files.append(open(file,'w'))
Then edit each file as I want and finish with
for file in files:
file.close()
Thanks in advance
Seems legit and works fine.
Doing operations would be hard for you this way. the list "files" doesn't contain the filenames. You would not know which file is what.
It is perfectly fine to open each file using open and later close all of them. However, you will want to make sure all of your files are closed properly.
Normally, you would do this for one file:
with open(filename,'w') as f:
do_something_with_the_file(f)
# the file is closed here, regardless of what happens
# i.e. even in case of an exception
You could do the same with multiple files:
with open(filename1,'w') as f1, open(filename2,'w') as f2:
do_something_with_the_file(f)
# both files are closed here
Now, if you have N files, you could write your own context manager, but that would probably be an overkill. Instead, I would suggest:
open_files = []
try:
for filename in list_of_filenames:
open_files.append(open(filename, 'w'))
# do something with the files here
finally:
for file in open_files:
file.close()
BTW, your own code deltes the contents of all files in the current directory. I am not sure you wanted that:
for file in os.listdir(os.curdir):
files.append(open(file,'w')) # open(file,'w') empties the file!!!
Maybe you wanted open(file, 'r') or open(file, 'a') instead? (see https://docs.python.org/2/library/functions.html#open)
Your solution will certainly work but the recommended way would be to use contextmanager so that the files gets handled seamlessly. For example
for filename in os.listdir(os.curdir):
with open(filename, 'w') as f:
# do some actions on the file
The with statement will take care of closing the file for you.
Related
At the moment I have a piece of code that writes the output from previous lines into a .html file in the same location as where the .py file is at....
I am looking for a line of code where it will simply write into the folder ./output/ of the same location the .py file is located
Below is the code I have but cant seem get it to work...
fullhtml = '<html><body><pre><h1>%s</h1><h2>#KCBB</h2><table>%s</table></pre></body></html>'%(end_date,myhtml)
with open('%s KCBB Bands.html'%end_date, 'w') as f:
f.write(fullhtml)
it usually writes it to that location, however, you can use __file__ to refer to that location.
__file__ will give you the python file root.
import os
print(os.path.dirname(os.path.realpath(__file__)))
As mentioned, the filename of your script is in __file__ (unless you are running in an IDE like jupyter). You can just use it with functions in the os module to get that path you want.
import os
fullhtml = '<html><body><pre><h1>%s</h1><h2>#KCBB</h2><table>%s</table></pre></body></html>'%(end_date,myhtml)
filename = os.path.join(os.path.dirname(__file__), "output",
'%s KCBB Bands.html' % end_date)
with open(filename, 'w') as f:
f.write(fullhtml)
Nevermind, i managed to figure it out....
Replacing
with open('%s KCBB Bands.html'%end_date, 'w') as f:
With
with open('./output/%s - KCBB Signal Output.html'%end_date, 'w') as f:
Notice the ./output/ < thats the directory i wanted the files to go into.
I'm new here,too. I think you're trying to keep your output files in a subdirectory of the working directory where the .py files are, is that right? If so, I think you just need to include the folder name in the open statement like so:
with open('output/%s KCBB Bands.html'%end_date, 'w') as f:
f.write(fullhtml)
The folder must already exist though, or you'll get an exception.
I have a Python script that opens a lot (over 2 million) small text files in a for loop. However, it stops when I reach approximately 150'000 files, which indicates for me that I reached the default limit of open files in the Linux kernel.
But, I'm closing the files, so I'm not sure why I hit that limit. The interesting part breaks down to that:
import os
files = os.listdir('/var/tmp/files')
for file in files:
fd = open('/var/tmp/files/{}'.format(file), 'r')
content = fd.readlines()
# Doing stuff
fd.close()
The code works, but apparently it doesn't close files. At first i tried the better with open() statement, but that didn't work either.
Why doesn't Python close the files?
Thanks guys. The problem was that my user had no access to one specific file. So, Python did everything as it should.
I expected that it had to do with Linux' max number of open files as the number of processed files were really near to that max value. It was a coincidence, though.
Thanks for all your help, and sorry for the noise.
I don't know if this will solve the problem, but try. It may be that the program is opening multiple files in the same variable or loop prevents the program to close the files.
import os
files = os.listdir('/var/tmp/files')
fd = list()
for file in files:
if files > 100000:
break
fd.append(open('/var/tmp/files/{}'.format(file), 'r'))
content = fd[file].readlines()
# Doing stuff
for file in files:
if files > 100000:
break
fd[files].close()
I think that you use multiprocess or something in your "Doing Stuff" block. You can assume that you will always face problems related file descriptor when you use multiprocess, so you need more attention.
To solve this problem, simply don't open file before you start another process. You should open file after another process started.
There must be something else happening in your code. Check the status of your file object before you reopen the file:
import os
files = os.listdir('/var/tmp/files')
fileClosed=True
for file in files:
if fileClosed:
with open('/var/tmp/files/{}'.format(file), 'r') as fd:
content = fd.readlines()
## DO Stuff
else:
print "File not closed properly"
break
fileClosed = fd.closed
I'm using python 2.7.8. I'm working with 30 open docx files simultaneously.
Is there some way with python code to close all the files simultaneously instead closing every file separately ?
UPDATE:
I'm using different files every day so the names of the files change every time. My code must generally without specific names files (if it is possible)
I suggest is using the with statement when opening files:
with open('file1.txt', 'w') as file1, open('file2.txt', 'w') as file2:
file1.write('stuff to write')
file2.write('stuff to write')
...do other stuff...
print "Both files are closed because I'm out of the with statement"
When you leave the with statement, your file closes. You can even open all of your files on one line, but it's not recommended unless you are actively using all 20 files at once.
You need to find the pid of your word files and then use kill method to terminate the word file's process. e.g
import os
os.kill(pid, signal.SIGTERM)
first you have to append all opened file object in list.
l = []
f1 = open('f1.txt'):
#...do something
l.append(f1)
f2 = open('f2.txt'):
#...do something
l.append(f2)
Now get all files object from list and close them.
for files in l:
files.close()
I have the following code which is intended to remove specific lines of a file. When I run it, it prints the two filenames that live in the directory, then deletes all information in them. What am I doing wrong? I'm using Python 3.2 under Windows.
import os
files = [file for file in os.listdir() if file.split(".")[-1] == "txt"]
for file in files:
print(file)
input = open(file,"r")
output = open(file,"w")
for line in input:
print(line)
# if line is good, write it to output
input.close()
output.close()
open(file, 'w') wipes the file. To prevent that, open it in r+ mode (read+write/don't wipe), then read it all at once, filter the lines, and write them back out again. Something like
with open(file, "r+") as f:
lines = f.readlines() # read entire file into memory
f.seek(0) # go back to the beginning of the file
f.writelines(filter(good, lines)) # dump the filtered lines back
f.truncate() # wipe the remains of the old file
I've assumed that good is a function telling whether a line should be kept.
If your file fits in memory, the easiest solution is to open the file for reading, read its contents to memory, close the file, open it for writing and write the filtered output back:
with open(file_name) as f:
lines = list(f)
# filter lines
with open(file_name, "w") as f: # This removes the file contents
f.writelines(lines)
Since you are not intermangling read and write operations, the advanced file modes like "r+" are unnecessary here, and only compicate things.
If the file does not fit into memory, the usual approach is to write the output to a new, temporary file, and move it back to the original file name after processing is finished.
One way is to use the fileinput stdlib module. Then you don't have to worry about open/closing and file modes etc...
import fileinput
from contextlib import closing
import os
fnames = [fname for fname in os.listdir() if fname.split(".")[-1] == "txt"] # use splitext
with closing(fileinput.input(fnames, inplace=True)) as fin:
for line in fin:
# some condition
if 'z' not in line: # your condition here
print line, # suppress new line but adjust for py3 - print(line, eol='') ?
When using inplace=True - the fileinput redirects stdout to be to the file currently opened. A backup of the file with a default '.bak' extension is created which may come in useful if needed.
jon#minerva:~$ cat testtext.txt
one
two
three
four
five
six
seven
eight
nine
ten
After running the above with a condition of not line.startswith('t'):
jon#minerva:~$ cat testtext.txt
one
four
five
six
seven
eight
nine
You're deleting everything when you open the file to write to it. You can't have an open read and write to a file at the same time. Use open(file,"r+") instead, and then save all the lines to another variable before writing anything.
You should not open the same file for reading and writing at the same time.
"w" means create a empty for writing. If the file already exists, its data will be deleted.
So you can use a different file name for writing.
I'm new to Python, and I need to do a parsing exercise. I got a file, and I need to parse it (just the headers), but after the process, i need to keep the file the same format, the same extension, and at the same place in disk, but only with the differences of new headers..
I tried this code...
for line in open ('/home/name/db/str/dir/numbers/str.phy'):
if line.startswith('ENS'):
linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
print linepars
..and it does the job, but I don't know how to "overwrite" the file with the new parsing.
The easiest way, but not the most efficient (by far, and especially for long files) would be to rewrite the complete file.
You could do this by opening a second file handle and rewriting each line, except in the case of the header, you'd write the parsed header. For example,
fr = open('/home/name/db/str/dir/numbers/str.phy')
fw = open('/home/name/db/str/dir/numbers/str.phy.parsed', 'w') # Name this whatever makes sense
for line in fr:
if line.startswith('ENS'):
linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
fw.write(linepars)
else:
fw.write(line)
fw.close()
fr.close()
EDIT: Note that this does not use readlines(), so its more memory efficient. It also does not store every output line, but only one at a time, writing it to file immediately.
Just as a cool trick, you could use the with statement on the input file to avoid having to close it (Python 2.5+):
fw = open('/home/name/db/str/dir/numbers/str.phy.parsed', 'w') # Name this whatever makes sense
with open('/home/name/db/str/dir/numbers/str.phy') as fr:
for line in fr:
if line.startswith('ENS'):
linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
fw.write(linepars)
else:
fw.write(line)
fw.close()
P.S. Welcome :-)
As others are saying here, you want to open a file and use that file object's .write() method.
The best approach would be to open an additional file for writing:
import os
current_cfg = open(...)
parsed_cfg = open(..., 'w')
for line in current_cfg:
new_line = parse(line)
print new_line
parsed.cfg.write(new_line + '\n')
current_cfg.close()
parsed_cfg.close()
os.rename(....) # Rename old file to backup name
os.rename(....) # Rename new file into place
Additionally I'd suggest looking at the tempfile module and use one of its methods for either naming your new file or opening/creating it. Personally I'd favor putting the new file in the same directory as the existing file to ensure that os.rename will work atomically (the configuration file named will be guaranteed to either point at the old file or the new file; in no case would it point at a partially written/copied file).
The following code DOES the job.
I mean it DOES overwrite the file ON ONESELF; that's what the OP asked for. That's possible because the transformations are only removing characters, so the file's pointer fo that writes is always BEHIND the file's pointer fi that reads.
import re
regx = re.compile('\AENS([A-Z]+)0+([0-9]{6})')
with open('bomo.phy','rb+') as fi, open('bomo.phy','rb+') as fo:
fo.writelines(regx.sub('\\1\\2',line) for line in fi)
I think that the writing isn't performed by the operating system one line at a time but through a buffer. So several lines are read before a pool of transformed lines are written. That's what I think.
newlines = []
for line in open ('/home/name/db/str/dir/numbers/str.phy').readlines():
if line.startswith('ENS'):
linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
newlines.append( linepars )
open ('/home/name/db/str/dir/numbers/str.phy', 'w').write('\n'.join(newlines))
(sidenote: Of course if you are working with large files, you should be aware that the level of optimization required may depend on your situation. Python by nature is very non-lazily-evaluated. The following solution is not a good choice if you are parsing large files, such as database dumps or logs, but a few tweaks such as nesting the with clauses and using lazy generators or a line-by-line algorithm can allow O(1)-memory behavior.)
targetFile = '/home/name/db/str/dir/numbers/str.phy'
def replaceIfHeader(line):
if line.startswith('ENS'):
return re.sub('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
else:
return line
with open(targetFile, 'r') as f:
newText = '\n'.join(replaceIfHeader(line) for line in f)
try:
# make backup of targetFile
with open(targetFile, 'w') as f:
f.write(newText)
except:
# error encountered, do something to inform user where backup of targetFile is
edit: thanks to Jeff for suggestion