Develop Reference

Python 3: Open file in read mode without raising an exception? [duplicate]

This question already has answers here:
How do I check whether a file exists without exceptions?
(40 answers)
Closed 1 year ago.
I am trying to write a block of code which opens a new file every time a Python3 script is run.
I am constructing the filename using an incrementing number.
For example, the following are some examples of valid filenames which should be produced:
output_0.csv
output_1.csv
output_2.csv
output_3.csv
On the next run of the script, the next filename to be used should be output_4.csv.
In C/C++ I would do this in the following way:
Enter an infinite loop
Try to open the first filename, in "read" mode
If the file is open, increment the filename number and repeat
If the file is not open, break out of the loop and re-open the file in "write" mode
This doesn't seem to work in Python 3, as opening a non-existing file in read mode causes an exception to be raised.
One possible solution might be to move the open file code block inside a try-catch block. But this doesn't seem like a particularly elegant solution.
Here is what I tried so far in code
# open a file to store output data
filename_base = "output"
filename_ext = "csv"
filename_number = 0
while True:
filename_full = f"{filename_base}_{filename_number}.{filename_ext}"
with open(filename_full, "r") as f:
if f.closed:
print(f"Writing data to {filename_full}")
break
else:
print(f"File {filename_full} exists")
filename_number += 1
with open(filename_full, "w") as f:
pass
As explained above this code crashes when trying to open a file which does not exist in "read" mode.

Using pathlib you can check with Path.is_file() which returns True when it encounters a file or a symbolic link to a file.
from pathlib import Path
filename_base = "output"
filename_ext = "csv"
filename_number = 0
filename_full = f"{filename_base}_{filename_number}.{filename_ext}"
p = Path(filename_full)
while p.is_file() or p.is_dir():
filename_number += 1
p = Path(f"{filename_base}_{filename_number}.{filename_ext}")
This loop should exit when the file isn’t there so you can open it for writing.

you can check if a file exists prior using
os.path.exists(filename)

You could use the OS module to check if the file path is a file, and then open it:
import os
file_path = './file.csv'
if(os.path.isfile(file_path)):
with open(file_path, "r") as f:

This should work:
filename_base = "output"
filename_ext = "csv"
filename_number = 0
while True:
filename_full = f"{filename_base}_{filename_number}.{filename_ext}"
try:
with open(filename_full, "r") as f:
print(f"File {filename_full} exists")
filename_number += 1
except FileNotFoundError:
print("Creating new file")
open(filename_full, 'w');
break;

You might os.path.exists to check if file already exists for example
import os
print(os.path.exists("output_0.csv"))
or harness fact that your names
output_0.csv
output_1.csv
output_2.csv
output_3.csv
are so regular, exploit glob.glob like so
import glob
existing = glob.glob("output_*.csv")
print(existing) # list of existing files

Python retry to open the file

What is the best practice to try to open the file and retry after n seconds?
Currently, I do:
import os
from os import path
import shutil
dir_path = path.abspath(path.join("foo", "bar"))
destination_path = path.abspath(path.join("foo", "dest_dir"))
for f in dir_path:
try:
# try to open the file if the file isn't used any longer
file_opened = open(f, 'r')
# move the file on successful opening
shutil.move(file_opened, destination_path)
file_opened.close()
except IOError:
return False
So, at the moment I do not handle the exception. I think about creation of extra function to open the file and recall the function on excepth with time.sleep(n)
However, I am sure there must be something else ...
I do not use
with open(f, 'rb') as file_opened:
do whatever`
EDIT:
One process creates the file and I want Python process to move the file once I am sure the file writting / creation is completed. So, I have added shutil.move in the above code to show the whole situation.
EDIT:
Please find below the code I have developed to solve the problem. I ended with writing own custom solution to handle it:
import os
from os import path
import psutil
from retry import retry
import shutil
from subprocess import check_output,Popen, PIPE
import glob
import time
class FileHandler:
def __init__(self, fn_source, dir_source):
self.file_source_name = fn_source
self.file_source_path = path.join(dir_source, self.file_source_name)
self.dir_dest_path = path.join(dir_source, "test")
self.file_dest_path = path.join(self.dir_dest_path, self.file_source_name)
def check_file(self):
if os.path.exists(self.file_source_path):
try:
os.rename(self.file_source_path, self.file_source_path)
print("file renamed")
return True
except:
print("can not rename the file..retrying")
time.sleep(1)
self.check_file()
else:
print("source file does not exist...retrying")
time.sleep(5)
self.check_file()
def check_destination(self):
if os.path.exists(self.file_source_path) and not os.path.exists(self.file_dest_path):
return True
elif os.path.exists(self.file_source_path) and os.path.exists(self.file_dest_path):
try:
print(self.file_dest_path, self.file_source_name)
os.remove(self.file_dest_path)
return True
except Exception as e:
print("can not remove the file..retrying")
time.sleep(5)
self.check_destination()
def move_file(self):
if self.check_destination():
print(self.file_source_path)
shutil.move(self.file_source_path, self.file_dest_path)
print("moved", str(self.file_source_path))
return True
else:
print("can not move the file..retrying")
time.sleep(1)
self.move_file()
def file_ops(self):
if self.check_file():
self.move_file()
else:
print("source does not exist")
time.sleep(1)
self.file_ops()
return True
def each_file_ops(fn, dir_source):
fh = FileHandler(fn, dir_source)
return fh.file_ops()
def main(dir_source):
dir_files = glob.glob(path.join(dir_source, '*.txt'))
if dir_files:
[each_file_ops(f, dir_source) for f in dir_files]
else:
print("source dir is empty")
time.sleep(1)
main(dir_source)
if __name__ == '__main__':
main(path.join(""))

You can use the retry module for these kind of retrying. This makes the code to look much cleaner. pip install retry should install the module
from retry import retry
import shutil
#retry((FileNotFoundError, IOError), delay=1, backoff=2, max_delay=10, tries=100)
def attempt_to_move_file(fname, dest_path):
# Your attempt to move file
# shutil.move(fname, destination_path)
With the above code, when attempt_to_move_file is invoked it would be retried (upto a max of 100 tries) whenever we hit FileNotFoundError or IOError and the retry happens with a sleep 1, 2, 4, 8, 10, 10, 10 ... seconds between attempts

you do not need to open a file to move it. Because once you have opened a file it is in open state hence you can not move it, Take it like this, when you are playing a video in a player and you try to delete it or cut and paste system won't allow you.
import shutil
file_name = 'trytry.csv'
shutil.copyfile(file_name, '/Users/umeshkaushik/PycharmProjects/newtry.csv')
shutil.move(file_name, '/Users/umeshkaushik/PycharmProjects/newtry1.csv')
Above code runs fine. Just make sure you are providing correct input and output path.

I need to prevent a python script from running twice

I need to prevent a python script from running more than once. So far I have:
import fcntl
def lockFile(lockfile):
fp = open(lockfile, 'w')
try:
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
return False
return True
if not lockFile("myfile.lock"):
sys.exit(0)
Problem is the sys.exit() never gets called even if the file is there. Maybe this is a platform dependent way of doing things? I just need to write a lockfile, check for its existence and if it's not there or stale, create a new one. Ideas?

Writing a file will create a new file if none exist; you could instead try to read the file first: if there is none, an error is raised and a file a file written; if there is a file, the program exits.
try:
with open('lockfile.txt', 'r') as f:
lock = f.readline().strip().split()
if lock[0] == 'locked':
print('exiting')
sys.exit(0)
except FileNotFoundError:
with open('lockfile.txt', 'w') as f:
f.write('locked')
print('file written')
if __name__ == '__main__':
pass
If you need something more sophisticated, you could look up the atexit module

You can check if you file exists by using os.path.exists (docs here). If it does, then you can use the sys.exit you mentioned before. If you need something more than sys.exit, try the atexit module #Reblochon suggested. The script will then assume the file is ready to lock and the method will report its success back to the user via a boolean.
import os
import sys
import fcntl
FILE_NAME = 'myfile.lock'
def lockFile(lockfile):
fp = open(lockfile, 'w') # create a new one
try:
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
success = True # the file has been locked
except IOError:
success = False # an error occurred
fp.close() # make sure to close your file
return success
if os.path.exists(FILE_NAME): # exit the script if it exists
sys.exit(0)
print('success', lockFile(FILE_NAME))

Make python code continue after exception

I'm trying to read all files from a folder that matches a certain criteria. My program crashes once I have an exception raised. I am trying to continue even if there's an exception but it still stops executing.
This is what I get after a couple of seconds.
error <type 'exceptions.IOError'>
Here's my code
import os
path = 'Y:\\Files\\'
listing = os.listdir(path)
try:
for infile in listing:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
print "error "+str(IOError)
pass

Put your try/except structure more in-wards. Otherwise when you get an error, it will break all the loops.
Perhaps after the first for-loop, add the try/except. Then if an error is raised, it will continue with the next file.
for infile in listing:
try:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
pass
This is a perfect example of why you should use a with statement here to open files. When you open the file using open(), but an error is catched, the file will remain open forever. Now is better than never.
for infile in listing:
try:
if infile.startswith("ABC"):
with open(infile,"r") as fo
for line in fo:
if line.startswith("REVIEW"):
print infile
except:
pass
Now if an error is caught, the file will be closed, as that is what the with statement does.

Move the try/except inside the for loop.
Like in:
import os
path = 'C:\\'
listing = os.listdir(path)
for infile in listing:
try:
if infile.startswith("ABC"):
fo = open(infile,"r")
for line in fo:
if line.startswith("REVIEW"):
print infile
fo.close()
except:
print "error "+str(IOError)

You're code is doing exactly what you're telling it to do. When you get an exception, it jumps down to this section:
except:
print "error "+str(IOError)
pass
Since there's nothing after that, the program ends.
Also, that pass is superfluous.

As per strictest interpretation of the question "continue even if there's an exception". Python gives us a keyword "finally" which executes a block of code no matter what precedes it. The only issue with this method will run a block of code regardless of the type of error, which might not be desirable for all cases.
try:
unreal = 3/0 # raises divide by zero exception
print(unreal)
# handles zerodivision exception
except :
print("Can't divide by zero, 0 has no multiplicative inverse")
finally:
# this block is always executed
print("Brahmagupta claimed that “zero divided by a zero is zero.”)

Error when creating a new text file with python?

This function doesn't work and raises an error. Do I need to change any arguments or parameters?
import sys
def write():
print('Creating new text file')
name = input('Enter name of text file: ')+'.txt' # Name of text file coerced with +.txt
try:
file = open(name,'r+') # Trying to create a new file or open one
file.close()
except:
print('Something went wrong! Can\'t tell what?')
sys.exit(0) # quit Python
write()

If the file does not exists, open(name,'r+') will fail.
You can use open(name, 'w'), which creates the file if the file does not exist, but it will truncate the existing file.
Alternatively, you can use open(name, 'a'); this will create the file if the file does not exist, but will not truncate the existing file.

instead of using try-except blocks, you could use, if else
this will not execute if the file is non-existent,
open(name,'r+')
if os.path.exists('location\filename.txt'):
print "File exists"
else:
open("location\filename.txt", 'w')
'w' creates a file if its non-exis

following script will use to create any kind of file, with user input as extension
import sys
def create():
print("creating new file")
name=raw_input ("enter the name of file:")
extension=raw_input ("enter extension of file:")
try:
name=name+"."+extension
file=open(name,'a')
file.close()
except:
print("error occured")
sys.exit(0)
create()

This works just fine, but instead of
name = input('Enter name of text file: ')+'.txt'
you should use
name = raw_input('Enter name of text file: ')+'.txt'
along with
open(name,'a') or open(name,'w')

import sys
def write():
print('Creating new text file')
name = raw_input('Enter name of text file: ')+'.txt' # Name of text file coerced with +.txt
try:
file = open(name,'a') # Trying to create a new file or open one
file.close()
except:
print('Something went wrong! Can\'t tell what?')
sys.exit(0) # quit Python
write()
this will work promise :)

You can os.system function for simplicity :
import os
os.system("touch filename.extension")
This invokes system terminal to accomplish the task.

You can use open(name, 'a')
However, when you enter filename, use inverted commas on both sides, otherwise ".txt"cannot be added to filename