Is there an option I can pass open() that will cause an IOerror when trying to write a nonexistent file? I am using python to read and write block devices via symlinks, and if the link is missing I want to raise an error rather than create a regular file. I know I could add a check to see if the file exists and manually raise the error, but would prefer to use something built-in if it exists.
Current code looks like this:
device = open(device_path, 'wb', 0)
device.write(data)
device.close()
Yes.
open(path, 'r+b')
Specifying the "r" option means the file must exist and you can read.
Specifying "+" means you can write and that you will be positioned at the end.
https://docs.python.org/3/library/functions.html?#open
Use os.path.islink() or os.path.isfile() to check if the file exists.
Doing the check each time is a nuisance, but you can always wrap open():
import os
def open_if_exists(*args, **kwargs):
if not os.path.exists(args[0]):
raise IOError('{:s} does not exist.'.format(args[0]))
f = open(*args, **kwargs)
return f
f = open_if_exists(r'file_does_not_exist.txt', 'w+')
This is just quick and dirty, so it doesn't allow for usage as: with open_if_exists(...).
Update
The lack of a context manager was bothering me, so here goes:
import os
from contextlib import contextmanager
#contextmanager
def open_if_exists(*args, **kwargs):
if not os.path.exists(args[0]):
raise IOError('{:s} does not exist.'.format(args[0]))
f = open(*args, **kwargs)
try:
yield f
finally:
f.close()
with open_if_exists(r'file_does_not_exist.txt', 'w+') as f:
print('foo', file=f)
I am afraid you can't perform the check of file existence and raise error using the open() function.
Below is the signature of open() in python where name is the file_name, mode is the access mode and buffering to indicate if buffering is to be performed while accessing a file.
open(name[, mode[, buffering]])
Instead, you can check if the file exists or not.
>>> import os
>>> os.path.isfile(file_name)
This will return True or False depending on if the file exists. To test a file specifically, you can use this.
To test the existence of both files and directories, you can use:
>>> os.path.exists(file_path)
Related
In my code, user uploads file which is saved on server and read using the server path. I'm trying to delete the file from that path after I'm done reading it. But it gives me following error instead:
An error occurred while reading file. [WinError 32] The process cannot access the file because it is being used by another process
I'm reading file using with, and I've tried f.close() and also f.closed but its the same error every time.
This is my code:
f = open(filePath)
with f:
line = f.readline().strip()
tempLst = line.split(fileSeparator)
if(len(lstHeader) != len(tempLst)):
headerErrorMsg = "invalid headers"
hjsonObj["Line No."] = 1
hjsonObj["Error Detail"] = headerErrorMsg
data['lstErrorData'].append(hjsonObj)
data["status"] = True
f.closed
return data
f.closed
after this code I call the remove function:
os.remove(filePath)
Edit: using with open(filePath) as f: and then trying to remove the file gives the same error.
Instead of:
f.closed
You need to say:
f.close()
closed is just a boolean property on the file object to indicate if the file is actually closed.
close() is method on the file object that actually closes the file.
Side note: attempting a file delete after closing a file handle is not 100% reliable. The file might still be getting scanned by the virus scanner or indexer. Or some other system hook is holding on to the file reference, etc... If the delete fails, wait a second and try again.
Use below code:
import os
os.startfile('your_file.py')
To delete after completion:
os.remove('your_file.py')
This
import os
path = 'path/to/file'
with open(path) as f:
for l in f:
print l,
os.remove(path)
should work, with statement will automatically close the file after the nested block of code
if it fails, File could be in use by some external factor. you can use Redo pattern.
while True:
try:
os.remove(path)
break
except:
time.sleep(1)
There is probably an application that is opening the file; check and close the application before executing your code:
os.remove(file_path)
Delete files that are not used by another application.
I have a Matlab application that writes in to a .csv file and a Python script that reads from it. These operations happen concurrently and at their own respective periods (not necessarily the same). All of this runs on Windows 7.
I wish to know :
Would the OS inherently provide some sort of locking mechanism so that only one of the two applications - Matlab or Python - have access to the shared file?
In the Python application, how do I check if the file is already "open"ed by Matlab application? What's the loop structure for this so that the Python application is blocked until it gets access to read the file?
I am not sure about window's API for locking files
Heres a possible solution:
While matlab has the file open, you create an empty file called "data.lock" or something to that effect.
When python tries to read the file, it will check for the lock file, and if it is there, then it will sleep for a given interval.
When matlab is done with the file, it can delete the "data.lock" file.
Its a programmatic solution, but it is simpler than digging through the windows api and finding the right calls in matlab and python.
If Python is only reading the file, I believe you have to lock it in MATLAB because a read-only open call from Python may not fail. I am not sure how to accomplish that, you may want to read this question atomically creating a file lock in MATLAB (file mutex)
However, if you are simply consuming the data with python, did you consider using a socket instead of a file?
In Windows on the Python side, CreateFile can be called (directly or indirectly via the CRT) with a specific sharing mode. For example, if the desired sharing mode is FILE_SHARE_READ, then the open will fail if the file is already open for writing. If the latter call instead succeeds, then a future attempt to open the file for writing will fail (e.g. in Matlab).
The Windows CRT function _wsopen_s allows setting the sharing mode. You can call it with ctypes in a Python 3 opener:
import sys
import os
import ctypes as ctypes
import ctypes.util
__all__ = ['shdeny', 'shdeny_write', 'shdeny_read']
_SH_DENYRW = 0x10 # deny read/write mode
_SH_DENYWR = 0x20 # deny write mode
_SH_DENYRD = 0x30 # deny read
_S_IWRITE = 0x0080 # for O_CREAT, a new file is not readonly
if sys.version_info[:2] < (3,5):
_wsopen_s = ctypes.CDLL(ctypes.util.find_library('c'))._wsopen_s
else:
# find_library('c') may be deprecated on Windows in 3.5, if the
# universal CRT removes named exports. The following probably
# isn't future proof; I don't know how the '-l1-1-0' suffix
# should be handled.
_wsopen_s = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')._wsopen_s
_wsopen_s.argtypes = (ctypes.POINTER(ctypes.c_int), # pfh
ctypes.c_wchar_p, # filename
ctypes.c_int, # oflag
ctypes.c_int, # shflag
ctypes.c_int) # pmode
def shdeny(file, flags):
fh = ctypes.c_int()
err = _wsopen_s(ctypes.byref(fh),
file, flags, _SH_DENYRW, _S_IWRITE)
if err:
raise IOError(err, os.strerror(err), file)
return fh.value
def shdeny_write(file, flags):
fh = ctypes.c_int()
err = _wsopen_s(ctypes.byref(fh),
file, flags, _SH_DENYWR, _S_IWRITE)
if err:
raise IOError(err, os.strerror(err), file)
return fh.value
def shdeny_read(file, flags):
fh = ctypes.c_int()
err = _wsopen_s(ctypes.byref(fh),
file, flags, _SH_DENYRD, _S_IWRITE)
if err:
raise IOError(err, os.strerror(err), file)
return fh.value
For example:
if __name__ == '__main__':
import tempfile
filename = tempfile.mktemp()
fw = open(filename, 'w')
fw.write('spam')
fw.flush()
fr = open(filename)
assert fr.read() == 'spam'
try:
f = open(filename, opener=shdeny_write)
except PermissionError:
fw.close()
with open(filename, opener=shdeny_write) as f:
assert f.read() == 'spam'
try:
f = open(filename, opener=shdeny_read)
except PermissionError:
fr.close()
with open(filename, opener=shdeny_read) as f:
assert f.read() == 'spam'
with open(filename, opener=shdeny) as f:
assert f.read() == 'spam'
os.remove(filename)
In Python 2 you'll have to combine the above openers with os.fdopen, e.g.:
f = os.fdopen(shdeny_write(filename, os.O_RDONLY|os.O_TEXT), 'r')
Or define an sopen wrapper that lets you pass the share mode explicitly and calls os.fdopen to return a Python 2 file. This will require a bit more work to get the file mode from the passed in flags, or vice versa.
I am trying to write a script that will read all the files in a directory and dump them into a single file. What I have is:
from glob import glob
directory = glob('/Users/jmanley/Desktop/Table/*')
with outfile as open('/Users/jmanley/Desktop/Table.sql', 'wb'):
for file in directory:
with readfile as open(file, 'rb'):
outfile.write(readfile.read())
I get "can't assign to function call" as the error message, and IDLE marks the with keyword as the location of the error.
If I rewrite the script to use open() and close() methods rather than using the with keyword, it runs without issue:
from glob import glob
directory = glob('/Users/jmanley/Desktop/Table/*')
outfile = open('/Users/jmanley/Desktop/Table.sql', 'wb')
for file in directory:
readfile = open(file, 'rb')
outfile.write(readfile.read())
readfile.close()
outfile.close()
Why am I getting the "can't assign to function call" error? The only time I've seen this happen is if an assignment is reversed: a + b = variable. Am I just missing something incredibly obvious?
Note that:
with foo as bar:
is (very, very roughly) equivalent to:
bar = foo
(This is consistent with other uses of as in Python, e.g. except ValueError as err:.)
Therefore when you try:
with outfile as open('/Users/jmanley/Desktop/Table.sql', 'wb'):
you are actually trying to assign:
open('/Users/jmanley/Desktop/Table.sql', 'wb') = outfile
which is clearly incorrect. Instead, you need to reverse the statement:
with open('/Users/jmanley/Desktop/Table.sql', 'wb') as outfile:
See also the relevant PEP.
Instead of this:
FILE = open(f)
do_something(FILE)
FILE.close()
it's better to use this:
with open(f) as FILE:
do_something(FILE)
What if I have something like this?
if f is not None:
FILE = open(f)
else:
FILE = None
do_something(FILE)
if FILE is not None:
FILE.close()
Where do_something also has an "if FILE is None" clause, and still does something useful in that case - I don't want to just skip do_something if FILE is None.
Is there a sensible way of converting this to with/as form? Or am I just trying to solve the optional file problem in a wrong way?
If you were to just write it like this:
if f is not None:
with open(f) as FILE:
do_something(FILE)
else:
do_something(f)
(file is a builtin btw )
Update
Here is a funky way to do an on-the-fly context with an optional None that won't crash:
from contextlib import contextmanager
none_context = contextmanager(lambda: iter([None]))()
# <contextlib.GeneratorContextManager at 0x1021a0110>
with (open(f) if f is not None else none_context) as FILE:
do_something(FILE)
It creates a context that returns a None value. The with will either produce FILE as a file object, or a None type. But the None type will have a proper __exit__
Update
If you are using Python 3.7 or higher, then you can declare the null context manager for stand-in purposes in a much simpler way:
import contextlib
none_context = contextlib.nullcontext()
You can read more about these here:
https://docs.python.org/3.7/library/contextlib.html#contextlib.nullcontext
Since Python 3.7, you can also do
from contextlib import nullcontext
with (open(file) if file else nullcontext()) as FILE:
# Do something with `FILE`
pass
See the official documentation for more details.
This seems to solve all of your concerns.
if file_name is not None:
with open(file_name) as fh:
do_something(fh)
else:
do_something(None)
something like:
if file: #it checks for None,false values no need of "if file is None"
with open(file) as FILE:
do_something(FILE)
else:
FILE=None
In Python 3.3 and above, you can use contextlib.ExitStack to handle this scenario nicely
with contextlib.ExitStack() as stack:
FILE = stack.enter_context(open(f)) if f else None
do_something(FILE)
Python 3.7 supports contextlib.nullcontext, which can be used to avoid creating your own dummy context manager.
This examples shows how you can conditionally open a file or use the stdout:
import contextlib
import sys
def write_to_file_or_stdout(filepath=None, data):
with (
open(filepath, 'w') if filepath is not None else
contextlib.nullcontext(sys.stdout)
) as file_handle:
file_handle.write(data)
contextlib.nullcontext() can be called without any arguments if the value can be None.
While all of the other answers are excellent, and preferable, note that the with expression may be any expression, so you can do:
with (open(file) if file is not None else None) as FILE:
pass
Note that if the else clause were evaluated, to yield None this would result in an exception, because NoneType does not support the appropriate operations to be used as a context manager.
I have a script that accepts as an argument a filename than opens it and writes some stuff.
I use the with statement:
with open(file_name, 'w') as out_file:
...
out_file.write(...)
Now what if I want to write to sys.stdout if no file_name is provided?
Do I necessarily need to wrap all actions in a function and put a condition before?
if file_name is None:
do_everything(sys.stdout)
else:
with open(file_name, 'w') as out_file:
do_everything(out_file)
from contextlib import contextmanager
#contextmanager
def file_or_stdout(file_name):
if file_name is None:
yield sys.stdout
else:
with open(file_name, 'w') as out_file:
yield out_file
Then you can do
with file_or_stdout(file_name) as wfile:
do_stuff_writing_to(wfile)
How do you handle command line arguments? If you use argparse you could use the type and default parameters of add_argument to handle this. For example, try something like the following:
import sys
import argparse
def main(argv=None):
if argv is None:
argv=sys.argv[1:]
parser = argparse.ArgumentParser()
parser.add_argument('infile', nargs='?',
type=argparse.FileType('w'),
default=sys.stdin)
args = parser.parse_args(argv)
print args.infile
return 0
if __name__=="__main__":
sys.exit(main(sys.argv[1:]))
If a file name is present as an argument the the script argparse will automatically open and close this file and args.infile will be a handle to this file. Otherwise args.infile will simply be sys.stdin.
You could write your own context manager. I'll post sample code later if noone else does
if file_name is None:
fd = sys.stdout
else:
fd = open(file_name, 'w')
# write to fd
if fd != sys.stdout:
fd.close();
Using the with ... as construct is useful to close the file automatically. This means that using it with sys.stdout, as I guess you know, would crash your program, because it would attempt at closing the system stdout!
This means something like with open(name, 'w') if name else sys.stdout as: would not work.
This make me say there isn't any simple-nice way to write your snippet better... but there are probably better ways to think on how to construct such a code!
The main point to clarify is when you need to open (and, more importantly, close) the filehandler for file_name, when file_name exists.
Personally I would simply drop the with .. as and take care of opening the file - and, more importantly, close it! - somewhere else. Mileage for that might vary depending on how your software is working.
This means you can simply do:
out_file = open(file_name, 'w') if file_name else sys.stdout
and work with out_file throughout your program.
When you close, remember to check if it's a file or not :)
And have you thought about simply using the logging module? That easily allows you to add different handlers, print to file, print to stdout...