What is meant by "using the EAFP principle" in Python? Could you provide any examples?
From the glossary:
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.
An example would be an attempt to access a dictionary key.
EAFP:
try:
x = my_dict["key"]
except KeyError:
# handle missing key
LBYL:
if "key" in my_dict:
x = my_dict["key"]
else:
# handle missing key
The LBYL version has to search the key inside the dictionary twice, and might also be considered slightly less readable.
I'll try to explain it with another example.
Here we're trying to access the file and print the contents in console.
LBYL - Look Before You Leap :
We might want to check if we can access the file and if we can, we'll open it and print the contents. If we can't access the file we'll hit the else part. The reason that this is a race condition is because we first make an access-check. By the time we reach with open(my_file) as f: maybe we can't access it anymore due to some permission issues (for example another process gains an exclusive file lock). This code will likely throw an error and we won't be able to catch that error because we thought that we could access the file.
import os
my_file = "/path/to/my/file.txt"
# Race condition
if os.access(my_file, os.R_OK):
with open(my_file) as f:
print(f.read())
else:
print("File can't be accessed")
EAFP - Easier to Ask for Forgiveness than Permission :
In this example, we're just trying to open the file and if we can't open it, it'll throw an IOError. If we can, we'll open the file and print the contents. So instead of asking something we're trying to do it. If it works, great! If it doesn't we catch the error and handle it.
# # No race condition
try:
f = open(my_file)
except IOError as e:
print("File can't be accessed")
else:
with f:
print(f.read())
I call it "optimistic programming". The idea is that most times people will do the right thing, and errors should be few. So code first for the "right thing" to happen, and then catch the errors if they don't.
My feeling is that if a user is going to be making mistakes, they should be the one to suffer the time consequences. People who use the tool the right way are sped through.
Related
I found many articels about this topic, but it didn't become clear to me which is the correct or rather most secure way to open and close files in python. Maybe there are more ways to use files in python, but most often I have come across these two ways:
Example 1:
with open("example.txt", "r") as f:
# do things
Example 2:
f = open("example.txt", "r")
try:
# do things
finally:
f.close()
As far as I know, the only difference is that you can raise exceptions in the try..finally block. Is this correct or are there more differences? And still there is the question, which way is the correct way? I really appreciate any kind of help or suggestion, sheers!
Generally as from the python docs here
The with-statement is the ideal method to use here, especially as it gives many options (so you can also raise exceptions there, or check if everything went as planned or not)
The with-statement also allows custom call-backs for __exit__ and __enter__ with additional parameters to get the current state of the process.
It has some additional methods with context managers that can be more useful.
So in short, if you just use the with statement without any further functions, it basically works to make sure that the resource is closed (here being the data stream), while it also gives you the option to track the closing state, with many more options than a usual try/finally block.
I highly recommend going through the link for more information.
The most Pythonic way and a good practice is to open the file using the with statement:
with open("example.txt", "r") as f:
# do things
And you are correct, the only difference for
f = open("example.txt", "r")
try:
# do things
finally:
f.close()
is that you can have a custom behavior in the finally block.
Note that using with open("example.txt", "r") as f: Python internally behaves very similar to your try-finally block as the documentation states:
IOBase is also a context manager and therefore supports the with statement. In this example, file is closed after the with statement’s suite is finished—even if an exception occurs
I'm trying to check the readability of a file given the specified path. Here's what I have:
def read_permissions(filepath):
'''Checks the read permissions of the specified file'''
try:
os.access(filepath, os.R_OK) # Find the permissions using os.access
except IOError:
return False
return True
This works and returns True or False as the output when run. However, I want the error messages from errno to accompany it. This is what I think I would have to do (But I know that there is something wrong):
def read_permissions(filepath):
'''Checks the read permissions of the specified file'''
try:
os.access(filepath, os.R_OK) # Find the permissions using os.access
except IOError as e:
print(os.strerror(e)) # Print the error message from errno as a string
print("File exists.")
However, if I were to type in a file that does not exist, it tells me that the file exists. Can someone help me as to what I have done wrong (and what I can do to stay away from this issue in the future)? I haven't seen anyone try this using os.access. I'm also open to other options to test the permissions of a file. Can someone help me in how to raise the appropriate error message when something goes wrong?
Also, this would likely apply to my other functions (They still use os.access when checking other things, such as the existence of a file using os.F_OK and the write permissions of a file using os.W_OK). Here is an example of the kind of thing that I am trying to simulate:
>>> read_permissions("located-in-restricted-directory.txt") # Because of a permission error (perhaps due to the directory)
[errno 13] Permission Denied
>>> read_permissions("does-not-exist.txt") # File does not exist
[errno 2] No such file or directory
This is the kind of thing that I am trying to simulate, by returning the appropriate error message to the issue. I hope that this will help avoid any confusion about my question.
I should probably point out that while I have read the os.access documentation, I am not trying to open the file later. I am simply trying to create a module in which some of the components are to check the permissions of a particular file. I have a baseline (The first piece of code that I had mentioned) which serves as a decision maker for the rest of my code. Here, I am simply trying to write it again, but in a user-friendly way (not just True or just False, but rather with complete messages). Since the IOError can be brought up a couple different ways (such as permission denied, or non-existent directory), I am trying to get my module to identify and publish the issue. I hope that this helps you to help me determine any possible solutions.
os.access returns False when the file does not exist, regardless of the mode parameter passed.
This isn't stated explicitly in the documentation for os.access but it's not terribly shocking behavior; after all, if a file doesn't exist, you can't possibly access it. Checking the access(2) man page as suggested by the docs gives another clue, in that access returns -1 in a wide variety of conditions. In any case, you can simply do as I did and check the return value in IDLE:
>>> import os
>>> os.access('does_not_exist.txt', os.R_OK)
False
In Python it's generally discouraged to go around checking types and such before trying to actually do useful things. This philosophy is often expressed with the initialism EAFP, which stands for Easier to Ask Forgiveness than Permission. If you refer to the docs again, you'll see this is particularly relevant in the present case:
Note: Using access() to check if a user is authorized to e.g. open a file before actually doing so using open() creates a security
hole, because the user might exploit the short time interval between
checking and opening the file to manipulate it. It’s preferable to use
EAFP techniques. For example:
if os.access("myfile", os.R_OK):
with open("myfile") as fp:
return fp.read()
return "some default data"
is better written as:
try:
fp = open("myfile")
except IOError as e:
if e.errno == errno.EACCES:
return "some default data"
# Not a permission error.
raise
else:
with fp:
return fp.read()
If you have other reasons for checking permissions than second-guessing the user before calling open(), you could look to How do I check whether a file exists using Python? for some suggestions. Remember that if you really need an exception to be raised, you can always raise it yourself; no need to go hunting for one in the wild.
Since the IOError can be brought up a couple different ways (such as
permission denied, or non-existent directory), I am trying to get my
module to identify and publish the issue.
That's what the second approach above does. See:
>>> try:
... open('file_no_existy.gif')
... except IOError as e:
... pass
...
>>> e.args
(2, 'No such file or directory')
>>> try:
... open('unreadable.txt')
... except IOError as e:
... pass
...
>>> e.args
(13, 'Permission denied')
>>> e.args == (e.errno, e.strerror)
True
But you need to pick one approach or the other. If you're asking forgiveness, do the thing (open the file) in a try-except block and deal with the consequences appropriately. If you succeed, then you know you succeeded because there's no exception.
On the other hand, if you ask permission (aka LBYL, Look Before You Leap) in this that and the other way, you still don't know if you can successfully open the file until you actually do it. What if the file gets moved after you check its permissions? What if there's a problem you didn't think to check for?
If you still want to ask permission, don't use try-except; you're not doing the thing so you're not going to throw errors. Instead, use conditional statements with calls to os.access as the condition.
I'm fairly inexperienced with python, so I find myself at a loss as to how to approach this bug. I've inherited a python app that mainly just copies files and folders from one place to another. All the files and folders are on the local machine and the user has full admin rights, so there are no networking or security issues here.
What I've found is that a number of files fail to get copied from one directory to another unless I slow down the code somehow. If I just run the program it fails, but if I step through with a debugger or add print statements to the copy loop, it succeeds. The difference there seems to be either the timing of the loop or moving things around in memory.
I've seen this sort of bug in compiled languages before, and it usually indicates either a race condition or memory corruption. However, there is only one thread and no interaction with other processes, so a race condition seems impossible. Memory corruption remains a possibility, but I'm not sure how to investigate that possibility with python.
Here is the loop in question. As you can see, it's rather simple:
def concat_dirs(dir, subdir):
return dir + "/" + subdir
for name in fullnames:
install_name = concat_dirs(install_path, name)
dirname = os.path.dirname(install_name)
if not os.path.exists(dirname):
os.makedirs(dirname)
shutil.copyfile(concat_dirs(java_path, name), install_name)
That loop usually fails to copy the files unless I either step through it with a debugger or add this statement after the shutil.copyfile line.
print "copied ", concat_dirs(java_path, name), " to ", install_name
If I add that statement or step through in debug, the loop works perfectly and consistently. I'm tempted to say "good enough" with the print statement but I know that's just masking an underlying problem.
I'm not asking you to debug my code because I know you can't; I'm asking for a debug strategy. How do I approach finding this bug?
You do have a race condition: you check for the existence of dirname and then try to create it. That should cause the program to bomb if something unexpected happens, but...
In Python, we says that it's easier to ask forgiveness than permission. Go ahead and create that directory each time, then apologize if it already exists:
import errno
import os
for name in fullnames:
source_name = os.path.join(java_path, name)
dest_name = os.path.join(install_path, name)
dest_dir = os.path.dirname(dest_name)
try:
os.makedirs(dest_dir)
except OSError as exc:
if exc.errno != errno.EEXIST:
raise
shutil.copyfile(source_name, dest_name)
I'm not sure how I'd troubleshoot this, other than by trying it the non-racy way and seeing what happens. There may be a subtle filesystem issue that's making this run oddly.
Problem:
I have code that looks for a file and open it. By default it looks for file that starts with ####### (each # being a number).
Problem is sometimes the file name is ##-##### and other times #####.
I would like a way if the file cannot be found try looking for the other two ways the file could be written.
An IOError exception happens when the file is not found. What I was thinking was to have an except statement that says:
except File2:
Look for ##### in myfindFileFunction()
if file is still not found run except File3
except File3:
Look for ##-#### in myfindFileFuction()
except:
print "File not found"
What I am not sure of is how to set up custom exception to work this way, and/or if there is a more pythonic way to do this altogether...
Would setting up a pattern or the three possible file names and iterate thought each until the file is found work better?
Using try/except is indeed a very pythonic (and fast) way of doing things.
You have to weigh not only if it's pythonic, but what impact does that approach has in terms of readability. Will you still understand the code quickly when you look at it again in 6 months? Will somebody else?
I usually make sure that slightly complex try/except clauses to handle this kind of things are well commented. Asides from that... it's a perfectly reasonable way of doing it.
Also, to put your mind at ease regarding performance, a common concern when one is deciding between two approaches, take a look here: Python if vs try-except and you'll see that try/except constructs are fast in Python... really fast.
no custom exception needed
import errno
try:
open('somefile')
except IOError as e:
if e.errno == errno.ENOENT:
open('someotherfilename')
else:
raise e
(this is on *nix- im not sure if you're using windows)
It's easy enough to define your own exceptions -- just create a class derived from Exception. The doco is clear.
However creating separate exceptions per file type, or any exception at all, doesn't seem necessary. You could do something like:
files = ('#######', "##-#####', '#####')
fh = None
for f in files:
try:
fh = open(f)
break
except IOError as e:
if e.errno in (errno.ENOENT,):
pass
else:
raise
if not fh:
## all three tries failed
The use of if around e.errno lets you decide which IO errors mean go on to the next file and which are errors you want to know about . File does not exists (errno.ENOENT) means try next file. But others like 'Too many files open' (errno.ENFILE) probably need a different response.
I have a question regarding error checking in Python. Let's say I have a function that takes a file path as an input:
def myFunction(filepath):
infile = open(filepath)
#etc etc...
One possible precondition would be that the file should exist.
There are a few possible ways to check for this precondition, and I'm just wondering what's the best way to do it.
i) Check with an if-statement:
if not os.path.exists(filepath):
raise IOException('File does not exist: %s' % filepath)
This is the way that I would usually do it, though the same IOException would be raised by Python if the file does not exist, even if I don't raise it.
ii) Use assert to check for the precondition:
assert os.path.exists(filepath), 'File does not exist: %s' % filepath
Using asserts seems to be the "standard" way of checking for pre/postconditions, so I am tempted to use these. However, it is possible that these asserts are turned off when the -o flag is used during execution, which means that this check might potentially be turned off and that seems risky.
iii) Don't handle the precondition at all
This is because if filepath does not exist, there will be an exception generated anyway and the exception message is detailed enough for user to know that the file does not exist
I'm just wondering which of the above is the standard practice that I should use for my codes.
If all you want to do is raise an exception, use option iii:
def myFunction(filepath):
with open(filepath) as infile:
pass
To handle exceptions in a special way, use a try...except block:
def myFunction(filepath):
try:
with open(filepath) as infile:
pass
except IOError:
# special handling code here
Under no circumstance is it preferable to check the existence of the file first (option i or ii) because in the time between when the check or assertion occurs and when python tries to open the file, it is possible that the file could be deleted, or altered (such as with a symlink), which can lead to bugs or a security hole.
Also, as of Python 2.6, the best practice when opening files is to use the with open(...) syntax. This guarantees that the file will be closed, even if an exception occurs inside the with-block.
In Python 2.5 you can use the with syntax if you preface your script with
from __future__ import with_statement
Definitely don't use an assert. Asserts should only fail if the code is wrong. External conditions (such as missing files) shouldn't be checked with asserts.
As others have pointed out, asserts can be turned off.
The formal semantics of assert are:
The condition may or may not be evaluated (so don't rely on side effects of the expression).
If the condition is true, execution continues.
It is undefined what happens if the condition is false.
More on this idea.
The following extends from ~unutbu's example. If the file doesn't exist, or on any other type of IO error, the filename is also passed along in the error message:
path = 'blam'
try:
with open(path) as f:
print f.read()
except IOError as exc:
raise IOError("%s: %s" % (path, exc.strerror))
=> IOError: blam: No such file or directory
I think you should go with a mix of iii) and i). If you know for a fact, that python will throw the exception (i.e. case iii), then let python do it. If there are some other preconditions (e.g. demanded by your business logic) you should throw own exceptions, maybe even derive from Exception.
Using asserts is too fragile imho, because they might be turned off.