I've seen it many times but never understood what the as command does in Python 3.x. Can you explain it in plain English?
It's not a command per se, it's a keyword used as part of the with statement:
with open("myfile.txt") as f:
text = f.read()
The object after as gets assigned the result of the expression handled by the with context manager.
Another use is to rename an imported module:
import numpy as np
so you can use the name np instead of numpy from now on.
The third use is to give you access to an Exception object:
try:
f = open("foo")
except IOError as exc:
# Now you can access the Exception for more detailed analysis
It is a keyword used for object naming in several cases.
from some_module import something as some_alias
# `some_alias` is `some_module.something`
with open("filename") as f:
# `f` is the file object `open("filename")` returned
try:
Nonsense!
except Exception as e:
# `e` is the Exception thrown
Related
I have a function (not mine) which accepts a filename as input, then does things with the file content.
On the other hand, I receive the data which is the "file content" above from somewhere, not as a file.
To illustrate the case, consider the code below.
myfunc() is something I do not control
the expectation of the authors of myfunc() is that the data to process will be in a file ...
... but I have it as a string
# a function (which I do not control) which uses a filename as input
def myfunc(filename):
with open(filename) as f:
print(f.read())
# case 1: I have a real file
# creation of the file for the sake of the example.
# The authors of myfunc() expect that the the file is just there.
with open("afile.txt", "w") as f:
f.write("hello")
# normal usage of myfunc()
myfunc("afile.txt")
# case 2: I have the contents of the file and need to use myfunc()
# data below is received from somewhere, as a string
data = "world"
# I would like to pass it to myfunc
# ...
How to pass <something> as the argument of the function so that <something> holds the content of data but behaves as a file name?
The obvious solution is to dump data into a file and pass the filename to myfunc(). I would like to avoid this intermediate step to keep the whole thing in memory.
This is a bit convoluted, so bear with me... but if you absolutely want to avoid writing to disk you could shadow the builtin open() function with your own...
from io import StringIO
from contextlib import contextmanager
#contextmanager
def myfunc_open(*args, **kwargs):
resource = StringIO(args[0])
try:
yield resource
finally:
resource.close()
We'll use StringIO since it has the IO functions you'd expect within the with open() context. Depending on the context you might want to use BytesIO instead.
Then, before you call upon myfunc simply back up open() and replace with your own:
def myfunc(filepath):
with open(filepath, 'r') as file:
print(file.read())
try:
myfunc('This is doomed to fail')
except:
print("Normal myfunc won't work")
_backup_open = open # back up before you shadow it!
open = myfunc_open
myfunc('This will however be read')
open = _backup_open # Perfectly balanced, as all things should be.
Result:
# Normal myfunc won't work
# This will however be read
Note: If myfunc is imported from another module, it'll require a little bit of work around:
import myfunc_module
myfunc_module.open = myfunc_open
myfunc_module.myfunc('This works too')
myfunc_module.open = _backup_open
Note you do have to import the whole module instead of from module import func so you can shadow the open within its namespace. Sample here.
This may be the evilest stroke of genius I have so far in my hobbyist journey...
if you send data as an argument in myfunc() then it will give FileNotFoundError. if this file doesn't exist then that must mean it is the file content that you are trying to send as the argument. so, if use exception handling then this may do the trick
try :
myfunc(data)
except FileNotFoundError:
print(data)
Right now, I catch the exception in the except Exception: clause, and do print(exception). The result provides no information since it always prints <class 'Exception'>. I knew this used to work in python 2, but how do I do it in python3?
I'm guessing that you need to assign the Exception to a variable. As shown in the Python 3 tutorial:
def fails():
x = 1 / 0
try:
fails()
except Exception as ex:
print(ex)
To give a brief explanation, as is a pseudo-assignment keyword used in certain compound statements to assign or alias the preceding statement to a variable.
In this case, as assigns the caught exception to a variable allowing for information about the exception to stored and used later, instead of needing to be dealt with immediately.
(This is discussed in detail in the Python 3 Language Reference: The try Statement.)
There are other compound statements that use as. The first is the with statement:
#contextmanager
def opening(filename):
f = open(filename)
try:
yield f
finally:
f.close()
with opening(filename) as f:
# ...read data from f...
Here, with statements are used to wrap the execution of a block with methods defined by context managers. This functions like an extended try...except...finally statement in a neat generator package, and the as statement assigns the generator-produced result from the context manager to a variable for extended use.
(This is discussed in detail in the Python 3 Language Reference: The with Statement.)
As of Python 3.10, match statements also use as:
from random import randint
match randint(0, 2):
case 0|1 as low:
print(f"{low} is a low number")
case _:
print("not a low number")
match statements take an expression (in this case, randint(0, 2)) and compare its value to each case branch one at a time until one of them succeeds, at which point it executes that branch's block. In a case branch, as can be used to assign the value of the branch to a variable if that branch succeeds. If it doesn't succeed, it is not bound.
(The match statement is covered by the tutorial and discussed in detail in the Python 3 Language Reference: match Statements.)
Finally, as can be used when importing modules, to alias a module to a different (usually shorter) name:
import foo.bar.baz as fbb
This is discussed in detail in the Python 3 Language Reference: The import Statement.
These are the changes since python 2:
try:
1 / 0
except Exception as e: # (as opposed to except Exception, e:)
# ^ that will just look for two classes, Exception and e
# for the repr
print(repr(e))
# for just the message, or str(e), since print calls str under the hood
print(e)
# the arguments that the exception has been called with.
# the first one is usually the message. (OSError is different, though)
print(e.args)
You can look into the standard library module traceback for fancier stuff.
Try
try:
print(undefined_var)
except Exception as e:
print(e)
this will print the representation given by e.__str__():
"name 'undefined_var' is not defined"
you can also use:
print(repr(e))
which will include the Exception class name:
"NameError("name 'undefined_var' is not defined",)"
Here is the way I like that prints out all of the error stack.
import logging
try:
1 / 0
except Exception as _e:
# any one of the follows:
# print(logging.traceback.format_exc())
logging.error(logging.traceback.format_exc())
The output looks as the follows:
ERROR:root:Traceback (most recent call last):
File "/PATH-TO-YOUR/filename.py", line 4, in <module>
1 / 0
ZeroDivisionError: division by zero
LOGGING_FORMAT :
LOGGING_FORMAT = '%(asctime)s\n File "%(pathname)s", line %(lineno)d\n %(levelname)s [%(message)s]'
Although if you want a code that is compatible with both python2 and python3 you can use this:
import logging
try:
1/0
except Exception as e:
if hasattr(e, 'message'):
logging.warning('python2')
logging.error(e.message)
else:
logging.warning('python3')
logging.error(e)
[In Python3]
Let's say you want to handle an IndexError and print the traceback, you can do the following:
from traceback import print_tb
empty_list = []
try:
x = empty_list[100]
except IndexError as index_error:
print_tb(index_error.__traceback__)
Note: You can use the format_tb function instead of print_tb to get the traceback as a string for logging purposes.
Hope this helps.
Don't use print(e), since that won't print a stack trace, which is a nightmare for debugging. traceback.print_exception is what you're looking for:
import traceback
try:
assert False
except Exception as e:
traceback.print_exception(e)
I've use this :
except (socket.timeout, KeyboardInterrupt) as e:
logging.debug("Exception : {}".format(str(e.__str__).split(" ")[3]))
break
Let me know if it does not work for you !!
You can do:
with self.assertRaisesMessage(ValueError, 'invalid literal for int()'):
int('a')
Reference
I have a program that reads yaml configs, just read only. I am wondering which one of the following is more Pythonic
try:
config = yaml.load(open(filepath))
except Exception as error:
print error
vs using a with statement
try:
with open(filepath) as f:
config = yaml.load(f)
except Exception as error:
print error
I prefer the first one cause its simpler to read and since there are no write I don't think there will be issues with file closing gracefully. Thoughts?
Use the second. From the documentation:
It is good practice to use the with keyword when dealing with file
objects. This has the advantage that the file is properly closed after
its suite finishes, even if an exception is raised on the way.
I have tried the following which doesn't work:
try:
f = h5py.File(filepath)
except NameError:
f = scipy.io.loadmat(filepath)
Basically, a user will pass a particular input(a filepath) to a function which is
supposed to load the data in that file. But, I don't expect the user to know whether the function is defined for that input.
I'm getting the following error:
OSError: Unable to create file (Unable to open file: name = '/users/cyrilrocke/documents/c_elegans/data/chemotaxis/n2laura_benzaldehyde_l_2013_03_17__15_39_19___2____features.mat', errno = 17, error message = 'file exists', flags = 15, o_flags = a02)
Note: basically, I want to be able to switch between the function h5py.File() and scipy.io.loadmat() depending on which one doesn't work. Given the inputs, one of these functions must work.
I guess you want something like that:
flag = True;
try:
f = h5py.File(filepath)
except:
flag = False;
try:
if not flag:
f = scipy.io.loadmat(filepath)
except:
print('Error')
The problems in your code are:
you are catching for the wrong error. Without specifying the error type you can handle all kinds of error in a single way (you can also add multiple except clauses)
you are executing a potentially failing function inside the except clause. In this way you won't be able to handle new errors. I then splitted them in two separate try/except blocks.
First: your code should know whether a function is defined! That's not something external, it is something that is entirely in your power.
If an import might fail, maybe do something like
try:
import h5py
HAVE_H5PY = True
except ImportError:
HAVE_H5PY = False
and then use that to check.
Second, you could instead do something like
if 'h5py' in globals():
but that is ugly and unnecessary (you have to catch the ImportError anyway, if that is the problem you're trying to solve).
Third, your error message has nothing to do with any of this, it is returned by one of those two commands. It is trying to create a file that already exists.
error message = 'file exists'
Your program is getting EEXIST and if you see a difference between trials with h5py and scipy.io, it's because they have different flags that they're passing to open() and perhaps differing states if you cleanup the files created between trials.
From the h5py.File docs:
Valid modes are:
...
r Readonly, file must exist
w Create file, truncate if exists
w- or x Create file, fail if exists
a Read/write if exists, create otherwise (default)
Since you're looking to read the file (only?) you should specify 'r' as the mode.
If you're experiencing this problem only rarely, then perhaps it's a race generated by a TOCTOU-style error in h5py.
Indeed, such a TOCTOU design error exists in h5py.
https://github.com/h5py/h5py/blob/c699e741de64fda8ac49081e9ee1be43f8eae822/h5py/_hl/files.py#L99
# Open in append mode (read/write).
# If that fails, create a new file only if it won't clobber an
# existing one (ACC_EXCL)
try:
fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
except IOError:
fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
This is the reason for the O_EXCL flag used by posix-style OSs. h5f.ACC_EXCL likely does the same thing, but if it's only used upon failure of h5f.ACC_RDWR then its atomicity is diminished.
I can check python files or modules for error via some libraries, like: pylint, pychecker, pyflakes, etc. In most case, I must specify file or directory for checking. For example:
pylint directory/mymodule.py
It's ok, but not enough for me. I want analyze separated code block and get all detected errors and warnings. So, I must call a python code analyzer from my own module as a part of program.
import some_magic_checker
code = '''import datetime; print(datime.datime.now)'''
errors = some_magic_checker(code)
if len(errors) == 0:
exec(code) # code has no errors, I can execute its
else:
print(errors) # display info about detected errors
Does exist some python library like pylint or pyflakes, which provide functionality for python code checking without code compile? Thanks for your help.
UPD
I will try to explain what i mean in the simple example. I have one variable "codeString", which contains python source code. I must analyze this code (without any file creation and code execution, but I can compile code) and detect all warnings about incorrect blocks of a code. Let's look inside the pyflakes module and understand how it works.
There is a function "check" in module "pyflakes.api".
from pyflakes import checker
from pyflakes import reporter as modReporter
import _ast
import sys
def check(codeString, filename):
reporter = modReporter._makeDefaultReporter()
try:
tree = compile(codeString, filename, "exec", _ast.PyCF_ONLY_AST)
except SyntaxError:
value = sys.exc_info()[1]
msg = value.args[0]
(lineno, offset, text) = value.lineno, value.offset, value.text
# If there's an encoding problem with the file, the text is None.
if text is None:
# Avoid using msg, since for the only known case, it contains a
# bogus message that claims the encoding the file declared was
# unknown.
reporter.unexpectedError(filename, 'problem decoding source')
else:
reporter.syntaxError(filename, msg, lineno, offset, text)
return 1
except Exception:
reporter.unexpectedError(filename, 'problem decoding source')
return 1
# Okay, it's syntactically valid. Now check it.
w = checker.Checker(tree, filename)
w.messages.sort(key=lambda m: m.lineno)
for warning in w.messages:
reporter.flake(warning)
return len(w.messages)
How you can see, this function cannot work only with one parameter "codeString", we must also provide second parameter "filename". And this is my biggest problem, I don't have any file, just Python code in string variable.
pylint, pychecker, pyflakes and all libraries, what i know, works only with created files. So i try to find some solutions, which don't require link to Python file.
Built-in function "compile" allow to create compiled code or AST object and catch some errors without file creation and code execution. We must pass '<string>' value as a second parameter, because code wasn’t read from a file.
>>> def execute(code_string):
>>> output = list()
>>> try:
>>> tree = compile(code_string, '<string>', 'exec')
>>> except Exception as e:
>>> print(e)
>>> else:
>>> exec(tree)
>>> # Now I can check code before calling the "exec" function
>>> code_string = 'print("Hello_World!")'
>>> execute(code_string) # All is ok
Hello_World!
>>> code_string = 'ERROR! print("Hello_World!")'
>>> execute(code_string) # This code will not executed
invalid syntax (<string>, line 1)