I'm running code from homework assignments with open in them. Trouble is, the students were told not to submit the data they were given and assume we have it - and open doesn't look in sys.path.
Luckily, I am using Spyder, which allows me to choose a script to be executed when initializing a console. I figured I could override open, so I defined a new open function which calls the original open on the absolute path of the files. But when someone uses with open(...) as ..., it doesn't work.
I know this may not to a good thing to do, but I can't go over every file in every submitted assignment looking for and replacing the call to open...
My code is:
old_open = open
def open(*args, **kwrdargs):
try:
res = old_open(*args,**kwrdargs)
return res
except:
args= list(args)
if ('DS1' in args[0]):
args[0]=DS1
elif ('DS2_X' in args[0]):
args[0] = DS2_X
elif ('DS2_Y' in args[0]):
args[0] = DS2_Y
args = tuple(args)
res = old_open(*args,**kwrdargs)
return res
DS1,DS2_X, DS2_Y contain the absolute path to the files.
When executing:
with open('DS1.data', 'r') as f:
I get the error:
FileNotFoundError: [Errno 2] No such file or directory: 'DS1.data'
while using:
f=open('DS1.data','r')
works.
I debugged the code, and when using with, my open is not called, but when using f=open(...), it is. Why is this happening?
open is supposed to return a file-like object, it is that object (ie f in the below example) that is supposed to have an __enter__ and __exit__. For example you could write your with-statement as:
f = open(...)
with f as ...:
do_something()
If you don't return the object returned by open, but some own wrapper around the file object you have to wrap these too. But from your description it more look like you don't need that, but rather that you've somewhere didn't return a file. Your open should look something like:
def open(fname, *args, **kwds):
for p in sys.path:
fn = build_filename(p, fname)
try:
return _orig_open(fn, *args, **kwds)
except IOerror as e:
pass
return _orig_open(fn, *args, **kwds) # must return file or rais exception
Related
I would like to write a decorator that closes all open files. For example:
#close_files
def view(request):
f = open('myfile.txt').read()
return render('template.html')
Ignoring any threadsafe stuff, how could I write such a decorator to close out any open files after the function is returned? I'm not interested in writing a context manager, but something like this:
def close_files(func):
#wraps(func)
def wrapper_close_files(*args):
return_value = func(*args, **kwargs)
# close open files here?
return return_value
return wrapper_close_files
Unfortunately I cannot use something like this:
with open('myfile.txt') as _f: f = _f.read()
I'm asking about how to do a decorator to close files where we do not have direct access to the variable which references the file (handler).
According to python docs 3.6:
If you’re not using the with keyword, then you should call f.close() to close the file and immediately free up any system resources used by it. If you don’t explicitly close a file, Python’s garbage collector will eventually destroy the object and close the open file for you, but the file may stay open for a while. Another risk is that different Python implementations will do this clean-up at different times.
So if you want to do this, you need a decorator who captures the open command from the builtins module. However, since we don't know how many files will be opened, we can use ExitStack. Thus, you can use a function that keeps the file handle within the ExitStack context.
My version for that is:
def close_opened_files(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
import builtins
_open = getattr(builtins, 'open')
try:
with contextlib.ExitStack() as stack:
def myopen(*args, **kwargs):
f = stack.enter_context(_open(*args, **kwargs))
return f
setattr(builtins, 'open', myopen)
ret = func(*args, **kwargs)
return ret
finally:
setattr(builtins, 'open', _open)
return wrapper
At the end of the decorator, the original open function should be restored in the builtins module.
One of the problems with this type of approach is: the functions that are called inside func and that use the open function, will also have their files closed at the end of decorator, even if it represents an error.
Here are some examples:
The following examples show read and write operations, also with files being closed within the function.
#close_opened_files
def test_write(fn1, fn2=None):
f1 = open(fn1, 'w')
print('Hello world', file=f1)
if fn2:
f2 = open(fn2, 'w')
print('foo-bar', file=f2)
f2.close()
#close_opened_files
def test_read(fn1, fn2=None):
f1 = open(fn1, 'r')
for line in f1: print(line)
if fn2:
f2 = open(fn2, 'r')
for line in f2: print(line)
f2.close()
test_write('file1.txt', 'file2.txt')
test_read('file1.txt', 'file2.txt')
try: test_read('non_exist_filename.txt')
except FileNotFoundError as ex: print(ex)
try: test_exception('file1.txt')
except RuntimeError as ex: print(ex)
These last examples show the decorator under the exceptions: 'no file exists', or any other exception that happens.
The last case, file will be closed before the exception is raised.
I believe it would be possible to use ast to traverse the code for file openings, and attempt to close any files that were found to have been opened.
I have a huge code base before me, and I have a place where a file with name "foobar" gets written.
I have no clue where this file gets read.
My idea how to solve this:
do monkey patching or mocking. An exceptions should get raised if a file with this name gets opened.
run all tests and see where the exception gets raised.
How to let the interpreter raise an exception if a file with given name gets opened?
I am sure that the place I search is pure python, not a c-extension.
I use Python 2.7
You can override (shadow) builtin open function. Add this in your main module:
import __builtin__
open_file = __builtin__.open
def fake_open(filename, *args, **kwargs):
if filename == 'foobar':
raise Exception('foobar filename')
else:
return open_file(filename, *args, **kwargs)
__builtin__.open = fake_open
I'd like to write a function similar to open. I'd like to be able to call it with with, but also without with.
When I use contextlib.contextmanager, it makes my function work fine with with:
#contextmanager
def versioned(file_path, mode):
version = calculate_version(file_path, mode)
versioned_file = open(file_path, mode)
yield versioned_file
versioned_file.close()
So, I use it like this:
with versioned('file.txt', 'r') as versioned_file:
versioned_file.write(...)
How do I use it without with:
versioned_file = versioned('file.txt', 'r')
versioned_file.write(...)
versioned_file.close()
It complains:
AttributeError: 'GeneratorContextManager' object has no attribute 'write'
The problem is that contextmanager only provides exactly that; a context manager to be used in the with statement. Calling the function does not return the file object, but a special context generator which provides the __enter__ and __exit__ functions. If you want both the with statement and “normal” assignments to work, then you will have to have some object as the return value from your function that is fully usable and also provides the context functions.
You can do this pretty easily by creating your own type, and manually providing the context functions:
class MyOpener:
def __init__ (self, filename):
print('Opening {}'.format(filename))
def close (self):
print('Closing file.')
def write (self, text):
print('Writing "{}"'.format(text))
def __enter__ (self):
return self
def __exit__ (self, exc_type, exc_value, traceback):
self.close()
>>> f = MyOpener('file')
Opening file
>>> f.write('foo')
Writing "foo"
>>> f.close()
Closing file.
>>> with MyOpener('file') as f:
f.write('foo')
Opening file
Writing "foo"
Closing file.
You have this:
#contextmanager
def versioned(file_path, mode):
# some setup code
yield versioned_file
# some teardown code
Your basic problem of course is that what you yield from the context manager comes out of the with statement via as, but is not the object returned by your function. You want a function that returns something that behaves like the object open() returns. That is to say, a context manager object that yields itself.
Whether you can do that depends what you can do with the type of versioned_file. If you can't change it then you're basically out of luck. If you can change it then you need to implement the __enter__ and __exit__ functions as specified in PEP 343.
In your example code, though, it already has it, and your teardown code is the same as what it does itself on context exit already. So don't bother with contextlib at all, just return the result of open().
For other examples where you do need __enter__ and __exit__, if you like the contextlib style (and who doesn't?) you can bridge the two things. Write a function context that's decorated with #contextmanager and yields self. Then implement:
def __enter__(self):
self.context = context() # if context() is a method use a different name!
return self.context.__enter__()
def __exit__(self, *args):
return self.context.__exit__(*args)
It's basically up to you whether you find this better or worse than separating out the setup code into __enter__ and the teardown code into __exit__. I generally find it better.
Do you really need to use contextlib.contextmanager?
If you have a custom stream you would want to use Poke's solution.
But since you are just returning a file object, why go through all the hassle:
def versioned(file_path, mode):
version = calculate_version(file_path, mode)
return open(file_path, mode)
with versioned('test.conf', 'r') as stream:
print stream.read()
f = versioned('test.conf', 'r')
print f.read()
f.close()
Both will work perfectly fine :)
Let's say I have a list of the opened files (actually, of the file numbers):
import resource
import fcntl
def get_open_fds():
fds = []
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
for fd in range(3, soft):
try:
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
except IOError:
continue
fds.append(fd)
return fds
Now I would like to get the names of those files. How can I do this?
EDIT
Just to clarify, for those downvoting this: fd is an integer. It is NOT a filedescriptor. Sorry for confusing you with the name, but the code is self-explanatory.
EDIT2
I am getting flamed about this, I think because of my choice of fd to mean file number. I just checked the documentation:
All functions in this module take a file descriptor fd as their first
argument. This can be an integer file descriptor, such as returned by
sys.stdin.fileno(), or a file object, such as sys.stdin itself, which
provides a fileno() which returns a genuine file descriptor.
So fd is indeed an integer. It can also be a file object but, in the general case, fd has not .name.
As per this answer:
for fd in get_open_fds():
print fd, os.readlink('/proc/self/fd/%d' % fd)
I was in the same boat. What I ended up doing is writing my own open that keeps track of all the files opened. Then in the initial Python file the first thing that happens is the built-in open gets replaced by mine, and then later I can query it for the currently open files. This is what it looks like:
class Open(object):
builtin_open = open
_cache = {}
#classmethod
def __call__(cls, name, *args):
file = cls.builtin_open(name, *args)
cls._cache[name] = file
return file
#classmethod
def active(cls, name):
cls.open_files()
try:
return cls._cache[name]
except KeyError:
raise ValueError('%s has been closed' % name)
#classmethod
def open_files(cls):
closed = []
for name, file in cls._cache.items():
if file.closed:
closed.append(name)
for name in closed:
cls._cache.pop(name)
return cls._cache.items()
import __builtin__
__builtin__.open = Open()
then later...
daemon.files_preserve = [open.active('/dev/urandom')]
I want to wrap the default open method with a wrapper that should also catch exceptions. Here's a test example that works:
truemethod = open
def fn(*args, **kwargs):
try:
return truemethod(*args, **kwargs)
except (IOError, OSError):
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
open = fn
I want to make a generic method of it:
def wrap(method, exceptions = (OSError, IOError)):
truemethod = method
def fn(*args, **kwargs):
try:
return truemethod(*args, **kwargs)
except exceptions:
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
method = fn
But it doesn't work:
>>> wrap(open)
>>> open
<built-in function open>
Apparently, method is a copy of the parameter, not a reference as I expected. Any pythonic workaround?
The problem with your code is that inside wrap, your method = fn statement is simply changing the local value of method, it isn't changing the larger value of open. You'll have to assign to those names yourself:
def wrap(method, exceptions = (OSError, IOError)):
def fn(*args, **kwargs):
try:
return method(*args, **kwargs)
except exceptions:
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
return fn
open = wrap(open)
foo = wrap(foo)
Try adding global open. In the general case, you might want to look at this section of the manual:
This module provides direct access to all ‘built-in’ identifiers of Python; for example, __builtin__.open is the full name for the built-in function open(). See chapter Built-in Objects.
This module is not normally accessed explicitly by most applications, but can be useful in modules that provide objects with the same name as a built-in value, but in which the built-in of that name is also needed. For example, in a module that wants to implement an open() function that wraps the built-in open(), this module can be used directly:
import __builtin__
def open(path):
f = __builtin__.open(path, 'r')
return UpperCaser(f)
class UpperCaser:
'''Wrapper around a file that converts output to upper-case.'''
def __init__(self, f):
self._f = f
def read(self, count=-1):
return self._f.read(count).upper()
# ...
CPython implementation detail: Most modules have the name __builtins__ (note the 's') made available as part of their globals. The value of __builtins__ is normally either this module or the value of this modules’s __dict__ attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python.
you can just add return fn at the end of your wrap function and then do:
>>> open = wrap(open)
>>> open('bhla')
Traceback (most recent call last):
File "<pyshell#24>", line 1, in <module>
open('bhla')
File "<pyshell#18>", line 7, in fn
sys.exit('Can\'t open \'{0}\'. Error #{1[0]}: {1[1]}'.format(args[0], sys.exc_info()[1].args))
SystemExit: Can't open 'bhla'. Error #2: No such file or directory