I have a few functions that use context manager:
def f1():
with open("test.txt","r+") as f:
f.write("common Line")
f.write("f1 Line")
def f2():
with open("test.txt","r+") as f:
f.write("common Line")
f.write("f2 Line")
def f3():
with open("test.txt","r+") as f:
f.write("common Line")
f.write("f3 Line")
These functions have a few common lines. So I want to add a helper function. Something like this
def helperF():
with open("test.txt","r+") as f:
f.write("common Line")
And then somehow call it from my f1,f2,f3 functions to make the code DRY.
But I'm not quite sure how to deal with context manager in this situation. The following will not work because f is already closed by the time the function is called:
def f1():
commonHelper()
f.write("f1 Line")
def f2():
commonHelper()
f.write("f2 Line")
def f3():
commonHelper()
f.write("f3 Line")
If the three functions are each writing quite a bit to the file, I would recommend refactoring so that they return a list of strings to be written, instead of having multiple functions which write directly to the file.
def write_with_common_header(lines):
with open("test.txt", "r+") as f:
f.write("common Line")
for line in lines:
f.write(line)
def f1():
return ["f1 Line"]
def f2():
return ["f2 Line"]
def f3():
return ["f3 Line"]
# usage example:
write_with_common_header(f2())
If the lists returned by each function will always be the same, then there is no need for them to even be functions; you can just declare them as lists.
In the more general case where the context manager isn't necessarily a file, and the separate functions are doing more than just calling a single method, then we can't just pass them in as data, but the same technique can be applied: make the write_with_common_header function accept an argument so its behaviour can be parameterised. For full generality, the argument should be a function which accepts a reference to the managed resource f.
def common_helper(callback):
with open("test.txt", "r+") as f:
f.write("common Line")
callback(f)
def f1(f):
f.write("f1 Line")
def f2(f):
f.write("f2 Line")
def f3(f):
f.write("f3 Line")
# usage example:
common_helper(f2)
Related
I am learning some new topics in python, and I finally got to decorators, which seems cool and all except theres one major issue. Below is the code I have running:
def new_dec(ori_func):
def WrapFunc():
print("First Line")
ori_func()
print("Second line")
return WrapFunc
def new_func():
print("This is the new function")
ThisFunc = new_dec(new_func)
print(ThisFunc())
However, when this code is executed it outputs:
First Line
This is the new function
Second line
None
And I do not remember adding a None statement, is this possibly a type variable that has been added? Why is this happening, and how can this be fixed.
You have four print statements. So four things get printed. And by default a function returns None:
>>> def f(): pass
>>> print(f())
None
Check what happens if you define:
def new_dec(ori_func):
def WrapFunc():
print("First Line")
ori_func()
print("Second line")
return "Fourth Line"
return WrapFunc
Alternatively you can remove the fourth print:
# print(ThisFunc())
ThisFunc()
By default, a function returns None. You just have to call the function to get what you expect.
def new_dec(ori_func):
def WrapFunc():
print("First Line")
ori_func()
print("Second line")
return WrapFunc
def new_func():
print("This is the new function")
ThisFunc = new_dec(new_func)
ThisFunc() # get rid of the print statement
I am writing python API and I have one problem.
I have 3 different functions:
func1() -> return only text
func2(name) -> return text only but takes parameter
func3(name) -> this function create a file "name".txt
Now I have a problem with decorator, I want to create a log decorator that is called everytime function is called.
Problem is that I dont know how to simply do it, I know how to create it with no param or one param but I have no idea hot to create universal decorator that will work for all three functions.
Now i have something like this:
def log(func):
def wrapper(name):
func(name)
log = ('write something here')
f = open('log.txt', 'a+')
f.write(log + "\n")
f.close(name)
return wrapper
Your wrapper should accept an arbitrary number of arguments, with the *args and **kwargs syntax to capture both positional and keyword arguments. Make sure to return whatever the wrapped function returns:
def log(func):
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
log = ('write something here')
with open('log.txt', 'a+') as f:
f.write(log + "\n")
return result
return wrapper
You probably want to add in the #functools.wraps decorator; this copies across any documentation and other metadata from the original wrapped function to the new wrapper:
from functools import wraps
def log(func):
#wraps(func)
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
log = ('write something here')
with open('log.txt', 'a+') as f:
f.write(log + "\n")
return result
return wrapper
Last but not least, rather than reopening a log file yourself, take a look at the logging module to handle log files for you.
def log(func):
def wrapper(*args, **kwds):
log = func(*args, **kwds)
f = open('log.txt', 'a+')
f.write(log + "\n")
f.close()
return wrapper
#log
def func1():
return "Called function 1"
#log
def func2(name):
return "Called function 2 with " + name
#log
def func3(name):
f = open('name.txt', 'a+')
f.write(name + " from func3\n")
f.close()
return "Called function 3 with " + name
def main():
func1()
func2("func2")
func3("func3")
if __name__ == '__main__':
main()
Log.txt becomes:
Called function 1
Called function 2 with func2
Called function 3 with func3
I am newbie in Python.
I wonder if it is possible that all functions inherit the same line of code?
with open(filename, 'r') as f: as this line of code is the same in all three functions. Is it possible to inherit the code without using classes?
I tried to find the answer on stackoverflow and python documentation, but with no luck.
def word_count(filename):
with open(filename, 'r') as f:
return len(f.read().split())
def line_count(filename):
with open(filename, 'r') as f:
return len(f.read().splitlines())
def character_count(filename):
with open(filename, 'r') as f:
return len(f.read())
The common code in your case is
with open(filename, 'r') as f:
contents = f.read()
So just move it to its own function:
def get_file_contents(filename):
with open(filename, 'r') as f:
return f.read()
def word_count(filename):
return len(get_file_contents(filename).split())
def line_count(filename):
return len(get_file_contents(filename).splitlines())
def character_count(filename):
return len(get_file_contents(filename))
What I've done in the past is split the code out into another function, in your example
with open(filename, 'r') as f:
f.read()
Is common within all of your methods, so I'd look at rewriting it like so.
def read_file(filename):
with open(filename, 'r') as f:
return f.read()
def word_count(filename):
return len(read_file(filename).split())
def line_count(filename):
return len(read_file(filename).splitlines())
def character_count(filename):
return len(read_file(filename))
I would use a class:
class Count:
""" Object holds everything count-related """
def __init__(self, filename):
""" specify filename in class instance """
with open(filename, 'r') as f:
self.content = f.read()
def word_count(self):
return len(self.content.split())
def line_count(self):
return len(self.content.splitlines())
def character_count(self):
return len(self.content)
file = Count("whatever.txt")
print(file.word_count())
print(file.line_count())
print(file.character_count())
What you do differently is after you open the file, so if I were in your shoes, I would write a function which takes another function that is executed after the file is opened.
Let's illustrate this in an example:
>>> def operate_file(filename, func):
... with open(filename, 'r') as f:
... return func(f)
>>> def line_count(f):
... return len(f.read().splitlines())
>>> def word_count(f):
... return len(f.read().split())
>>> def character_count(f):
... return len(f.read())
>>> print operate_file('/tmp/file.txt', line_count)
1200
>>> print operate_file('/tmp/file.txt', word_count)
2800
>>> print operate_file('/tmp/file.txt', character_count)
29750
I would recommend decorators. It's sort of like the making the repeated line of code into a function, but since you are going to call that function on each input anyway, decorators can let you just write the functions as id f was the input.
The #open_file is a shorthand for word_count=open_file(word_count).
here is a good place to read more about python decorators.
def open_file(func):
def wrapped_func(filename):
with open(filename, 'r') as f:
return func(f)
return wrapped_func
#open_file
def word_count(f):
return len(f.read().split())
#open_file
def line_count(f):
return len(f.read().splitlines())
#open_file
def character_count(f):
return len(f.read())
It depends on, what you want to do with the results of your 3 functions. Every function is opening the same file. That happens 3 times just to get 3 different properties.
One good solution would be a class. But another would be to rearange your functions to just one. That could return a dictionary or named tuple with the results.
It would look something like this:
def file_count(filename):
with open(filename, 'r') as f:
content = f.read()
properties = {}
properties['words'] = len(content.split())
properties['lines'] = len(content.splitlines())
properties['chars'] = len(content)
return properties
I have the following code:
def function_reader(path):
line_no = 0
with open(path, "r") as myfile:
def readline():
line_no +=1
return myfile.readline()
Python keeps returning:
UnboundLocalError: local variable 'line_no' referenced before assignment
when executing line_no +=1.
I understand that the problem is that nested function declarations have weird scoping in python (though I do not understand why it was programmed this way). I'm mostly wondering if there is a simple way to help python resolve the reference, since I really like the functionality this would provide.
Unfortunately, there is not a way to do this in Python 2.x. Nested functions can only read names in the enclosing function, not reassign them.
One workaround would be to make line_no a list and then alter its single item:
def function_reader(path):
line_no = [0]
with open(path, "r") as myfile:
def readline():
line_no[0] += 1
return myfile.readline()
You would then access the line number via line_no[0]. Below is a demonstration:
>>> def outer():
... data = [0]
... def inner():
... data[0] += 1
... inner()
... return data[0]
...
>>> outer()
1
>>>
This solution works because we are not reassigning the name line_no, only mutating the object that it references.
Note that in Python 3.x, this problem would be easily solved using the nonlocal statement:
def function_reader(path):
line_no = 0
with open(path, "r") as myfile:
def readline():
nonlocal line_no
line_no += 1
return myfile.readline()
It's hard to say what you're trying to achieve here by using closures. But the problem is that with this approach either you'll end with an ValueError: I/O operation on closed file when you return readline from the outer function or just the first line if you return readline() from the outer function.
If all you wanted to do is call readline() repeatedly or loop over the file and also remember the current line number then better use a class:
class FileReader(object):
def __init__(self, path):
self.line_no = 0
self.file = open(path)
def __enter__(self):
return self
def __iter__(self):
return self
def next(self):
line = next(self.file)
self.line_no += 1
return line
def readline(self):
return next(self)
def __exit__(self, *args):
self.file.close()
Usage:
with FileReader('file.txt') as f:
print next(f)
print next(f)
print f.readline()
print f.line_no # prints 3
for _ in xrange(3):
print f.readline()
print f.line_no # prints 6
for line in f:
print line
break
print f.line_no # prints 7
The more Pythonic way to get the next line and keep track of the line number is with the enumerate builtin:
with open(path, "r") as my file:
for no, line in enumerate(myfile, start=1):
# process line
This will work in all current Python versions.
I'm writing a Python generator which looks like "cat". My specific use case is for a "grep like" operation. I want it to be able to break out of the generator if a condition is met:
summary={}
for fn in cat("filelist.dat"):
for line in cat(fn):
if line.startswith("FOO"):
summary[fn] = line
break
So when break happens, I need the cat() generator to finish and close the file handle to fn.
I have to read 100k files with 30 GB of total data, and the FOO keyword happens in the header region, so it is important in this case that the cat() function stops reading the file ASAP.
There are other ways I can solve this problem, but I'm still interested to know how to get an early exit from a generator which has open file handles. Perhaps Python cleans them up right away and closes them when the generator is garbage collected?
Thanks,
Ian
Generators have a close method that raises GeneratorExit at the yield statement. If you specifically catch this exception, you can run some tear-down code:
import contextlib
with contextlib.closing( cat( fn ) ):
...
and then in cat:
try:
...
except GeneratorExit:
# close the file
If you'd like a simpler way to do this (without using the arcane close method on generators), just make cat take a file-like object instead of a string to open, and handle the file IO yourself:
for filename in filenames:
with open( filename ) as theFile:
for line in cat( theFile ):
...
However, you basically don't need to worry about any of this, because the garbage collection will handle it all. Still,
explicit is better than implicit
By implementing the context protocol and the iterator protocol in the same object, you can write pretty sweet code like this:
with cat("/etc/passwd") as lines:
for line in lines:
if "mail" in line:
print line.strip()
break
This is a sample implementation, tested with Python 2.5 on a Linux box. It reads the lines of /etc/passwd until it finds the one for user audio, and then stops:
from __future__ import with_statement
class cat(object):
def __init__(self, fname):
self.fname = fname
def __enter__(self):
print "[Opening file %s]" % (self.fname,)
self.file_obj = open(self.fname, "rt")
return self
def __exit__(self, *exc_info):
print "[Closing file %s]" % (self.fname,)
self.file_obj.close()
def __iter__(self):
return self
def next(self):
line = self.file_obj.next().strip()
print "[Read: %s]" % (line,)
return line
def main():
with cat("/etc/passwd") as lines:
for line in lines:
if "mail" in line:
print line.strip()
break
if __name__ == "__main__":
import sys
sys.exit(main())
Or even simpler:
with open("/etc/passwd", "rt") as f:
for line in f:
if "mail" in line:
break
File objects implement the iterator protocol (see http://docs.python.org/library/stdtypes.html#file-objects)
Please also consider this example:
def itertest():
try:
for i in xrange(1000):
print i
yield i
finally:
print 'finally'
x = itertest()
for i in x:
if i > 2:
break
print 'del x'
del x
print 'exit'
0
1
2
3
del x
finally
exit
It shows that finally is run after the iterator is cleaned up. I think __del__(self) is calling self.close(), see also here: https://docs.python.org/2.7/reference/expressions.html#generator.close
There seems to be another possibility using try..finally (tested on Python 2.7.6):
def gen():
i = 0
try:
while True:
print 'yield %i' % i
yield i
i += 1
print 'will never get here'
finally:
print 'done'
for i in gen():
if i > 1:
print 'break'
break
print i
Gives me the following printout:
yield 0
0
yield 1
1
yield 2
break
done