I want to log the script output to a file while still displaying the output to the screen.
It works fine, except for some cases where not all the content is written to the file (one or two lines can be missed, if the output is long)
Below is my code:
class Tee(object):
def __init__(self, *files):
self.files = files
def write(self, obj):
for f in self.files:
f.write(obj)
f.flush()
write_log = open("log.txt", 'a', 0)
sys.stdout = Tee(sys.stdout, write_log)
sys.stderr = Tee(sys.stderr, write_log)
Tried all the following options at the end of the code, but the result is the same:
os.fsync(write_log.fileno())
write_log.flush()
write_log.close()
Try using the with statement or use try-except and explicitly close the file.
Related
I don't know if there is an easy way of doing this that doesn't rely on manually writing down what the saved outputs from a script are so open to any suggestions.
I want a function that runs at the end of my script and that automatically generates a text file with a name like:
"IO_track_scriptname_date_time"
Which has a list of the files I loaded and the files I saved (location links).
And then saves this txt file to the desired destination.
Thank you for your help
Edit:
Or any alternative way of keeping a log of inputs and outputs.
Here is a thin object wrapper around the open function that tracks all of the files that are opened.
class Open:
_open = open
def __init__(self):
self.opened_files = []
self.fp = None
def __call__(self,
file,
mode='r',
buffering=-1,
encoding=None,
errors=None,
newline=None,
closefd=True,
opener=None):
self.fp = self._open(file, mode, buffering, encoding, errors,
newline, closefd, opener)
self.opened_files.append((mode, file))
return self.fp
def __enter__(self, *args, **kwargs):
return self.__call__(*args, **kwargs)
def __exit__(self, *exc_details):
return self.fp.close()
def __getattr__(self, attr):
return getattr(self.fp, attr)
def export(self, filename):
with open(filename, 'w') as fp:
for m, fn in self.opened_files:
fp.write(f'({m}): {fn}\n')
To actually use it, you will need to overwrite the built-in open function with an instantiation of this class. If you have one file that you are calling, you can pop this into the __main__ block. i.e.
...
if __name__=='__main__':
# code defining Open class here
...
open = Open()
# other code in __main__ here
open.export("IO_track_scriptname_date_time.txt")
I'm using pandas to load a csv file that has few bad lines. This means that in few lines there are some extra commas and that is why pandas is not able to load it. Which is fine by me. I'm using error_bad_lines=False to ignore those lines. When those bad lines are ignored by pandas, it shows a message like this on console:
b'Skipping line 3: expected 3 fields, saw 4\n
What I want is to be able to load the data but log this skipping line number in a log file. I went throught a lot of tutorials on logging but couldn't find a way to log this auto generated message when pandas skip a line number while loading the data.
This is the simple piece of code I'm using to load a file.
import pandas as pd
import os
def main():
filename = "test_data3.csv"
data= pd.read_csv(filename,error_bad_lines=False)
print(data.head())
if __name__=="__main__":
main()
Here is the sample data I'm using
Col1,Col2,Col3
a,3,g4
b,4,s5,r
c,5,p9
f,6,v4,7
x,65,h5
as you can see line number 2 and 4 should be skipped. But it needs to be recorded in a log file.
You can use a context manager to temporarily intercept calls to sys.stderr.write and write the messages to a file:
import pandas as pd
import sys
class CaptureErrors:
def __init__(self, stderr, output_name):
self.stderr = stderr
self.output_name = output_name
self.output_file = None
def __enter__(self):
self.output_file = open(self.output_name, "w")
return self
def __exit__(self, exc_type, exc_value, traceback):
if self.output_file:
self.output_file.close()
sys.stderr = self.stderr
def write(self, message):
self.stderr.write(message)
self.output_file.write(message)
def main():
filename = "test_data3.csv"
with CaptureErrors(sys.stderr, 'error.txt') as sys.stderr:
data = pd.read_csv(filename, error_bad_lines=False)
print(data.head())
if __name__=="__main__":
main()
If this isn't what you are looking for, you may need to add more information to your question.
You can use Redirect the output into a file doubg:
python script.py > out.txt
I have written script that parses a web page and saves data of interest in a CSV file. Before I open the data and use it in a second script I check if the file with data exist and if not I am running the parser script first. The odd behaviour of the second script is, that it is able to detect that there is no file, then the file is created, but when it is read for the first time it is empty (part of else statement). I tried to provide some delay by using the time.sleep() method, but it does not work. The explorer clearly shows that the file is not empty, but at the first run, script recognizes the file as empty. At the subsequent runs the scripts clearly sees the file and is able to properly recognize it content.
Maybe You have some explanation for this behaviour.
def open_file():
# TARGET_DIR and URL are global variables.
all_lines = []
try:
current_file = codecs.open(TARGET_DIR, 'r', 'utf-8')
except FileNotFoundError:
procesed_data = parse_site(URL)
save_parsed(procesed_data)
compare_parsed()
open_file()
else:
time.sleep(10)
data = csv.reader(current_file, delimiter=';')
for row in data:
all_lines.append(row)
current_file.close()
return all_lines
You got some recursion going on.
Another way to do it—assuming I understand correctly—is this:
import os
def open_file():
# TARGET_DIR and URL are global variables.
all_lines = []
# If the file is not there, make it.
if not os.path.isfile(TARGET_DIR):
procesed_data = parse_site(URL)
save_parsed(procesed_data)
compare_parsed()
# Here I am assuming the file has been created.
current_file = codecs.open(TARGET_DIR, 'r', 'utf-8')
data = csv.reader(current_file, delimiter=';')
for row in data:
all_lines.append(row)
current_file.close()
return all_lines
you should return the result of your internal open_file call, or just opening the file in your except block:
def open_file():
# TARGET_DIR and URL are hopefully constants
try:
current_file = codecs.open(TARGET_DIR, 'r', 'utf-8')
except FileNotFoundError:
procesed_data = parse_site(URL)
save_parsed(procesed_data)
compare_parsed()
current_file = codecs.open(TARGET_DIR, 'r', 'utf-8')
data = csv.reader(current_file, delimiter=';')
all_lines = list(data)
current_file.close()
return all_lines
I'm using cgitb (python 2.7) to create html documents server end. I have on file that does a bunch of query and then produces html. I'd like to be able to link just the html so if I could print the html to a new file and link that that, it would work.
Is there a way to get the html the page will generate at the end of processing so that I can put it in a new file without keeping track of everything I've done so far along the way?
Edit: Found a snipped here: https://stackoverflow.com/a/616686/1576740
class Tee(object):
def __init__(self, name, mode):
self.file = open(name, mode)
self.stdout = sys.stdout
sys.stdout = self
def __del__(self):
sys.stdout = self.stdout
self.file.close()
def write(self, data):
self.file.write(data)
self.stdout.write(data)
You have to call it after you import cgi as it overrides stdout in what appears to be a less friendly way. But works like a charm.
I just did import cgi;.......
Tee(filname, "w") and then I have a link to the file.
From the Python Documentation
Optionally, you can save this information to a file instead of sending it to the browser.
In this case you would want to use
cgitb.enable(display=1, logdir=directory)
import cgitb
import sys
try:
...
except:
with open("/path/to/file.html", "w") as fd:
fd.write(cgitb.html(sys.exc_info()))
I would like to create a function that keeps a record of every print command, storing each command's string into a new line in a file.
def log(line):
with open('file.txt', "a") as f:
f.write('\n' + line)
This is what I have, but is there any way to do what I said using Python?
Try replacing stdout with custom class:
import sys
class LoggedStdout():
def __init__(self, filename = None):
self.filename = filename
def write(self, text):
sys.__stdout__.write(text)
if not self.filename is None:
self.log(text)
def log(self, line):
with open(self.filename, "a") as f:
f.write('\n' + line)
sys.stdout = LoggedStdout('file.txt')
print 'Hello world!'
This would affect not only print, but also any other function that prints something to stdout, but it is often even better.
For production-mode logging it's much better to use something like logging module, rather than home-made hooks over standard IO streams.