Develop Reference

Is there a way to log auto generated messages on Python console?

I'm using pandas to load a csv file that has few bad lines. This means that in few lines there are some extra commas and that is why pandas is not able to load it. Which is fine by me. I'm using error_bad_lines=False to ignore those lines. When those bad lines are ignored by pandas, it shows a message like this on console:
b'Skipping line 3: expected 3 fields, saw 4\n
What I want is to be able to load the data but log this skipping line number in a log file. I went throught a lot of tutorials on logging but couldn't find a way to log this auto generated message when pandas skip a line number while loading the data.
This is the simple piece of code I'm using to load a file.
import pandas as pd
import os
def main():
filename = "test_data3.csv"
data= pd.read_csv(filename,error_bad_lines=False)
print(data.head())
if __name__=="__main__":
main()
Here is the sample data I'm using
Col1,Col2,Col3
a,3,g4
b,4,s5,r
c,5,p9
f,6,v4,7
x,65,h5
as you can see line number 2 and 4 should be skipped. But it needs to be recorded in a log file.

You can use a context manager to temporarily intercept calls to sys.stderr.write and write the messages to a file:
import pandas as pd
import sys
class CaptureErrors:
def __init__(self, stderr, output_name):
self.stderr = stderr
self.output_name = output_name
self.output_file = None
def __enter__(self):
self.output_file = open(self.output_name, "w")
return self
def __exit__(self, exc_type, exc_value, traceback):
if self.output_file:
self.output_file.close()
sys.stderr = self.stderr
def write(self, message):
self.stderr.write(message)
self.output_file.write(message)
def main():
filename = "test_data3.csv"
with CaptureErrors(sys.stderr, 'error.txt') as sys.stderr:
data = pd.read_csv(filename, error_bad_lines=False)
print(data.head())
if __name__=="__main__":
main()
If this isn't what you are looking for, you may need to add more information to your question.

You can use Redirect the output into a file doubg:
python script.py > out.txt

python is not logging all content to file

I want to log the script output to a file while still displaying the output to the screen.
It works fine, except for some cases where not all the content is written to the file (one or two lines can be missed, if the output is long)
Below is my code:
class Tee(object):
def __init__(self, *files):
self.files = files
def write(self, obj):
for f in self.files:
f.write(obj)
f.flush()
write_log = open("log.txt", 'a', 0)
sys.stdout = Tee(sys.stdout, write_log)
sys.stderr = Tee(sys.stderr, write_log)
Tried all the following options at the end of the code, but the result is the same:
os.fsync(write_log.fileno())
write_log.flush()
write_log.close()

Try using the with statement or use try-except and explicitly close the file.

Get current console output in python

I want to get the current console output of my program in python. There are a lot of solutions to get the console output when running an external program, however, I couldn't find any solution for getting the console output of the current program. Am I missing something? I am looking for a solution which works under windows and linux.
For example:
print "Hello world"
output = get_console_output() # Returns "Hello World\n"
Edit:
The solution should preserve the console output, so just replacing stdout won't work, as the console will be empty then

If you want to access the output you need to redirect the standard output stdout somewhere. You can use StringIO for this for example:
from cStringIO import StringIO
import sys
sys.stdout = buffer = StringIO()
print "Hello World"
# get output via: buffer.getvalue()
If you rather want the output to a file you could instead redirect directly to a file:
import sys
sys.stdout = open('output.txt', 'w')
print 'Hello World'
Edit: If you want output to be appended to log (according to comment), I suggest a custom class:
import sys
class Log(object):
def __init__(self):
self.orgstdout = sys.stdout
self.log = open("log.txt", "a")
def write(self, msg):
self.orgstdout.write(msg)
self.log.write(msg)
sys.stdout = Log()
print('Hello World')

You can overwrite sys.stdout with any file-like object:
import sys
import StringIO
sys.stdout = StringIO.StringIO()
You should also think about using the logging module instead of print. Or simply write a function that stores and prints values.

Making a unicode file into a different file

Currently, I have a Python script that opens a file and appends a Tweet from Twitter to a file. Part of my listener for stream listener goes like this:
class listener(StreamListener):
def __init__(self, api=None, path=None,outname='output',MAX_NUMBER_OF_TWEETS=100,TWEETS_PER_FILE=10,progress_bar=None):
self.api = api
self.path = path
self.count = 0
self.outname = outname
self.progress_bar = progress_bar
self.MAX_NUMBER_OF_TWEETS = MAX_NUMBER_OF_TWEETS
self.TWEETS_PER_FILE = TWEETS_PER_FILE
def on_data(self, data):
all_data = json.loads(data)
with open(filename,"a") as fid:
print>>fid,all_data
However, the file that is printed out is a Unicode file.
How would I get let's say a text file or a JSON file?

Unicode is a text file. What you are probably seeing is the output in the wrong output code (Say, the file could be in UTF-8 and your console might be in another coding).
Python let's you select the coding of the output file if you use the io module. On opening a file for write, you can select the coding you want to use. If you want to type on the screen, you should use the correct for the screen. (In Linux, use set to check the locale. No idea how to do that in Windows)

How to implement the 'tempfile' module to this code

This is a small widget that I am designing that is designed to 'browse' while circumventing proxy settings. I have been told on Code Review that it would be beneficial here, but am struggling to put it in with my program's current logic. Here is the code:
import urllib.request
import webbrowser
import os
import tempfile
location = os.path.dirname(os.path.abspath(__file__))
proxy_handler = urllib.request.ProxyHandler(proxies=None)
opener = urllib.request.build_opener(proxy_handler)
def navigate(query):
response = opener.open(query)
html = response.read()
return html
def parse(data):
start = str(data)[2:-1]
lines = start.split('\\n')
return lines
while True:
url = input("Path: ")
raw_data = navigate(url)
content = parse(raw_data)
with open('cache.html', 'w') as f:
f.writelines(content)
webbrowser.open_new_tab(os.path.join(location, 'cache.html'))
Hopefully someone who has worked with these modules before can help me. The reason that I want to use tempfile is that my program gets raw html, parses it and stores it in a file. This file is overwritten every time a new input comes in, and would ideally be deleted when the program stops running. Also, the file doesn't have to exist when the program initializes so it seems logical from that view also.

Since you are passing the name of the file to webbrowser.open_new_tab(), you should use a NamedTemporaryFile
cache = tempfile.NamedTemporaryFile()
...
cache.seek(0)
cache.writelines(bytes(line, 'UTF-8') for line in content)
cache.seek(0)
webbrowser.open_new_tab('file://' + cache.name)