I've got a Python program that reads from sys.stdin, so I can call it with ./foo.py < bar.png. How do I test this code from within another Python module? That is, how do I set stdin to point to the contents of a file while running the test script? I don't want to do something like ./test.py < test.png. I don't think I can use fileinput, because the input is binary, and I only want to handle a single file. The file is opened using Image.open(sys.stdin) from PIL.
You should generalise your script so that it can be invoked from the test script, in addition to being used as a standalone program. Here's an example script that does this:
#! /usr/bin/python
import sys
def read_input_from(file):
print file.read(),
if __name__ == "__main__":
if len(sys.argv) > 1:
# filename supplied, so read input from that
filename = sys.argv[1]
file = open(filename)
else:
# no filename supplied, so read from stdin
file = sys.stdin
read_input_from(file)
If this is called with a filename, the contents of that file will be displayed. Otherwise, input read from stdin will be displayed. (Being able to pass a filename on the command line might be a useful improvement for your foo.py script.)
In the test script you can now invoke the function in foo.py with a file, for example:
#! /usr/bin/python
import foo
file = open("testfile", "rb")
foo.read_input_from(file)
Your function or class should accept a stream instead of choosing which stream to use.
Your main function will choose sys.stdin.
Your test method will probably choose a StringIO instance or a test file.
The program:
# foo.py
import sys
from PIL import Image
def foo(stream):
im = Image.open(stream)
# ...
def main():
foo(sys.stdin)
if __name__ == "__main__":
main()
The test:
# test.py
import StringIO, unittest
import foo
class FooTest(unittest.TestCase):
def test_foo(self):
input_data = "...."
input_stream = StringIO.StringIO(input_data)
# can use a test file instead:
# input_stream = open("test_file", "rb")
result = foo.foo(input_stream)
# asserts on result
if __name__ == "__main__":
unittest.main()
A comp.lang.python post showed the way: Substitute a StringIO() object for sys.stdout, and then get the output with getvalue():
def setUp(self):
"""Set stdin and stdout."""
self.stdin_backup = sys.stdin
self.stdout_backup = sys.stdout
self.output_stream = StringIO()
sys.stdout = self.output_stream
self.output_file = None
def test_standard_file(self):
sys.stdin = open(EXAMPLE_PATH)
foo.main()
self.assertNotEqual(
self.output_stream.getvalue(),
'')
def tearDown(self):
"""Restore stdin and stdout."""
sys.stdin = self.stdin_backup
sys.stdout = self.stdout_backup
You can always monkey patch Your stdin. But it is quite ugly way. So better is to generalize Your script as Richard suggested.
import sys
import StringIO
mockin = StringIO.StringIO()
mockin.write("foo")
mockin.flush()
mockin.seek(0)
setattr(sys, 'stdin', mockin)
def read_stdin():
f = sys.stdin
result = f.read()
f.close()
return result
print read_stdin()
Also, do not forget to restore stdin when tearing down your test.
Related
This is a simple ask but I can't find any information on how to do it: I have a python script that is designed to take in a text file of a specific format and perform functions on it--how do I pipe a test file into the python script such that it is recognized as input()? More specifically, the Python is derived from skeleton code I was given that looks like this:
def main():
N = int(input())
lst = [[int(i) for i in input().split()] for _ in range(N)]
intervals = solve(N, lst)
print_solution(intervals)
if __name__ == '__main__':
main()
I just need to understand how to, from the terminal, input one of my test files to this script (and see the print_solution output)
Use the fileinput module
input.txt
...input.txt contents
script.py
#!/usr/bin/python3
import fileinput
def main():
for line in fileinput.input():
print(line)
if __name__ == '__main__':
main()
pipe / input examples:
$ cat input.txt | ./script.py
...input.txt contents
$ ./script.py < input.txt
...input.txt contents
You can take absolute or relative path in your input() function and then open this path via open()
filename = input('Please input absolute filename: ')
with open(filename, 'r') as file:
# Do your stuff
Please let me know if I misunderstood your question.
You can either:
A) Use sys.stdin (import sys at the top of course)
or
B) Use the ArgumentParser (from argparse import ArgumentParser) and pass the file as an argument.
Assuming A it would look something like this:
python script.py < file.extension
Then in the script it would look like:
fData = []
for line in sys.stdin.readLines():
fData.append(line)
# manipulate fData
There are a number of ways to achieve what you want. This is what I came up with off the top of my head. It may not be the best / efficient way, but it should work. I do a lot of file I/O with python at work and this is one of the ways I've achieved it in the past.
Note: If you want to write the manipulated lines back to the file use the argparse library.
Edit:
from argparse import ArgumentParser
def parseInput():
parser = ArgumentParser(description = "Takes input file to read")
parser.add_argument('-f', type = str, default = None, required =
True, help = "File to perform I/O on")
args = parser.parse_args()
return args
def main():
args = parseInput()
fData = []
# perform rb
with open(args.f, 'r') as f:
for line in f.readlines():
fData.append(line)
# Perform data manipulations
# perform wb
with open(args.f, 'w') as f:
for line in fData:
f.write(line)
if __name__ == "__main__":
main()
Then on command line it would look like:
python yourScript.py -f fileToInput.extension
I created a python program, test.py, below:
import subprocess
import sys, os
FolderPath = subprocess.getoutput("cd . && pwd")
ProgramName = sys.argv[0]
LogName = ProgramName[:-3]+'_printout.txt'
ProgramFile = FolderPath+'/'+ProgramName
LogFile = FolderPath+'/'+LogName
_stdin = sys.stdin
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdin = open(LogFile, 'w')
sys.stdout = open(LogFile, 'a')
sys.stderr = open(LogFile, 'a')
Prog = open(ProgramFile, 'r')
print(Prog.read())
TEST = str(input("Enter the name: \n TEST_NAME: "))
print(TEST)
sys.stdin = _stdin.flush()
sys.stdout = _stdout.flush()
sys.stderr = _stderr.flush()
After I executed on linux with command python test.py, I got the error in test_printout.txt.
Enter the name:
TEST_NAME: Traceback (most recent call last):
File "test.py", line 21, in <module>
TEST = str(input("Enter the name: \n TEST_NAME: "))
io.UnsupportedOperation: not readable
I modified the code:
import subprocess
import sys, os
FolderPath = subprocess.getoutput("cd . && pwd")
ProgramName = sys.argv[0]
LogName = ProgramName[:-3]+'_printout.txt'
ProgramFile = FolderPath+'/'+ProgramName
LogFile = FolderPath+'/'+LogName
_stdin = sys.stdin
_stdout = sys.stdout
_stderr = sys.stderr
sys.stdin = open(LogFile, 'w+')
sys.stdout = open(LogFile, 'a')
sys.stderr = open(LogFile, 'a')
Prog = open(ProgramFile, 'r')
print(Prog.read())
TEST = str(input("Enter the name: \n TEST_NAME: "))
print(TEST)
sys.stdin = _stdin.flush()
sys.stdout = _stdout.flush()
sys.stderr = _stderr.flush()
But got:
Enter the name:
TEST_NAME: import subprocess
It did not let me type anything. What I want is to let me type string and it also save to test_printout.txt.
Enter the name:
TEST_NAME: This Is What I Type And Save!
Does anyone know how to fix it?
Also, if I use w+ instead of w mode, it will take longer time to write to the test_printout.txt if I changed the program to import pandas.DataFrame and manipulate data.
Is there a way to only write simple print words to test_printout.txt without reading entire thing?
UPDATE
I modified the code as below:
import subprocess, sys, os
FolderPath = subprocess.getoutput("cd . && pwd")
ProgramName = sys.argv[0]
LogName = ProgramName[:-3]+'_printout.txt'
ProgramFile = FolderPath+'/'+ProgramName
LogFile = FolderPath+'/'+LogName
_stdin = sys.stdin
_stdout = sys.stdout
_stderr = sys.stderr
class stdout_Logger(object):
def __init__(self):
self.stdout = sys.stdout
self.log = open(LogFile, "a")
def write(self, message):
self.stdout.write(message)
self.log.write(message)
def flush(self):
#this flush method is needed for python 3 compatibility.
#this handles the flush command by doing nothing.
#you might want to specify some extra behavior here.
pass
sys.stdout = stdout_Logger()
class stderr_Logger(object):
def __init__(self):
self.stderr = sys.stderr
self.log = open("test_printout.txt", "a")
def write(self, message):
self.stderr.write(message)
self.log.write(message)
def flush(self):
#this flush method is needed for python 3 compatibility.
#this handles the flush command by doing nothing.
#you might want to specify some extra behavior here.
pass
sys.stderr = stderr_Logger()
Prog = open(ProgramFile, 'r')
print(Prog.read())
##START Program
TEST = str(input("Enter the name: \n TEST_NAME: "))
print(TEST)
#END Program
sys.stdin = _stdin.flush()
sys.stdout = _stdout.flush()
sys.stderr = _stderr.flush()
This got almost what I want. This also save my program to test_printout.txt at the top and do print(TEST) in the bottom.
However, it also prints all program to the linux terminal console which is not I desire. I only want it to print "Enter the name: \n TEST_NAME: " in linux terminal and I can type string instead of printing entire program.
I think the issue came from sys.stdin.
I think I figured it out. The problem is that when you substitute input with a file-handle in write mode you ban input() from reading it. You can get the same error if you tried this:
file = open("foo.txt",'w')
content = file.read()
The way to go around it is to log streams without redirecting them. So either you dump your console to file with python test.py > test_printout.txt or create a logger class to wrap around the streams (check out this answer: How to redirect stdout to both file and console with scripting?).
Perhaps its worth for you to look into the logging module, as I believe it handles these issues rather neatly.
EDIT:
From what you laid out in the comments, this is what you want:
import subprocess, sys, os
FolderPath = subprocess.getoutput("cd . && pwd")
ProgramName = sys.argv[0]
LogName = ProgramName[:-3]+'_printout.txt'
ProgramFile = FolderPath+'/'+ProgramName
LogFile = FolderPath+'/'+LogName
Prog = open(ProgramFile, 'r')
with open(LogFile, 'w') as logfile:
logfile.write(Prog.read())
_stdin = sys.stdin
_stdout = sys.stdout
_stderr = sys.stderr
class stdout_Logger(object):
def __init__(self):
self.stdout = sys.stdout
self.log = open(LogFile, "a")
def write(self, message):
self.stdout.write(message)
self.log.write(message)
def flush(self):
#this flush method is needed for python 3 compatibility.
#this handles the flush command by doing nothing.
#you might want to specify some extra behavior here.
pass
class stderr_Logger(object):
def __init__(self):
self.stderr = sys.stderr
self.log = open("test_printout.txt", "a")
def write(self, message):
self.stderr.write(message)
self.log.write(message)
def flush(self):
#this flush method is needed for python 3 compatibility.
#this handles the flush command by doing nothing.
#you might want to specify some extra behavior here.
pass
sys.stdout = stdout_Logger()
sys.stderr = stderr_Logger()
##START Program
TEST = str(input("Enter the name: \n TEST_NAME: "))
print(TEST)
#END Program
sys.stdin = _stdin.flush()
sys.stdout = _stdout.flush()
sys.stderr = _stderr.flush()
I'm using pytest and want to test that a function writes some content to a file. So I have writer.py which includes:
MY_DIR = '/my/path/'
def my_function():
with open('{}myfile.txt'.format(MY_DIR), 'w+') as file:
file.write('Hello')
file.close()
I want to test /my/path/myfile.txt is created and has the correct content:
import writer
class TestFile(object):
def setup_method(self, tmpdir):
self.orig_my_dir = writer.MY_DIR
writer.MY_DIR = tmpdir
def teardown_method(self):
writer.MY_DIR = self.orig_my_dir
def test_my_function(self):
writer.my_function()
# Test the file is created and contains 'Hello'
But I'm stuck with how to do this. Everything I try, such as something like:
import os
assert os.path.isfile('{}myfile.txt'.format(writer.MYDIR))
Generates errors which lead me to suspect I'm not understanding or using tmpdir correctly.
How should I test this? (If the rest of how I'm using pytest is also awful, feel free to tell me that too!)
I've got a test to work by altering the function I'm testing so that it accepts a path to write to. This makes it easier to test. So writer.py is:
MY_DIR = '/my/path/'
def my_function(my_path):
# This currently assumes the path to the file exists.
with open(my_path, 'w+') as file:
file.write('Hello')
my_function(my_path='{}myfile.txt'.format(MY_DIR))
And the test:
import writer
class TestFile(object):
def test_my_function(self, tmpdir):
test_path = tmpdir.join('/a/path/testfile.txt')
writer.my_function(my_path=test_path)
assert test_path.read() == 'Hello'
I want to get the current console output of my program in python. There are a lot of solutions to get the console output when running an external program, however, I couldn't find any solution for getting the console output of the current program. Am I missing something? I am looking for a solution which works under windows and linux.
For example:
print "Hello world"
output = get_console_output() # Returns "Hello World\n"
Edit:
The solution should preserve the console output, so just replacing stdout won't work, as the console will be empty then
If you want to access the output you need to redirect the standard output stdout somewhere. You can use StringIO for this for example:
from cStringIO import StringIO
import sys
sys.stdout = buffer = StringIO()
print "Hello World"
# get output via: buffer.getvalue()
If you rather want the output to a file you could instead redirect directly to a file:
import sys
sys.stdout = open('output.txt', 'w')
print 'Hello World'
Edit: If you want output to be appended to log (according to comment), I suggest a custom class:
import sys
class Log(object):
def __init__(self):
self.orgstdout = sys.stdout
self.log = open("log.txt", "a")
def write(self, msg):
self.orgstdout.write(msg)
self.log.write(msg)
sys.stdout = Log()
print('Hello World')
You can overwrite sys.stdout with any file-like object:
import sys
import StringIO
sys.stdout = StringIO.StringIO()
You should also think about using the logging module instead of print. Or simply write a function that stores and prints values.
I have log files located in:
/mfs/log/scribe/clicklog/*/clicklog_current
which I want to process in realtime with Python, so I created a transform.py file:
tail -f /mfs/log/scribe/clicklog/*/clicklog_current | grep 'pattern' | ./transform.py
in tranform.py:
def process_line(line):
print real_process(line)
the problem is: How can I call process_line everytime there is a new line from stdin?
Whenever redirection or piping happening, the standard input stream will be set to that. So you can directly read from sys.stdin, like this
import sys
for line in sys.stdin:
process_line(line)
If the buffering bites you, you can adjust/disable the input buffering, like mentioned in this answer
Reduce the buffering size:
import os
import sys
for line in os.fdopen(sys.stdin.fileno(), 'r', 100):
process_line(line)
Now it buffers only 100 bytes max.
Disable the buffering:
Quoting the official documentation,
-u
Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.
The fileinput library may be able to do what you're looking for.
import fileinput
for line in fileinput.input():
if line == '': pass
process_line(line)
You can get rid of the tail -f part completely by using watchdog and grep by using the re module (although in this case, you don't even need that as your search criteria can be written as a simple membership test).
Here is a simple example (modified from the documentation) that would do what you require:
import sys
import time
from watchdog.observers import Observer
from watchdog.handlers import FileSystemEventHandler
class WatchFiles(FileSystemEventHandler):
def process_file(self, event):
"""
does stuff the file
"""
with open(event.src_path, 'r') as f:
for line in f:
if 'pattern' in line:
do_stuff(line)
def on_modified(self, event):
self.process_file(event)
def on_created(self, event):
self.process_file(event)
if __name__ == "__main__":
path = sys.argv[1] if len(sys.argv) > 1 else '.'
observer = Observer()
observer.schedule(WatchFiles(), path, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
This way, your application is not only more portable but all parts of it are self-contained.