I'm writing a command-line interface in Python. It uses the readline module to provide command history and completion.
While everything works fine in interactive mode, I'd like to run automated tests on the completion feature. My naive first try involved using a file for standard input:
my_app < command.file
The command file contained a tab, in the hopes that it would invoke the completion feature. No luck. What's the right way to do the testing?
For this I would use Pexpect (Python version of Expect). The readline library needs to be speaking to a terminal to do interactive tab-completion and such—it can't do this if it is only getting one-way input from a redirected file.
Pexpect works for this because it creates a pseudo terminal, which consists of two parts: the slave, where the program you are testing runs, and the master, where the Python pexpect code runs. The pexpect code emulates the human running the test program. It is responsible for sending characters to the slave, including characters such as newline and tab, and reacting to the expected output (this is where the phrase "expect" comes from).
See the program ftp.py from the examples directory for a good example of how you would control your test program from within expect. Here is a sample of the code:
child = pexpect.spawn('ftp ftp.openbsd.org')
child.expect('(?i)name .*: ')
child.sendline('anonymous')
child.expect('(?i)password')
child.sendline('pexpect#sourceforge.net')
child.expect('ftp> ')
rlcompleter might accomplish what you want
From the documentation:
The rlcompleter module is designed for use with Python’s interactive mode. A user can add the following lines to his or her initialization file (identified by the PYTHONSTARTUP environment variable) to get automatic Tab completion:
try:
import readline
except ImportError:
print "Module readline not available."
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
https://docs.python.org/2/library/rlcompleter.html
Check out ScriptTest:
from scripttest import TestFileEnvironment
env = TestFileEnvironment('./scratch')
def test_script():
env.reset()
result = env.run('do_awesome_thing testfile --with extra_win --file %s' % filename)
And play around with passing the arguments as you please.
You can try using Sikuli to test the end-user interaction with your application.
However, this is a complete overkill, requires a lot of extra dependencies, will work slowly, and will fail if the terminal font/colors change. But, still, you will be able to test actual user interaction.
The documentation homepage links to a slideshow and a FAQ question about writing tests using Sikuli.
Related
I'm currently doing some work in a server (Ubuntu) without admin rights nor contact with the administrator. When using the help(command) in the python command line I get an error.
Here's an example:
>>> help(someCommand)
/bin/sh: most: command not found
So, this error indicates that most pager is not currently installed. However, the server I'm working on has "more" and "less" pagers installed. So, how can I change the default pager configuration for this python utility?
This one is annoyingly difficult to research, but I think I found it.
The built-in help generates its messages using the standard library pydoc module (the module is also intended to be usable as a standalone script). In that documentation, we find:
When printing output to the console, pydoc attempts to paginate the output for easier reading. If the PAGER environment variable is set, pydoc will use its value as a pagination program.
So, presumably, that's been set to most on your system. Assuming it won't break anything else on your system, just unset or change it. (It still pages without a value set - even on Windows. I assume it has a built-in fallback.)
You can make a custom most script that just invokes less (or even more).
The steps would be:
Set up a script called most, the contents of which are:
#!/bin/sh
less ${#:1} # wierdess is just "all arguments except argument 0"
Put that script in a location that is on your PATH
Then most filename should just run less on that file, and that command should get called from in your python interpreter.
To be honest though, I'd just use Karl's approach.
You can view the various pager options in the source code. That function can be replaced to return whatever is desired. For example:
import pydoc
pydoc.getpager = lambda: lambda text: pydoc.pipepager(text, 'less')
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
This Python program enters fdisk. I see the output. fdisk is an interactive program. How do I get the Python program to pass an "m" to the first field and press enter?
import subprocess
a = "dev/sda"
x = subprocess.call(["fdisk", a])
print x
I'd rather not import a new module/library, but I could. I've tried different syntax with subprocess.call() and extra parameters in the above. Nothing seems to work. I get different errors. I've reviewed Python documentation. I want to feed input and press Enter in the subsequent, interactive menu options of fdisk.
Check out the pexpect library (I know you didn't want an extra module, but you want to use the best tool for the job). It's pure Python, with no compiled submodules, so installation is a snap. Basically, it does the same thing in Python as the classic Unix utility expect - spawns child applications, controls them, and responds to expected patterns in their output. It's great for automation, and especially application testing, where you can quickly feed the newest build of a command-line program a series of inputs and guide the interaction based on what output appears.
In case you just don't want another module at all, you can always fall back on the subprocess module's Popen() constructor. It spawns and creates a connection to a child process, allowing you to communicate with it as needed, and in fact pexpect relies a great deal on it. I personally think using pexpect is more intuitive than subprocess.Popen(), but that's just me. YMMV.
I am playing with some programming challenges that will check the submission by:
python my_submission < in.txt > out.txt
When I try and make my submission, I want to read some cases/numbers/whatever from in.txt to see what is happening. Currently I am doing that by:
import sys
file = open('in.txt')
sys.stdin = file
for line in sys.stdin:
case1 = line.split()
some_function(case1)
So when I run my python program (hit cmd+B) in Sublime text, I can see whether I manage to read the input correctly, process one test case correctly, etc.... Then I just commend out the 2nd and 3rd line when my program should be submitted to the submission judge.
I was just wondering: is this the "preffered workflow" for dealing with this? Do pro programmers write some kind of unit test template function to do this?
The preferred workflow is to let the shell doing the redirection so you don't have to change the program code all the time.
But your IDE (sublime text) doesn't allow you to specify such arguments, so it limits your options.
Solutions/workarounds:
Start the program from a shell. Which means you need to switch between the terminal window and sublime all the time.
Write a second program which runs the first and which sets up the input redirection. This way, you just need to switch tabs in sublime.
Instead of reading from stdin directly, use the fileinput module. See How do you read from stdin in Python? This will allow you to write proper unit tests for your code. You can then use the Python Unittest Helper plugin for Sublime.
How to send string/data to STDIN of a running process in python?
i'd like to create a front end for a CLI program. eg. i want to pass multiple string to this Pascal application:
program spam;
var a,b,c:string;
begin
while e <> "no" do
begin
writeln('what is your name?');
readln(a);
writeln('what is your quest?');
readln(b);
writeln('what is your favorite color?');
readln(c);
print(a,b,c);
end;
end.
how do i pass string to this program from python (using subprocess module in python). thankyou. sorry for my english.
If you want to control another interactive program, it could be worth trying the Pexpect module to do so. It is designed to look for prompt messages and so on, and interact with the program. Note that it doesn't currently work directly on Windows - it does work under Cygwin.
A possible non-Cygwin Windows variant is WinPexpect, which I found via this question. One of the answers on that question suggests the latest version of WinPexpect is at http://sage.math.washington.edu/home/goreckc/sage/wexpect/, but looking at the modification dates I think the BitBucket (the first link) is actually the latest.
As Windows terminals are somewhat different to Unix ones, I don't think there is a direct cross-platform solution. However, the WinPexpect docs say the only difference in the API between it and pexpect is the name of the spawn function. You could probably do something like the following (untested) code to get it to work in both:
try:
import pexpect
spawn = pexpect.spawn
except ImportError:
import winpexpect
spawn = winpexpect.winspawn
# NB. Errors may occur when you run spawn rather than (or as
# well as) when you import it, so you may have to wrap this
# up in a try...except block and handle them appropriately.
child = spawn('command and args')