Develop Reference

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

pexpect not executing command by steps

I have this Python3 code which use Pexpect.
import pexpect
import getpass
import sys
def ssh(username,password,host,port,command,writeline):
child = pexpect.spawn("ssh -p {} {}#{} '{}'".format(port,username,host,command))
child.expect("password: ")
child.sendline(password)
if(writeline):
print(child.read())
def scp(username,password,host,port,file,dest):
child = pexpect.spawn("scp -P {} {} {}#{}:{}".format(port,file,username,host,dest))
child.expect("password: ")
child.sendline(password)
try:
filename = sys.argv[1]
print("=== sendhw remote commander ===")
username = input("Username: ")
password = getpass.getpass("Password: ")
ssh(username,password,"some.host.net","22","mkdir ~/srakrnSRV",False)
scp(username,password,"some.host.net","22",filename,"~/srakrnSRV")
ssh(username,password,"some.host.net","22","cd srakrnSRV && sendhw {}".format(filename),True)
except IndexError:
print("No homework name specified.")
My aim is to:
SSH into the host with the ssh function, create the directory srakrnSRV, then
upload a file into the srakrnSRV directory, which is previously created
cd into srakrnSRV, and execute the sendhw <filename> command. The filename variable is defined by command line parameteres, and print the result out.
After running the entire code, Python prints out
b'\r\nbash: line 0: cd: srakrnSRV: No such file or directory\r\n'
which is not expected, as the directory should be previously created.
Also, I tried manually creating the srakrnSRV folder in my remote host. After running the command again, it appears that scp function is also not running. The only runnning pexpect coomand was the last ssh function.
How to make it execute in order? Thanks in advance!

You may lack permission for executing commands through ssh. Also there is possibility that your program sends scp before prompt occurs.

Script to capture everything on screen

So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)

Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log

I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect

I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)

Error sh: logic: command not found when trying to run logic from python

I need to run this program on some data files. When I type it in the command line it usually works like so:
logic ./ case
I try to call this from a python script like so:
def runLogic(pProtocol):
p = 2
base = os.path.basename(pProtocol[0])
path = pProtocol[0].rstrip(base)
for i in range (1,int(pProtocol[1])+1):
pAlpha = float(pProtocol[p])
pl = float(pProtocol[p+1])
case = 'Einlauf_Lippe_alpha%sdeg_l%smm'%(pAlpha, pl)
# Write mini Python scriptfile
pyPath = str(pProtocol[0])+'/Einlauf_Lippe_alpha%sdeg_l%smm/'%(pAlpha, pl)
PyScript = "import os\n\ndef executeLogic():\n case = '%s'\n os.system('logic ./ %%s'%%case)\n\nexecuteLogic()"%case
pyfile=open(pyPath + 'executeLogic.py', 'w')
pyfile.write(PyScript)
pyfile.close()
# Run local PyScript
os.system('python %sexecuteLogic.py'%pyPath)
# Remove local PyScript
os.system('rm %sexecuteLogic.py'%pyPath)
p = p + 2
i = i + 1
In order to create a local python script in the directory where the files to be worked on an then try to execute it. The mini script looks like so:
import os
def executeLogic():
case = 'Einlauf_Lippe_alpha7.5deg_l9.0mm'
#print 'logic ./ %s'%case
os.system('logic ./ %s'%case)
executeLogic()
When I just type the very line logic ./ case into my terminal it works just fine. When I try to start it from python I get the following:
sh: logic: command not found

python 3 open terminal and run program

I made a small script in sublime that will extract commands from a json file that is on the user's computer and then it will open the terminal and run the settings/command. This works, except that it doesn't really open up the terminal. It only runs the command (and it works, as in my case it will run gcc to compile a simple C file), and pipes to STDOUT without opening up the terminal.
import json
import subprocess
import sublime_plugin
class CompilerCommand(sublime_plugin.TextCommand):
def get_dir(self, fullpath):
path = fullpath.split("\\")
path.pop()
path = "\\".join(path)
return path
def get_settings(self, path):
_settings_path = path + "\\compiler_settings.json"
return json.loads(open(_settings_path).read())
def run(self, edit):
_path = self.get_dir(self.view.file_name())
_settings = self.get_settings(_path)
_driver = _path.split("\\")[0]
_command = _driver + " && cd " + _path + " && " + _settings["compile"] + " && " + _settings["exec"]
proc = subprocess.Popen(_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I'm not sure if using subprocess.Popen is the right way to go about it as I'm new to Python.
So to re-iterate; I want it to open up the terminal, run the command, and have the terminal stay open until the user presses ENTER or something. I'm running Windows 7 and Python 3, if that matters.

subprocess.Popen simply creates a subprocess with the given command. It is in no way related to opening a terminal window or any other windows for that matter.
You'll have to look into your platform specific UI automation solutions in order to achieve what you want. Or see if maybe the Sublime plugins mechanism can already do that.
NOTES:
Also, you should be using os.path.join/os.path.split/os.path.sep etc for your path operations—Sublime also runs on OS X for example, and OS X does not use backslashes. Also, file handles need to be closed, so use:
with open(...) as f:
return json.load(f) # also not that there is no nead to f.read()+json.loads()
# if you can just json.load() on the file handle
Furthermore, strings should usually be built using string interpolation:
_command = "{} && cd {} && {} && {}".format(_driver, _path, _settings["compile"], _settings["exec"])
...and, you should not be prefixing your local variables with _—it doesn't look nice and serves no purpose in Python either; and while we're at it, I might as well use the chance to recommend you to read PEP8: http://www.python.org/dev/peps/pep-0008/.