so i have gone from a linux working environment to windows during the pandemic working from home.
I am using what was the family computer so cant really change to linux on it.
I am trying to combine two files into a third file on linux i can do this using the following
cat file1 file2 > file3
in python i can do this using subprocess this worked fine at my office
on windows i was told it can be done using type with the following command
type file1 file2 > file3
if i run this manually via cmd it works fine and results in a 7mb file but if i try and run the command using subprocess in python the result is a 165byte file
if i print the stdout i simply get the following
b''
the function i am using to run the sub process is as follows
def RUN_SUBPROCESS(COMMAND, LOG):
print('#### STARTING SUB PROCESS')
SUB_PROCESS = subprocess.Popen(COMMAND, shell=True, stdout=subprocess.PIPE)
out, err = SUB_PROCESS.communicate()
SUB_PROCESS.wait()
if LOG == 'true':
print('#### OUTPUT')
print(out)
print('#### ERROR')
print(err)
print('#### SUB PROCESS COMPLETE')
return
Related
I am working with the subprocess module in Python. I am trying to run a series of terminals to automate a process.
To break it down:
I am suppose to have 3 terminals open to run a set of commands
like so:
Terminal 1: `cd src` -> `./run_script.sh`
Terminal 2: cd data -> `python prepare_data.py`
Terminal 3: `cd src` -> `./do_something.sh` #runs some docker container
Terminal 4: `cd src` -> `./do_another.sh`
Terminal 3: `./another_bash.sh`
To automate this the following:
class AutomateProcesses:
def run_terminal_1(self):
subprocess.call('./run_script.sh', shell=True, cwd='../src')
def run_terminal_2(self):
subprocess.call('python prepare_data.py', shell=True, cwd='../../data')
def run_terminal_3(self):
subprocess.call('./do_something.sh.sh', shell=True, cwd='../src')
def run_terminal_4(self):
subprocess.call('./do_another.sh', shell=True, cwd='../src')
How do I get back to terminal 3 to run the command?
It looks like you want to run several commands on a "terminal" (actually you don't see any terminal), it is just a sub-process that runs a shell.
I use the tool called pexpect (https://pexpect.readthedocs.io/en/latest/overview.html), it has the Windows-variant wexpect (https://pypi.org/project/wexpect/).
Below is the code sample, using the child variable, you can keep the "terminal" and send commands to it.
import pexpect
# log file to capture all the commands sent to the shell and their responses
output_file = open('log.txt','wb')
# create the bash shell sub-process
child = pexpect.spawn('/bin/bash', logfile=output_file)
child.stdout = output_file
child.expect(bytes('>', 'utf-8'))
# make sure you use the pair (sendline() and expect()) to wait until the command finishes
child.sendline(bytes('ls', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
child.sendline(bytes('echo Hello World', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
output_file.close()
I have a cmd file "file.cmd" containing 100s of lines of command.
Example
pandoc --extract-media -f docx -t gfm "sample1.docx" -o "sample1.md"
pandoc --extract-media -f docx -t gfm "sample2.docx" -o "sample2.md"
pandoc --extract-media -f docx -t gfm "sample3.docx" -o "sample3.md"
I am trying to run these commands using a script so that I don't have to go to a file and click on it.
This is my code, and it results in no output:
file1 = open('example.cmd', 'r')
Lines = file1.readlines()
# print(Lines)
for i in Lines:
print(i)
os.system(i)
You don't need to read the cmd file line by line. you can simply try the following:
import os
os.system('myfile.cmd')
or using the subprocess module:
import subprocess
p = subprocess.Popen(['myfile.cmd'], shell = True, close_fds = True)
stdout, stderr = proc.communicate()
Example:
myfile.cmd:
#ECHO OFF
ECHO Grettings From Python!
PAUSE
script.py:
import os
os.system('myfile.cmd')
The cmd will open with:
Greetings From Python!
Press any key to continue ...
You can debug the issue by knowing the return exit code by:
import os
return_code=os.system('myfile.cmd')
assert return_code == 0 #asserts that the return code is 0 indicating success!
Note: os.system works by calling system() in C can only take up to 65533 arguments after a command (so it is a 16 bit issue). Giving one more argument will result in the return code 32512 (which implies the exit code 127).
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function (os.system('command')).
since it is a command file (cmd), and only the shell can run it, then shell argument must set to be true. since you are setting the shell argument to true, the command needs to be string form and not a list.
use the Popen method for spawn a new process and the communicte for waiting on that process (you can time it out as well). if you whish to communicate with the child process, provide the PIPES (see mu example, but you dont have to!)
the code below for python 3.3 and beyond
import subprocess
try:
proc=subprocess.Popen('myfile.cmd', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
outs, errs = proc.communicate(timeout=15) #timing out the execution, just if you want, you dont have to!
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
for older python versions
proc = subprocess.Popen('myfile.cmd', shell=True)
t=10
while proc.poll() is None and t >= 0:
print('Still waiting')
time.sleep(1)
t -= 1
proc.kill()
In both cases (python versions) if you dont need the timeout feature and you dont need to interact with the child process, then just, use:
proc = subprocess.Popen('myfile.cmd', shell=True)
proc.communicate()
I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')
I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code
So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)
Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log
I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect
I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)
I would like to know how to execute this java process using the windows command line, from inside Python 2.7 on Windows 8.
I thought I had already solved this problem, but I recently changed computers from Windows 7 to Windows 8 and my code stopped working. I have confirmed that the windows command used in the script below executes properly when run directly from cmd.exe
import os
import subprocess
def FileProcess(inFile):
#Create the startup info so the java program runs in the background (for windows computers)
startupinfo = None
if os.name == 'nt':
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
#Execute Stanford Core NLP from the command line
print inFile
cmd = ['java', '-Xmx1g','-cp', 'stanford-corenlp-1.3.5.jar;stanford-corenlp-1.3.5-models.jar;xom.jar;joda-time.jar', 'edu.stanford.nlp.pipeline.StanfordCoreNLP', '-annotators', 'tokenize,ssplit,pos,parse', '-file', inFile]
output = subprocess.call(cmd, startupinfo=startupinfo)
print inFile[(str(inFile).rfind('\\'))+1:] + '.xml'
outFile = file(inFile[(str(inFile).rfind('\\'))+1:] + '.xml')
FileProcess("C:\\NSF_Stuff\\ErrorPropagationPaper\\RandomTuftsPlain\\PreprocessedTufts8199PLAIN.txt")
When this code is executed, I receive the error message that the output file does not exist. The java process I am executing should output an xml file when it is done.
It is my belief that for some reason subprocess.call is never successfully executing the command. I have tried using subprocesss.popen for the same task and I get the same results.
EDIT: I have changed my code so that I can capture error messages and I think I am beginning to understand the problem.
I changed my code to
import os
import subprocess
def FileProcess(inFile):
#Create the startup info so the java program runs in the background (for windows computers)
startupinfo = None
if os.name == 'nt':
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
#Execute Stanford Core NLP from the command line
print inFile
cmd = ['java', '-Xmx1g','-cp', 'stanford-corenlp-1.3.5.jar;stanford-corenlp-1.3.5-models.jar;xom.jar;joda-time.jar', 'edu.stanford.nlp.pipeline.StanfordCoreNLP', '-annotators', 'tokenize,ssplit,pos,parse', '-file', inFile]
proc = subprocess.Popen(cmd, stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=True)
print proc
stdoutdata, stderrdata = proc.communicate()
print stdoutdata
print stderrdata
outFile = file(inFile[(str(inFile).rfind('\\'))+1:] + '.xml')
FileProcess("C:\\NSF_Stuff\\ErrorPropagationPaper\\RandomTuftsPlain\\PreprocessedTufts8199PLAIN.txt")
stdoutdata contains the message "'java' is not recognized as an internal or external command, operable program or batch file."
Now this is a very bizarre message because java is definitely a recognized command when I run it from the cmd.exe . There is some issue here where executing the command from python is messing with my system environment variables such that java is no longer recognized as a command.
I was able to solve my problem by adding the location of java to my PATH variable. Apparently java wasn't in my path variable. I didn't even bother checking this originally because I was having no problems executing java commands from the windows command line. I'm guessing that commands executed directly from cmd.exe use a different environment variable to find the java executable than commands executed indirectly from the subprocess module.
By trying your code it prints out PreprocessedTufts8199PLAIN.txt.xml file name. I'm not sure if the .txt.xml extension was the desired result. If your file has only .xml extension, then you're not stripping away the original .txt header.
Try to change this line:
outFile = file(inFile[(str(inFile).rfind('\\'))+1:] + '.xml')
Into this code:
fnameext = inFile[(str(inFile).rfind('\\'))+1:]
fname,fext = os.path.splitext(fnameext)
xmlfname = fname + '.xml'
xmlfpath = os.path.join(".", xmlfname)
print "xmlfname:", xmlfname, " xmlfpath:", xmlfpath
print "current working directory:", os.getcwd()
outFile = open(xmlfpath, "r")
Answer for extension stripping.