Develop Reference

Redirecting shell command output to a file does not work using subprocess.Popen in Python

I am using Python 2.6.6 and failed to re-direct the Beeline(Hive) SQL query output returning multiple rows to a file on Unix using ">". For simplicity's sake, I replaced the SQL query with simple "ls" command on current directory and outputting to a text file.
Please ignore syntax of function sendfile. I want help to tweak the function "callcmd" to pipe the stdout onto the text file.
def callcmd(cmd, shl):
logging.info('> '+' '.join(map(str,cmd)))
#return 0;
start_time = time.time()
command_process = subprocess.Popen(cmd, shell=shl, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
command_output = command_process.communicate()[0]
logging.info(command_output)
elapsed_time = time.time() - start_time
logging.info(time.strftime("%H:%M:%S",time.gmtime(elapsed_time))+' = time to complete (hh:mm:ss)')
if (command_process.returncode != 0):
logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
logging.error('ERROR CODE: '+str(ret_code))
return command_process.returncode
cmd=['ls', ' >', '/home/input/xyz.txt']
ret_code = callcmd(cmd, False)

Your command (i.e. cmd) could be ['sh', '-c', 'ls > ~/xyz.txt']. That would mean that the output of ls is never passed to Python, it happens entirely in the spawned shell – so you can't log the output. In that case, I'd have used return_code = subprocess.call(cmd), no need for Popen and communicate.
Equivalently, assuming you use bash or similar, you can simply use
subprocess.call('ls > ~/test.txt', shell=True)
If you want to access the output, e.g. for logging, you could use
s = subprocess.check_output(['ls'])
and then write that to a file like you would regularly in Python. To check for a non-zero exit code, handle the CalledProcessError that is raised in such cases.

Here the stdout in command_output is written to a file. You don't need to use any redirection although an alternative might be to have the python print to stdout, and then you would redirect that in your shell to a file.
#!/usr/bin/python
import subprocess
cmd=['ls']
command_process = subprocess.Popen(
cmd,
shell='/bin/bash',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True
)
command_output = command_process.communicate()[0]
if (command_process.returncode != 0):
logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
logging.error('ERROR CODE: '+str(ret_code))
f = open('listing.txt','w')
f.write(command_output)
f.close()

I added this piece of code to my code and It works fine.Thanks to #Snohdo
f = open('listing.txt','w')
f.write(command_output)
f.close()

Python subprocess call to xpdf's pdftotext not working with encoding

I am trying to run pdftotext using python subprocess module.
import subprocess
pdf = r"path\to\file.pdf"
txt = r"path\to\out.txt"
pdftotext = r"path\to\pdftotext.exe"
cmd = [pdftotext, pdf, txt, '-enc UTF-8']
response = subprocess.check_output(cmd,
shell=True,
stderr=subprocess.STDOUT)
TB
CalledProcessError: Command '['path\\to\\pdftotext.exe',
'path\\to\\file.pdf', 'path\\to\\out.txt', '-enc UTF-8']'
returned non-zero exit status 99
When I remove last argument '-enc UTF-8' from cmd, it works OK in python.
When I run pdftotext pdf txt -enc UTF-8 in cmd, it works ok.
What I am missing?
Thanks.

subprocess has some complicated rules for handling commands. From the docs:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
More details explained in this answer here.
So, as the docs explain, you should convert your command to a string:
cmd = r"""{} "{}" "{}" -enc UTF-8""".format('pdftotext', pdf, txt)
Now, call subprocess as:
subprocess.call(cmd, shell=True, stderr=subprocess.STDOUT)

Subprocess wont take the output of dxl script

Im using Python's subprocess module to run a dxl script. My Problem is when i try to catch the Output (In this example a print-statement or a error message) of my dxl script, it is shown in the command prompt, but when i try to catch it with stdout=subprocess.PIPE or subprocess.check_output it always returns an empty string. Is there a way to catch the output or how could I get the Error messages from Doors?
It's important that you dont see the GUI of DOORS.
Here is my quick example that shows my problem:
test.dxl
print "Hello World"
test.py
import subprocess
doorsPath = "C:\\Program Files (x86)\\IBM\\Rational\\DOORS\\9.5\\bin\\doors.exe"
userInfo = ' -user dude -password 1234 -d 127.0.0.1 -batch ".\\test.dxl"'
dxl = " -W"
output = subprocess.check_output(doorsPath+dxl+userInfo)
print(output)
Edit: Using Windows 7 , DOORS 9.5 and Python 2.7

I know this post is pretty old, but the solution to the problem is to use
cout << ... instead of print. You can override the print perms like shown here
DOORS Print Redirect Tutorial for print, cout and logfiles

I'm feeling lucky here,
change print "Hello World" to cout << "Hello World"
and userInfo = ' -user dude -password 1234 -d 127.0.0.1 -batch ".\\test.dxl > D:\output.txt"', as in cmd promt the text can be directly exported to a text file.

your script have many error try this link for example for subprocess
and try this :
import subprocess
import sys
path = "C:\\Program Files(x86)\\IBM\\Rational\\DOORS\\9.5\\bin\\doors.exe"
userInfo = "C:\\Program Files (x86)\\IBM\\Rational\\DOORS\\9.5\\bin\\doors.exe"
proc = subprocess.Popen([path,userInfo,"-W"])
proc.communicate()
i hape it work on your system!

How to print the output of shell instantly by python script

I executed some commands in shell with python. I need to show the command response in shell. But the commands will execute 10s . I need to wait. How can I show the echo of the commands instantly. Following is my code
cmd = "commands"
output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
print(output.stdout.read())
And I need to use the output of the command. so I can't use subprocess.call

Read from output.stdout in a loop:
cmd = "commands"
output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in output.stdout:
print(line)
edit: seems then in python2 this still doesn't work in evey case, but this will:
for line in iter(output.stdout.readline, ''):
print(line)

Failing to capture stdout from application

I have the following script:
import subprocess
arguments = ["d:\\simulator","2332.txt","2332.log", "-c"]
output=subprocess.Popen(arguments, stdout=subprocess.PIPE).communicate()[0]
print(output)
which gives me b'' as output.
I also tried this script:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE)
process.wait()
print(process.stdout.read())
print("ERROR:" + str(process.stderr))
which gives me the output: b'', ERROR:None
However when I run this at the cmd prompt I get a 5 lines of text.
d:\simulator atp2332.txt atp2332.log -c
I have added to simulator a message box which pops up when it launches. This is presented for all three cases. So I know that I sucessfully launch the simulator. However the python scripts are not caturing the stdout.
What am I doing wrong?
Barry.

If possible (not endless stream of data) you should use communicate() as noted on the page.
Try this:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sout, serr = process.communicate()
print(sout)
print(serr)

The following code gives me text output on stdout.
Perhaps you could try it, and then substitute your command for help
import subprocess
arguments = ["help","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.wait()
print 'Return code', process.returncode
print('stdout:', process.stdout.read())
print("stderr:" + process.stderr.read())