Develop Reference

How to fix Variables send not the good Values [duplicate]

This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to assign the output of a command I run using os.system to a variable and prevent it from being output to the screen. But, in the below code ,the output is sent to the screen and the value printed for var is 0, which I guess signifies whether the command ran successfully or not. Is there any way to assign the command output to the variable and also stop it from being displayed on the screen?
var = os.system("cat /etc/services")
print var #Prints 0

From this question which I asked a long time ago, what you may want to use is popen:
os.popen('cat /etc/services').read()
From the docs for Python 3.6,
This is implemented using subprocess.Popen; see that class’s
documentation for more powerful ways to manage and communicate with
subprocesses.
Here's the corresponding code for subprocess:
import subprocess
proc = subprocess.Popen(["cat", "/etc/services"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
print("program output:", out)

You might also want to look at the subprocess module, which was built to replace the whole family of Python popen-type calls.
import subprocess
output = subprocess.check_output("cat /etc/services", shell=True)
The advantage it has is that there is a ton of flexibility with how you invoke commands, where the standard in/out/error streams are connected, etc.

The commands module is a reasonably high-level way to do this:
import commands
status, output = commands.getstatusoutput("cat /etc/services")
status is 0, output is the contents of /etc/services.

For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code. Since you are only interested in the output, you can write a utility wrapper like this.
from subprocess import PIPE, run
def out(command):
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
return result.stdout
my_output = out("echo hello world")
# Or
my_output = out(["echo", "hello world"])

I know this has already been answered, but I wanted to share a potentially better looking way to call Popen via the use of from x import x and functions:
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
print cmdline("cat /etc/services")
print cmdline('ls')
print cmdline('rpm -qa | grep "php"')
print cmdline('nslookup google.com')

I do it with os.system temp file:
import tempfile, os
def readcmd(cmd):
ftmp = tempfile.NamedTemporaryFile(suffix='.out', prefix='tmp', delete=False)
fpath = ftmp.name
if os.name=="nt":
fpath = fpath.replace("/","\\") # forwin
ftmp.close()
os.system(cmd + " > " + fpath)
data = ""
with open(fpath, 'r') as file:
data = file.read()
file.close()
os.remove(fpath)
return data

Python 2.6 and 3 specifically say to avoid using PIPE for stdout and stderr.
The correct way is
import subprocess
# must create a file object to store the output. Here we are getting
# the ssid we are connected to
outfile = open('/tmp/ssid', 'w');
status = subprocess.Popen(["iwgetid"], bufsize=0, stdout=outfile)
outfile.close()
# now operate on the file

from os import system, remove
from uuid import uuid4
def bash_(shell_command: str) -> tuple:
"""
:param shell_command: your shell command
:return: ( 1 | 0, stdout)
"""
logfile: str = '/tmp/%s' % uuid4().hex
err: int = system('%s &> %s' % (shell_command, logfile))
out: str = open(logfile, 'r').read()
remove(logfile)
return err, out
# Example:
print(bash_('cat /usr/bin/vi | wc -l'))
>>> (0, '3296\n')```

Log output of background process to a file

I have time consuming SNMP walk task to perform which I am running as a background process using Popen command. How can I capture the output of this background task in a log file. In the below code, I am trying to do snampwalk on each IP in ip_list and logging all the results to abc.txt. However, I see the generated file abc.txt is empty.
Here is my sample code below -
import subprocess
import sys
f = open('abc.txt', 'a+')
ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]
for ip in ip_list:
cmd = "snmpwalk.exe -t 1 -v2c -c public "
cmd = cmd + ip
print(cmd)
p = subprocess.Popen(cmd, shell=True, stdout=f)
p.wait()
f.close()
print("File output - " + open('abc.txt', 'r').read())
the sample output from the command can be something like this for each IP -
sysDescr.0 = STRING: Software: Whistler Version 5.1 Service Pack 2 (Build 2600)
sysObjectID.0 = OID: win32
sysUpTimeInstance = Timeticks: (15535) 0:02:35.35
sysContact.0 = STRING: unknown
sysName.0 = STRING: UDLDEV
sysLocation.0 = STRING: unknown
sysServices.0 = INTEGER: 72
sysORID.4 = OID: snmpMPDCompliance
I have already tried Popen. But it does not logs output to a file if it is a time consuming background process. However, it works when I try to run background process like ls/dir. Any help is appreciated.

The main issue here is the expectation of what Popen does and how it works I assume.
p.wait() here will wait for the process to finish before continuing, that is why ls for instance works but more time consuming tasks doesn't. And there's nothing flushing the output automatically until you call p.stdout.flush().
The way you've set it up is more meant to work for:
Execute command
Wait for exit
Catch output
And then work with it. For your usecase, you'd better off using an alternative library or use the stdout=subprocess.PIPE and catch it yourself. Which would mean something along the lines of:
import subprocess
import sys
ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]
with open('abc.txt', 'a+') as output:
for ip in ip_list:
print(cmd := f"snmpwalk.exe -t 1 -v2c -c public {ip}")
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) # Be wary of shell=True
while process.poll() is None:
for c in iter(lambda: process.stdout.read(1), ''):
if c != '':
output.write(c)
with open('abc.txt', 'r') as log:
print("File output: " + log.read())
The key things to take away here is process.poll() which checks if the process has finished, if not, we'll try to catch the output with process.stdout.read(1) to read one byte at a time. If you know there's new lines coming, you can switch those three lines to output.write(process.stdout.readline()) and you're all set.

Redirecting shell command output to a file does not work using subprocess.Popen in Python

I am using Python 2.6.6 and failed to re-direct the Beeline(Hive) SQL query output returning multiple rows to a file on Unix using ">". For simplicity's sake, I replaced the SQL query with simple "ls" command on current directory and outputting to a text file.
Please ignore syntax of function sendfile. I want help to tweak the function "callcmd" to pipe the stdout onto the text file.
def callcmd(cmd, shl):
logging.info('> '+' '.join(map(str,cmd)))
#return 0;
start_time = time.time()
command_process = subprocess.Popen(cmd, shell=shl, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
command_output = command_process.communicate()[0]
logging.info(command_output)
elapsed_time = time.time() - start_time
logging.info(time.strftime("%H:%M:%S",time.gmtime(elapsed_time))+' = time to complete (hh:mm:ss)')
if (command_process.returncode != 0):
logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
logging.error('ERROR CODE: '+str(ret_code))
return command_process.returncode
cmd=['ls', ' >', '/home/input/xyz.txt']
ret_code = callcmd(cmd, False)

Your command (i.e. cmd) could be ['sh', '-c', 'ls > ~/xyz.txt']. That would mean that the output of ls is never passed to Python, it happens entirely in the spawned shell – so you can't log the output. In that case, I'd have used return_code = subprocess.call(cmd), no need for Popen and communicate.
Equivalently, assuming you use bash or similar, you can simply use
subprocess.call('ls > ~/test.txt', shell=True)
If you want to access the output, e.g. for logging, you could use
s = subprocess.check_output(['ls'])
and then write that to a file like you would regularly in Python. To check for a non-zero exit code, handle the CalledProcessError that is raised in such cases.

Here the stdout in command_output is written to a file. You don't need to use any redirection although an alternative might be to have the python print to stdout, and then you would redirect that in your shell to a file.
#!/usr/bin/python
import subprocess
cmd=['ls']
command_process = subprocess.Popen(
cmd,
shell='/bin/bash',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True
)
command_output = command_process.communicate()[0]
if (command_process.returncode != 0):
logging.error('ERROR ON COMMAND: '+' '.join(map(str,cmd)))
logging.error('ERROR CODE: '+str(ret_code))
f = open('listing.txt','w')
f.write(command_output)
f.close()

I added this piece of code to my code and It works fine.Thanks to #Snohdo
f = open('listing.txt','w')
f.write(command_output)
f.close()

Python redirect subprocess to file and to console [duplicate]

This question already has answers here:
Python subprocess get children's output to file and terminal?
(2 answers)
Closed 7 years ago.
Using Python, I want to create a subprocess and have its output redirected to both a file AND the console.
I found this post explaining how to print to both console and file, but solutions does not work when creating a subprocess:
sys.stdout = Logger()
print( "Hello") # printed to console and file
res = subprocess.call(cmd) # command output is printed to console only
Same behaviour here:
with Tee('outfile.log', 'w'):
print( "Hello" )
res = subprocess.call(cmd)
How can I redirect subprocess output both to console (for user) and to file (for me to check it from my code).
Note: Im' on Windows, so using system's tee is not appropriate.

Connect cmd's standard output to tee's stdin:
# cmd = ['/bin/echo', 'blah blah']
tee = subprocess.Popen(['/usr/bin/tee', 'outfile.log'], stdin=subprocess.PIPE)
subprocess.call(cmd, stdout=tee.stdin)
tee.stdin.close()
UPDATE For systems that does not have tee, read the output of the command and write it into stdout, and the log file:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
with open('outfile.log', 'w') as f:
while True:
data = proc.stdout.read(1024)
if not data:
break
sys.stdout.write(data)
f.write(data)

Executing shell command in python with file as stdin

In my Python code, I have
executable_filepath = '/home/user/executable'
input_filepath = '/home/user/file.in'
I want to analyze the output I would get in shell from command
/home/user/executable </home/user/file.in
I tried
command = executable_filepath + ' <' + input_filepath
p = subprocess.Popen([command], stdout=subprocess.PIPE)
p.wait()
output = p.stdout.read()
but it doesn't work. The only solution that I can think of now is creating another pipe, and copying input file through it, but there must be a simple way.

from subprocess import check_output
with open("/home/user/file.in", "rb") as file:
output = check_output(["/home/user/executable"], stdin=file)

You need to specify shell=True in the call to Popen. By default, [command] is passed directly to a system call in the exec family, which doesn't understand shell redirection operators.
Alternatively, you can let Popen connect the process to the file:
with open(input_filepath, 'r') as input_fh:
p = subprocess.Popen( [executable_filepath], stdout=subprocess.PIPE, stdin=input_fh)
p.wait()
output=p.stdout.read()