Develop Reference

Python run piped command and output results live on the screen

I am trying to run a piped command from python3 and would like the contents printed to the screen. After googling for an hour and reading multiple stackoverflow questions I have not been able to execute the command and also output the contents to the screen live.
I am using subprocess but am up to any solution that accomplishes the task. Security is not a requirement. I will need the ability to run multiple commands one by one in order.
import subprocess
from datetime import datetime
import os
currentDateTime = datetime.today().strftime('%Y%m%d_%H%M%S')
domain = "tesla.com"
waybackurlsDir = "/opt/project/recon/{0}/_waybackurls".format(domain)
os.makedirs(waybackurlsDir)
payload = '/usr/bin/echo "{0}" | /root/go-workspace/bin/waybackurls > {1}/{2}_waybackurls.txt'.format(domain, waybackurlsDir, currentDateTime)
process = subprocess.Popen(payload, shell=True, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''):
sys.stdout.buffer.write(c)
process.buffer.write(c)

Assuming you want this.
Pass parameters to subprocess and capture the output/error of the execution
you can try this
child = subprocess.Popen(['command'], stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=False)
out,err=child.communicate('your parameters')
print(out,err)

I figured out how to solve my problem. Answer is below:
import subprocess
from datetime import datetime
import sys
import os
currentDateTime = datetime.today().strftime('%Y%m%d_%H%M%S')
domain = "tesla.com"
waybackurlsDir = "/opt/myproject/recon/{0}/_waybackurls".format(domain)
if not os.path.exists(waybackurlsDir):
os.makedirs(waybackurlsDir)
p1 = subprocess.Popen('/usr/bin/echo "tesla.com"', shell=True, stdout=subprocess.PIPE)
p2 = subprocess.Popen('waybackurls', shell=True, stdin=p1.stdout, stdout=subprocess.PIPE)
waybackurlsLogFile = '{0}/{1}_waybackurls.txt'.format(waybackurlsDir, currentDateTime)
waybackurlsFile = open(waybackurlsLogFile, "w+")
while p2.poll() is None:
for stdout_line in iter(p2.stdout.readline, ""):
if len(stdout_line) == 0:
break
waybackurlsFile.writelines(p2.stdout.readline().decode("utf-8"))
print("[waybackurls] {0}".format(p2.stdout.readline().rstrip().decode("utf-8")))
waybackurlsFile.close()

Log output of background process to a file

I have time consuming SNMP walk task to perform which I am running as a background process using Popen command. How can I capture the output of this background task in a log file. In the below code, I am trying to do snampwalk on each IP in ip_list and logging all the results to abc.txt. However, I see the generated file abc.txt is empty.
Here is my sample code below -
import subprocess
import sys
f = open('abc.txt', 'a+')
ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]
for ip in ip_list:
cmd = "snmpwalk.exe -t 1 -v2c -c public "
cmd = cmd + ip
print(cmd)
p = subprocess.Popen(cmd, shell=True, stdout=f)
p.wait()
f.close()
print("File output - " + open('abc.txt', 'r').read())
the sample output from the command can be something like this for each IP -
sysDescr.0 = STRING: Software: Whistler Version 5.1 Service Pack 2 (Build 2600)
sysObjectID.0 = OID: win32
sysUpTimeInstance = Timeticks: (15535) 0:02:35.35
sysContact.0 = STRING: unknown
sysName.0 = STRING: UDLDEV
sysLocation.0 = STRING: unknown
sysServices.0 = INTEGER: 72
sysORID.4 = OID: snmpMPDCompliance
I have already tried Popen. But it does not logs output to a file if it is a time consuming background process. However, it works when I try to run background process like ls/dir. Any help is appreciated.

The main issue here is the expectation of what Popen does and how it works I assume.
p.wait() here will wait for the process to finish before continuing, that is why ls for instance works but more time consuming tasks doesn't. And there's nothing flushing the output automatically until you call p.stdout.flush().
The way you've set it up is more meant to work for:
Execute command
Wait for exit
Catch output
And then work with it. For your usecase, you'd better off using an alternative library or use the stdout=subprocess.PIPE and catch it yourself. Which would mean something along the lines of:
import subprocess
import sys
ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]
with open('abc.txt', 'a+') as output:
for ip in ip_list:
print(cmd := f"snmpwalk.exe -t 1 -v2c -c public {ip}")
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) # Be wary of shell=True
while process.poll() is None:
for c in iter(lambda: process.stdout.read(1), ''):
if c != '':
output.write(c)
with open('abc.txt', 'r') as log:
print("File output: " + log.read())
The key things to take away here is process.poll() which checks if the process has finished, if not, we'll try to catch the output with process.stdout.read(1) to read one byte at a time. If you know there's new lines coming, you can switch those three lines to output.write(process.stdout.readline()) and you're all set.

run shell commands from python, and How to store result in a file(text or excell or csv etc..)

import subprocess
from subprocess import Popen
p = Popen(['nmap', '-sn', '***.***.*.*'])
out, err = p.communicate()
print(out)
THE O/p as BELOW:----> IS MUST BE STORE INTO TEXT FILE
Starting Nmap 7.70 ( https://nmap.org ) at 2018-12-11 04:26 *****n Standard Time
Nmap scan report for ***.***.*.*
Host is up (0.00s latency).
MAC Address: **:**:**:**:**:BF (***** *****(******)Co.)
Nmap done: 1 IP address (1 host up) scanned in 3.82 seconds

According to the docs:
Note that if you want to send data to the process’s stdin, you need to
create the Popen object with stdin=PIPE. Similarly, to get anything
other than None in the result tuple, you need to give stdout=PIPE
and/or stderr=PIPE too.
So in your case you should write:
from subprocess import Popen, PIPE
p = Popen(['nmap', '-sn', '***.***.*.*'], stdout = PIPE, stderr = PIPE)
out, err = p.communicate()
This should work.

Your code is sending the output you want to the screen. To send it to a file instead, change this:
print(out)
to
with open("my_output.txt", "w") as outfile:
print(out, file=outfile)

Please check
import subprocess
out = subprocess.check_output(['nmap', '-sn', '***.***.*.*'])
with open("./my.txt", "wb") as f:
f.write(out)

Output manipulation from subprocess

I'd like to manipulate the output from a subprocess call and only print specific details. Now I was originally going down the PIPE route and piping the output to another subprocess of grep but this seemed too limiting, meaning I could only print a few lines (some of which I didn't want), by using the -A or -B grep arguments. So now I'm taking the stdout.readline route. My question is, is there a better way to do achieve what I am trying to do? Is pattern matching with regular expressions and re the best option?
Below is the code for both trails of thought, just to give some context.
PIPE to another subprocess:
def nmap_ips(ip_list):
for ip in ip_list:
cmd1 = subprocess.Popen(["nmap", "-sC", "--open", "-Pn", "-p80,443", ip], stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
cmd2 = subprocess.Popen(["grep", "title", "-A", "1"], stdin=cmd1.stdout)
cmd1.stdout.close()
output = cmd2.communicate()[0]
stdout read lines: #Im not actually getting output from this function at the moment, I'm guessing it's my regex
def nmap_ips(ip_list):
regs = ["title", "commonName"]
combined = "|".join(regs)
for ip in ip_list:
p = subprocess.Popen(["nmap", "-sC", "--open", "-Pn", "-p80,443", ip], stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
ml = re.match(combined, line)
if ml:
print ml.group()
Example of data I am trying to manipulate
Starting Nmap 6.40 ( http://nmap.org ) at 2014-10-10 23:47 BST
Nmap scan report for BTHomeHub.home (192.168.1.254)
Host is up (0.0054s latency).
PORT STATE SERVICE
80/tcp open http
| http-title: Home Hub Manager - Please Reset Your Password
|_Requested resource was http://BTHomeHub.home/html/home/a_firstlogin_configuration.html
443/tcp open https
| ssl-cert: Subject: commonName=bthomehub.home/countryName=UK
| Not valid before: 2011-01-15T16:52:23+00:00
|_Not valid after: 2024-01-17T16:52:23+00:00
|_ssl-date: 2014-10-10T22:47:18+00:00; +14s from local time.

Your approach is fine. Both grep 'title\|commonName' and re.search(b'title|commonName', line) should work.
Note: re.match() starts matching at the begining of its input string. re.search() finds the match anywhere in the string.
If you want to save these lines in a variable then the grep process is unnecessary. It is enough to start only nmap.
from subprocess import Popen, PIPE, STDOUT
def nmap_something(ip_list):
match = re.compile(b"title|commonName").search
p = Popen(["nmap", "-sC", "--open", "-Pn", "-p80,443"] + ip_list,
stdout=PIPE, stderr=STDOUT, bufsize=1)
result = list(filter(match, p.stdout))
p.stdout.close()
p.wait()
return result
I'm not sure about the exact syntax to pass multiple ips to a single nmap child process but it supports it i.e., you don't need to start multiple processes to scan multiple ips.
If you want to print ssl certificates for web servers then you don't need nmap, you could use a pure Python solution e.g., see How to get server's ssl certificate in a human readable form?

After some advice from J.F.Sebastian, I changed to re.search and found printing the line directly achieved what I was after.
def nmap_ips(ip_list):
regs = ["title", "commonName"]
combined = "|".join(regs)
for ip in ip_list:
p = subprocess.Popen(["nmap", "-sC", "--open", "-Pn", "-p80,443", ip], stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
ml = re.search(combined, line)
if ml:
print line

Print SSH output from Cisco Router to a text file

I am a newbie to Python as well as the programming world. After a bit of research for the past 2 days am now able to successfully SSH into the Cisco router and execute set of commands. However my original goal is to print the resultant output to a text file. Checked lots of posts by forum members which helped me in constructing the code, but I couldn't get the result printed on the text file. Please help.
Here is my code:
import paramiko
import sys
import os
dssh = paramiko.SSHClient()
dssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
dssh.connect('10.0.0.1', username='cisco', password='cisco')
stdin, stdout, stderr = dssh.exec_command('sh ip ssh')
print stdout.read()
f = open('output.txt', 'a')
f.write(stdout.read())
f.close()
dssh.close()

stdout.read() will read the content and move the file pointer forward. As such, subsequent calls will not be able to read the content again. So if you want to print the content and write it to a file, you should store it in a variable first and then print and write that.
Instead of mentioning the IP address directly on the code, is it possible for me to fetch it from list of IP addresses (mentioned line by line) in a text file?
You can read lines from a file like this:
with open('filename') as f:
for line in f:
# Each line will be iterated; so you could call a function here
# that does the connection via SSH
print(line)

I know this is very late but the code below is what I'm using to do exactly what is being asked.
from __future__ import print_function
from netmiko import ConnectHandler
import sys
import time
import select
import paramiko
import re
fd = open(r'C:\Users\NewdayTest.txt','w')
old_stdout = sys.stdout
sys.stdout = fd
platform = 'cisco_ios'
username = 'Username'
password = 'Password'
ip_add_file = open(r'C:\Users\\IPAddressList.txt','r')
for host in ip_add_file:
device = ConnectHandler(device_type=platform, ip=host, username=username, password=password)
output = device.send_command('terminal length 0')
output = device.send_command('enable')
print('##############################################################\n')
print('...................CISCO COMMAND SHOW RUN OUTPUT......................\n')
output = device.send_command('sh run')
print(output)
print('##############################################################\n')
print('...................CISCO COMMAND SHOW IP INT BR OUTPUT......................\n')
output = device.send_command('sh ip int br')
print(output)
print('##############################################################\n')
fd.close()
Or, if you wanted to print from a single host, use this slight edit. That simply removes looking for a list to get the IP address:
from __future__ import print_function
from netmiko import ConnectHandler
import sys
import time
import select
import paramiko
import re
fd = open(r'C:\Users\NewdayTest.txt','w')
old_stdout = sys.stdout
sys.stdout = fd
host = '10.10.10.10'
platform = 'cisco_ios'
username = 'Username'
password = 'Password'
device = ConnectHandler(device_type=platform, ip=host, username=username, password=password)
output = device.send_command('terminal length 0')
output = device.send_command('enable')
print('##############################################################\n')
print('...................CISCO COMMAND SHOW RUN OUTPUT......................\n')
output = device.send_command('sh run')
print(output)
print('##############################################################\n')
print('...................CISCO COMMAND SHOW IP INT BR OUTPUT......................\n')
output = device.send_command('sh ip int br')
print(output)
print('##############################################################\n')
fd.close()