Develop Reference

Parse filename from Dumpcap output

I am trying to parse the filename from the ouput of running dumpcap in the terminal in linux in order to automatically attach it to an email. This is the relevant functions from a larger script. proc1, stdout, and eventfile are initialized to"" and DUMPCAP is the command line string dumpcap -a duration:300 -b duration:2147483647 -c 500 -i 1 -n -p -s 2 -w test -B 20
def startdump():
global DUMPCAP
global dumpdirectory
global proc1
global stdout
global eventfile
setDumpcapOptions()
print("dumpcap.exe = " + DUMPCAP)
os.chdir(dumpdirectory)
#subprocess.CREATE_NEW_CONSOLE
proc1 = subprocess.Popen(DUMPCAP, shell=True, stdout=subprocess.PIPE)
for line in proc1:
if 'File: ' in line:
parsedfile = line.split(':')
eventfile = parsedfile[1]
if dc_mode == "Dumpcap Only":
proc1.communicate()
mail_man(config_file)
return proc1
def startevent():
global EVENT
global proc1
global eventfile
setEventOptions()
print(EVENT)
# subprocess.CREATE_NEW_CONSOLE
proc2 = subprocess.Popen(EVENT, shell=True, stdout=subprocess.PIPE)
if dc_mode == "Dumpcap+Event" or dc_mode == "Trigger" or dc_mode == "Event Only":
proc2 = proc1.communicate()
mail_man(config_file)
return proc2
the problem I keep having is that I can't figure out how to parse the file name from the output of dumpcap. It keeps parsing ""from the output no matter what I do. I apologize if this seems unresearched. I am a month into learning python and linux on my own and the documentation is terse and confusing online.
Should I create a function to parse the eventfile from dumpcap's output or do it right there in the script? I'm truly at a loss here. I'm not sure how dumpcap stores its output either.
The output of dumcap in the terminal is:
dumpcap.exe = dumpcap -a duration:300 -b duration:2147483647 -c 500 -i 1 -n -p -s 2 -w test -B 20
-i 1 - f icmp and host 156.24.31.29 - c 2
/bin/sh: -i: command not found
Capturing on 'eth0'
File: test_00001_20150714141827
Packets captured: 500
Packets received/dropped on interface 'eth0': 500/0 (pcap:0/dumpcap:0/flushed:0/ps_ifdrop:0) (100.0%)
[Errno 2] No such file or directory: ''
the line File: ... contains the randomly generated name of the pcap file saved by dumpcap I am trying to parse that line from the terminal to get everything after the File: set to a variable but the conventional .split method doesn't seem to be working
The other error it gives is that Popen cannot be indexed

It looks like basically you need a regexp.
import re
rx = re.compile('^File: (\S+)$', re.MULTILINE)
m = rx.search(stdout_contents)
if m:
file_name = m.group(1)
# else: file name not detected.
Update: a minimal complete example of reading pipe's stdout; hope this helps.
import subprocess
proc = subprocess.Popen("echo foo", shell=True, stdout=subprocess.PIPE)
result = proc.communicate()
print result[0] # must be `foo\n`

Dumpcap outputs to its stderr as opposed to stdout. So I've managed to redirect the stderr to a txt file which I can then parse.
def startdump():
global DUMPCAP, dumpdirectory, proc1
global eventfile, dc_capfile
DUMPCAP = ''
print("========================[ MAIN DUMPCAP MONITORING ]===========================")
setDumpcapOptions()
os.chdir(dumpdirectory)
if platform == "Linux":
DUMPCAP = "dumpcap " + DUMPCAP
elif platform == "Windows":
DUMPCAP = "dumpcap.exe " + DUMPCAP
proc1 = subprocess.Popen(DUMPCAP, shell=True, stderr=subprocess.PIPE)
#procPID = proc1.pid
if dc_mode == "Dumpcap Only":
time.sleep(5)
with open("proc1stderr.txt", 'w+') as proc1stderr:
proc1stderr.write(str(proc1.stderr))
for line in proc1.stderr:
print("%s" % line)
if "File:" in line:
print(line)
raweventfile = line.split('File: ')[1]
eventfile = raweventfile.strip('\[]').rstrip('\n')
mail_man()
proc1.communicate()

Linux Command for Perl in Python

I am trying to this Linux command for Perl (v5.10.1) in Python (v2.6)
perl tilt.pl *.pdb > final.txt
What do I do so that I can apply the Perl script to every PDB file, examples would be best?
my current script is this:
import shlex, subprocess
arg_str = "perl tilt.pl frames > final.txt"
arg = shlex.split(arg_str)
print(arg)
import os
framespdb = os.listdir("prac_frames")
for frames in framespdb:
subprocess.Popen(arg, stdout=True)

You can use shell=True
BTW: I think you try to put variable frames in place of text frames in command so I use %s and line % frames to do this.
import os
import subprocess
line = "perl tilt.pl %s > final.txt"
framespdb = os.listdir("prac_frames")
for frames in framespdb:
cmd = line % frames
print(cmd)
subprocess.Popen(cmd, shell=True)
EDIT: if you need results in different files you can use again %s to add unique filename to command - for example:
import os
import subprocess
line = "perl tilt.pl %s > final-%s.txt" # second `%s`
framespdb = os.listdir("prac_frames")
for frames in framespdb:
cmd = line % (frames, frames) # second `frames`
print(cmd)
subprocess.Popen(cmd, shell=True)
EDIT: normally in shell you can send result on screen or redirect it to file > file.txt. To get text on screen and save it in file you need shell command tee and python command subprocess.check_output():
import os
import subprocess
line = "perl tilt.pl %s | tee final-%s.txt"
framespdb = os.listdir("prac_frames")
for frames in framespdb:
cmd = line % (frames, frames)
print(cmd)
output = subprocess.check_output(cmd, shell=True)
print 'output:', output

You can call a subprocess like from the shell if you override the shell parameter:
subprocess.call(arg_str, shell=True)

Python - how do i add the long command to the subprocess, when i am applything in one line the whole command is not working

How do i add this full export DISPLAY=:0.0 && python /home/src/upload.py %s %s & command to the subprocess?
# valid command
u ="""export DISPLAY=:0.0 && python /home/src/upload.py %s %s &""" % (s[1], s[2])
d ="""export DISPLAY=:0.0 && python /home/src/download.py %s %s &""" % (s[1], s[2])
# seems to work like this but need to add the u and d on this
subprocess.Popen(["python","/home/src/upload.py"], stdout=subprocess.PIPE)
subprocess.Popen(["python","/home/src/download.py"], stdout=subprocess.PIPE)

You would add the DISPLAY variable to the environment variables; the arguments are extre elements of the argument list:
import os
env = os.environ.copy()
env['DISPLAY'] = ':0.0'
p1 = subprocess.Popen(["python","/home/src/upload.py"] + s[1:3],
env=env, stdout=subprocess.PIPE)
p2 = subprocess.Popen(["python","/home/src/download.py"] + s[1:3],
env=env, stdout=subprocess.PIPE)

What your commands basically do is setting an environment variable and then invoking a python script with parameters. You can achieve this by modifying your code to read:
import os
import subprocess
new_env = os.environ.copy()
new_env['DISPLAY'] = ':0.0'
subprocess.Popen(["python","/home/src/upload.py", s[1], s[2]], stdout=subprocess.PIPE, env=new_env)
subprocess.Popen(["python","/home/src/download.py", s[1], s[2]], stdout=subprocess.PIPE, env=new_env)

python command substitution for linux commands

I am trying to use command substitution for building a linux command from a python script, but am not able to get the following simple example to work:
LS="/bin/ls -l"
FILENAME="inventory.txt"
cmd = "_LS _FILENAME "
ps= subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = ps.communicate()[0]
print output
Thanks!
JB

Use string substitution:
cmd = '{} {}'.format(LS, FILENAME)
or (in Python2.6):
cmd = '{0} {1}'.format(LS, FILENAME)
import subprocess
import shlex
LS="/bin/ls -l"
FILENAME="inventory.txt"
cmd = '{} {}'.format(LS, FILENAME)
ps = subprocess.Popen(shlex.split(cmd),
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT)
output, err = ps.communicate()
print(output)
Or, using the sh module:
import sh
FILENAME = 'inventory.txt'
print(sh.ls('-l', FILENAME, _err_to_out=True))

Interacting with bash from python

I've been playing around with Python's subprocess module and I wanted to do an "interactive session" with bash from python. I want to be able to read bash output/write commands from Python just like I do on a terminal emulator. I guess a code example explains it better:
>>> proc = subprocess.Popen(['/bin/bash'])
>>> proc.communicate()
('user#machine:~/','')
>>> proc.communicate('ls\n')
('file1 file2 file3','')
(obviously, it doesn't work that way.) Is something like this possible, and how?
Thanks a lot

Try with this example:
import subprocess
proc = subprocess.Popen(['/bin/bash'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stdout = proc.communicate('ls -lash')
print stdout
You have to read more about stdin, stdout and stderr. This looks like good lecture: http://www.doughellmann.com/PyMOTW/subprocess/
EDIT:
Another example:
>>> process = subprocess.Popen(['/bin/bash'], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.stdin.write('echo it works!\n')
>>> process.stdout.readline()
'it works!\n'
>>> process.stdin.write('date\n')
>>> process.stdout.readline()
'wto, 13 mar 2012, 17:25:35 CET\n'
>>>

Use this example in my other answer: https://stackoverflow.com/a/43012138/3555925
You can get more details in that answer.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys
import select
import termios
import tty
import pty
from subprocess import Popen
command = 'bash'
# command = 'docker run -it --rm centos /bin/bash'.split()
# save original tty setting then set it to raw mode
old_tty = termios.tcgetattr(sys.stdin)
tty.setraw(sys.stdin.fileno())
# open pseudo-terminal to interact with subprocess
master_fd, slave_fd = pty.openpty()
# use os.setsid() make it run in a new process group, or bash job control will not be enabled
p = Popen(command,
preexec_fn=os.setsid,
stdin=slave_fd,
stdout=slave_fd,
stderr=slave_fd,
universal_newlines=True)
while p.poll() is None:
r, w, e = select.select([sys.stdin, master_fd], [], [])
if sys.stdin in r:
d = os.read(sys.stdin.fileno(), 10240)
os.write(master_fd, d)
elif master_fd in r:
o = os.read(master_fd, 10240)
if o:
os.write(sys.stdout.fileno(), o)
# restore tty settings back
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_tty)

This should be what you want
import subprocess
import threading
p = subprocess.Popen(["bash"], stderr=subprocess.PIPE,shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
exit = False
def read_stdout():
while not exit:
msg = p.stdout.readline()
print("stdout: ", msg.decode())
def read_stderro():
while not exit:
msg = p.stderr.readline()
print("stderr: ", msg.decode())
threading.Thread(target=read_stdout).start()
threading.Thread(target=read_stderro).start()
while not exit:
res = input(">")
p.stdin.write((res + '\n').encode())
p.stdin.flush()
Test result:
>ls
>stdout: 1.py
stdout: 2.py
ssss
>stderr: bash: line 2: ssss: command not found

An interactive bash process expects to be interacting with a tty. To create a pseudo-terminal, use os.openpty(). This will return a slave_fd file descriptor that you can use to open files for stdin, stdout, and stderr. You can then write to and read from master_fd to interact with your process. Note that if you're doing even mildly complex interaction, you'll also want to use the select module to make sure that you don't deadlock.

I wrote a module to facilitate the interaction between *nix shell and python.
def execute(cmd):
if not _DEBUG_MODE:
## Use bash; the default is sh
print 'Output of command ' + cmd + ' :'
subprocess.call(cmd, shell=True, executable='/bin/bash')
print ''
else:
print 'The command is ' + cmd
print ''
Check out the whole stuff at github: https://github.com/jerryzhujian9/ez.py/blob/master/ez/easyshell.py