I have the following code
import subprocess
import re
from itertools import *
command = ['ffprobe', '-i', '/media/some_file.mp4']
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
text = p.stderr.read()
retcode = p.wait()
text = text.decode('utf-8')
p = re.compile("Duration(.*)")
num = 0 #for debugging
for line in iter(text.splitlines()):
print(str(num) + line) #for debugging
m = p.match(str(line))
if m != None:
print(m.group(1))
When I look at the output there is a line that says "Duration" on it, however it is not captured, print(m.group(1)) is never reached. If I change the text variable to a hardcoded string of "Duration blahblah" I get " blahblah", which is what I expect. It seems like the regex doesn't recognize the text coming back from stderr. How can I get the text into a format that the regex will recognize and match on?
I have come up with the following solution, should it help anyone else attempting to capture duration from ffmpeg using python
import subprocess
import re
command = ['ffprobe', '-i', '/media/some_file.mp4']
p = subprocess.Popen(command, stderr=subprocess.PIPE)
text = p.stderr.read()
retcode = p.wait()
text = text.decode('utf-8')
p = re.compile(".*Duration:\s([0-9:\.]*),", re.MULTILINE|re.DOTALL)
m = p.match(text)
print(m.group(1))
p = re.compile(r".*?Duration(.*)")
Try this.match starts from the begining while there may might be something before duration.
Related
In the program I maintain it is done as in:
# count the files in the archive
length = 0
command = ur'"%s" l -slt "%s"' % (u'path/to/7z.exe', srcFile)
ins, err = Popen(command, stdout=PIPE, stdin=PIPE,
startupinfo=startupinfo).communicate()
ins = StringIO.StringIO(ins)
for line in ins: length += 1
ins.close()
Is it really the only way ? I can't seem to find any other command but it seems a bit odd that I can't just ask for the number of files
What about error checking ? Would it be enough to modify this to:
proc = Popen(command, stdout=PIPE, stdin=PIPE,
startupinfo=startupinfo)
out = proc.stdout
# ... count
returncode = proc.wait()
if returncode:
raise Exception(u'Failed reading number of files from ' + srcFile)
or should I actually parse the output of Popen ?
EDIT: interested in 7z, rar, zip archives (that are supported by 7z.exe) - but 7z and zip would be enough for starters
To count the number of archive members in a zip archive in Python:
#!/usr/bin/env python
import sys
from contextlib import closing
from zipfile import ZipFile
with closing(ZipFile(sys.argv[1])) as archive:
count = len(archive.infolist())
print(count)
It may use zlib, bz2, lzma modules if available, to decompress the archive.
To count the number of regular files in a tar archive:
#!/usr/bin/env python
import sys
import tarfile
with tarfile.open(sys.argv[1]) as archive:
count = sum(1 for member in archive if member.isreg())
print(count)
It may support gzip, bz2 and lzma compression depending on version of Python.
You could find a 3rd-party module that would provide a similar functionality for 7z archives.
To get the number of files in an archive using 7z utility:
import os
import subprocess
def count_files_7z(archive):
s = subprocess.check_output(["7z", "l", archive], env=dict(os.environ, LC_ALL="C"))
return int(re.search(br'(\d+)\s+files,\s+\d+\s+folders$', s).group(1))
Here's version that may use less memory if there are many files in the archive:
import os
import re
from subprocess import Popen, PIPE, CalledProcessError
def count_files_7z(archive):
command = ["7z", "l", archive]
p = Popen(command, stdout=PIPE, bufsize=1, env=dict(os.environ, LC_ALL="C"))
with p.stdout:
for line in p.stdout:
if line.startswith(b'Error:'): # found error
error = line + b"".join(p.stdout)
raise CalledProcessError(p.wait(), command, error)
returncode = p.wait()
assert returncode == 0
return int(re.search(br'(\d+)\s+files,\s+\d+\s+folders', line).group(1))
Example:
import sys
try:
print(count_files_7z(sys.argv[1]))
except CalledProcessError as e:
getattr(sys.stderr, 'buffer', sys.stderr).write(e.output)
sys.exit(e.returncode)
To count the number of lines in the output of a generic subprocess:
from functools import partial
from subprocess import Popen, PIPE, CalledProcessError
p = Popen(command, stdout=PIPE, bufsize=-1)
with p.stdout:
read_chunk = partial(p.stdout.read, 1 << 15)
count = sum(chunk.count(b'\n') for chunk in iter(read_chunk, b''))
if p.wait() != 0:
raise CalledProcessError(p.returncode, command)
print(count)
It supports unlimited output.
Could you explain why buffsize=-1 (as opposed to buffsize=1 in your previous answer: stackoverflow.com/a/30984882/281545)
bufsize=-1 means use the default I/O buffer size instead of bufsize=0 (unbuffered) on Python 2. It is a performance boost on Python 2. It is default on the recent Python 3 versions. You might get a short read (lose data) if on some earlier Python 3 versions where bufsize is not changed to bufsize=-1.
This answer reads in chunks and therefore the stream is fully buffered for efficiency. The solution you've linked is line-oriented. bufsize=1 means "line buffered". There is minimal difference from bufsize=-1 otherwise.
and also what the read_chunk = partial(p.stdout.read, 1 << 15) buys us ?
It is equivalent to read_chunk = lambda: p.stdout.read(1<<15) but provides more introspection in general. It is used to implement wc -l in Python efficiently.
Since I already have 7z.exe bundled with the app and I surely want to avoid a third party lib, while I do need to parse rar and 7z archives I think I will go with:
regErrMatch = re.compile(u'Error:', re.U).match # needs more testing
r"""7z list command output is of the form:
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2015-06-29 21:14:04 ....A <size> <filename>
where ....A is the attribute value for normal files, ....D for directories
"""
reFileMatch = re.compile(ur'(\d|:|-|\s)*\.\.\.\.A', re.U).match
def countFilesInArchive(srcArch, listFilePath=None):
"""Count all regular files in srcArch (or only the subset in
listFilePath)."""
# https://stackoverflow.com/q/31124670/281545
command = ur'"%s" l -scsUTF-8 -sccUTF-8 "%s"' % ('compiled/7z.exe', srcArch)
if listFilePath: command += u' #"%s"' % listFilePath
proc = Popen(command, stdout=PIPE, startupinfo=startupinfo, bufsize=-1)
length, errorLine = 0, []
with proc.stdout as out:
for line in iter(out.readline, b''):
line = unicode(line, 'utf8')
if errorLine or regErrMatch(line):
errorLine.append(line)
elif reFileMatch(line):
length += 1
returncode = proc.wait()
if returncode or errorLine: raise StateError(u'%s: Listing failed\n' +
srcArch + u'7z.exe return value: ' + str(returncode) +
u'\n' + u'\n'.join([x.strip() for x in errorLine if x.strip()]))
return length
Error checking as in Python Popen - wait vs communicate vs CalledProcessError by #JFSebastien
My final(ish) based on accepted answer - unicode may not be needed, kept it for now as I use it everywhere. Also kept regex (which I may expand, I have seen things like re.compile(u'^(Error:.+|.+ Data Error?|Sub items Errors:.+)',re.U). Will have to look into check_output and CalledProcessError.
def countFilesInArchive(srcArch, listFilePath=None):
"""Count all regular files in srcArch (or only the subset in
listFilePath)."""
command = [exe7z, u'l', u'-scsUTF-8', u'-sccUTF-8', srcArch]
if listFilePath: command += [u'#%s' % listFilePath]
proc = Popen(command, stdout=PIPE, stdin=PIPE, # stdin needed if listFilePath
startupinfo=startupinfo, bufsize=1)
errorLine = line = u''
with proc.stdout as out:
for line in iter(out.readline, b''): # consider io.TextIOWrapper
line = unicode(line, 'utf8')
if regErrMatch(line):
errorLine = line + u''.join(out)
break
returncode = proc.wait()
msg = u'%s: Listing failed\n' % srcArch.s
if returncode or errorLine:
msg += u'7z.exe return value: ' + str(returncode) + u'\n' + errorLine
elif not line: # should not happen
msg += u'Empty output'
else: msg = u''
if msg: raise StateError(msg) # consider using CalledProcessError
# number of files is reported in the last line - example:
# 3534900 325332 75 files, 29 folders
return int(re.search(ur'(\d+)\s+files,\s+\d+\s+folders', line).group(1))
Will edit this with my findings.
I want to execute a shell script with 3 arguments from a python script. (as described here: Python: executing shell script with arguments(variable), but argument is not read in shell script)
Here is my code:
subprocess.call('/root/bin/xen-limit %s %s %s' % (str(dom),str(result),str('--nosave'),), shell=True)
variables dom and result are containing strings.
And here is the output:
/bin/sh: --nosave: not found
UPDATE:
That is the variable "result":
c1 = ['/bin/cat', '/etc/xen/%s.cfg' % (str(dom))]
p1 = subprocess.Popen(c1, stdout=subprocess.PIPE)
c2 = ['grep', 'limited']
p2 = subprocess.Popen(c2, stdin=p1.stdout, stdout=subprocess.PIPE)
c3 = ['cut', '-d=', '-f2']
p3 = subprocess.Popen(c3, stdin=p2.stdout, stdout=subprocess.PIPE)
c4 = ['tr', '-d', '\"']
p4 = subprocess.Popen(c4, stdin=p3.stdout, stdout=subprocess.PIPE)
result = p4.stdout.read()
After that, the variable result is containing a number with mbit (for example 16mbit)
And dom is a string like "myserver"
from subprocess import Popen, STDOUT, PIPE
print('Executing: /root/bin/xen-limit ' + str(dom) + ' ' + str(result) + ' --nosave')
handle = Popen('/root/bin/xen-limit ' + str(dom) + ' ' + str(result) + ' --nosave', shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE)
print(handle.stdout.read())
If this doesn't work i honestly don't know what would.
This is the most basic but yet error describing way of opening a 3:d party application or script while still giving you the debug you need.
Why not you save --nosave to a variable and pass the variable in subprocess
It's simpler (and safer) to pass a list consisting of the command name and its arguments.
subprocess.call(['/root/bin/xen-limit]',
str(dom),
str(result),
str('--nosave')
])
str('--nosave') is a no-op, as '--nosave' is already a string. The same may be true for dom and result as well.
In the snipet of my python script below, I think that temp2 doesn't wait for temp to finish running, the output can be large, but is just text. This truncates the result ('out') from temp, it stops mid line. 'out' from temp works fine until temp 2 is added. I tried adding time.wait() as well as subprocess.Popen.wait(temp). These both allow temp to run to completion so that 'out' is not truncated but disrupt the chaining process so that there is no 'out2'. Any ideas?
temp = subprocess.Popen(call, stdout=subprocess.PIPE)
#time.wait(1)
#subprocess.Popen.wait(temp)
temp2 = subprocess.Popen(call2, stdin=temp.stdout, stdout=subprocess.PIPE)
out, err = temp.communicate()
out2, err2 = temp2.communicate()
According to the Python Docs communicate() can accept a stream to be sent as input. If you change stdin of temp2 to subprocess.PIPE and put out in communicate(), the data is properly piped.
#!/usr/bin/env python
import subprocess
import time
call = ["echo", "hello\nworld"]
call2 = ["grep", "w"]
temp = subprocess.Popen(call, stdout=subprocess.PIPE)
temp2 = subprocess.Popen(call2, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = temp.communicate()
out2, err2 = temp2.communicate(out)
print("Out: {0!r}, Err: {1!r}".format(out, err))
# Out: b'hello\nworld\n', Err: None
print("Out2: {0!r}, Err2: {1!r}".format(out2, err2))
# Out2: b'world\n', Err2: None
Following "Replacing shell pipeline" section from the docs:
temp = subprocess.Popen(call, stdout=subprocess.PIPE)
temp2 = subprocess.Popen(call2, stdin=temp.stdout, stdout=subprocess.PIPE)
temp.stdout.close()
out2 = temp2.communicate()[0]
I am trying to use Python to create a tool for imaging CF cards with a Raspberry Pi.
I had most of it working until I implemented compressed images with dd.
When I try and pipe the output of gzip to ddI lose the ability to poke the dd process and get a progress.
I have tried to use multiple sub processes but keep getting broken pipe or no such file errors.
Below is my code:
#!/usr/bin/env python
from Adafruit_CharLCD import Adafruit_CharLCD
import os
import sys
import time
import signal
from subprocess import Popen, PIPE
lcd = Adafruit_CharLCD()
lcd.begin(16,2)
imgpth = '/home/pi/image/image_V11.img.gz'
line0 = ""
line1 = ""
q = 0
r = 0
s = 0
def lcdPrint(column, row, message, clear=False):
if ( clear == True ):
lcd.clear()
lcd.setCursor(column, row)
lcd.message(message)
lcd.clear()
lcdPrint(0, 0, 'Preparing Copy', True)
lcdPrint(0, 1, '')
gz = Popen(['gunzip -c /home/pi/image/image_V11.img.gz'], stdout=PIPE)
dd = Popen(['dd of=/dev/sda'],stdin=gz.stdout, stderr=PIPE)
filebyte = os.path.getsize(imgpth)
flsz = filebyte/1024000
while dd.poll() is None:
time.sleep(1)
dd.send_signal(signal.SIGUSR1)
while 1:
l = dd.stderr.readline()
if '(' in l:
param, value = l.split('b',1)
line1 = param.rstrip()
r = float(line1)
s = r/1024000
break
lcdPrint(0, 0, 'Copying....', True)
q = round(s/flsz*100, 2)
per = str(q)
lcdPrint(0, 1, per + '% Complete',)
lcdPrint(0, 0, 'Copy Complete', True)
time.sleep(1)
exit()
How can I fix this?
I stumbled across this question because I am doing exactly the same. My complete solution is here:
http://github.com/jrmhaig/Bakery
I've tried to pick out some differences between what I have and yours that might show you the solution.
When starting the dd I redirected both stderr and stdout to the pipe.
dd = subprocess.Popen(['dd', 'of=/dev/sda', 'bs=1M'], bufsize=1, stdin=unzip.stdout, stdout=PIPE, stderr=STDOUT)
I don't think this should really make a difference. Everything you need should go to stderr but for some reason it appeared to get mixed up for me.
I use a separate thread to pick up the output from dd:
def read_pipe(out, queue):
for line in iter(out.readline, b''):
queue.put(str(line))
out.close()
dd_queue = queue.Queue()
dd_thread = threading.Thread(target = read_pipe, args=(dd.stdout, dd_queue))
dd_thread.daemon = True
dd_thread.start()
Then when you call:
dd.send_signal(signal.SIGUSR1)
the output gets caught on dd_queue.
I also found that the uncompressed size of an gzipped file is stored in the last 4 bytes:
fl = open(str(imgpath), 'rb')
fl.seek(-4, 2)
r = fl.read()
fl.close()
size = struct.unpack('<I', r)[0]
os.path.getsize(imgpth) will only give you the compressed size so the percentage calculation will be wrong.
I have an application that takes input, either from the terminal directly or I can use a pipe to pass the output of another program into the stdin of this one. What I am trying to do is use python to generate the output so it's formatted correctly and pass that to the stdin of this program all from the same script. Here is the code:
#!/usr/bin/python
import os
import subprocess
import plistlib
import sys
def appScan():
os.system("system_profiler -xml SPApplicationsDataType > apps.xml")
appList = plistlib.readPlist("apps.xml")
sys.stdout.write( "Mac_App_List\n"
"Delimiters=\"^\"\n"
"string50 string50\n"
"Name^Version\n")
appDict = appList[0]['_items']
for x in appDict:
if 'version' in x:
print x['_name'] + "^" + x['version'] + "^"
else:
print x['_name'] + "^" + "no version found" + "^"
proc = subprocess.Popen(["/opt/altiris/notification/inventory/lib/helpers/aex- sendcustominv","-t","-"], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.communicate(input=appScan())
For some reason this subprocess I am calling doesn't like what is coming into stdin. However if I remove the subprocess items and just have the script print to stdout and then call the script from the terminal (python appScan.py | aex-sendcustominv), aex-sendcustominv is able to accept the input just fine. Is there any way to take a functions output in python and send it to the stdin of an subprocess?
The problem is that appScan() only prints to stdout; appScan() returns None, so proc.communicate(input=appScan()) is equivalent to proc.communicate(input=None). You need appScan to return a string.
Try this (not tested):
def appScan():
os.system("system_profiler -xml SPApplicationsDataType > apps.xml")
appList = plistlib.readPlist("apps.xml")
output_str = 'Delimiters="^"\nstring50 string50\nName^Version\n'
appDict = appList[0]['_items']
for x in appDict:
if 'version' in x:
output_str = output_str + x['_name'] + "^" + x['version'] + "^"
else:
output_str = output_str + x['_name'] + "^" + "no version found" + "^"
return output_str
proc = subprocess.Popen(["/opt/altiris/notification/inventory/lib/helpers/aex- sendcustominv","-t","-"], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.communicate(input=appScan())