Develop Reference

python invoking PV-Wave

'/usr/local/bin/wave' only accepts a filename as input, so I need to invoke the process, then "send in" the commands, and wait for the output file to be written. Then my process can proceed to read the output file. Here is my code that does not write to the output file:
hdfFile = "/archive/HDF/16023343.hdf"
pngFile = "/xrfc_calib/xrfc.130.png"
lpFile = os.environ['DOCUMENT_ROOT'] + pngFile
waveCmd = "hdfview, '" + hdfFile + "', outfile='" + lpFile + "', web, view='RASTER', /neg"
os.environ['WAVE_PATH'] = "/oudvmt/wave/pro:/dvmt/wave/pro"
wfile = subprocess.Popen ('/usr/local/bin/wave >&2', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
wfile.stdin = "\#hdf_startup\n\#hdf_common\n" + waveCmd + "\nquit\n"

I found what I was missing. The change is to the last 2 lines. They are:
wfile = subprocess.Popen ('/usr/local/bin/wave', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
wfile.communicate("\#hdf_startup\n\#hdf_common\n" + waveCmd + "\nquit\n")
I needed to set "stdout" to avoid extra output from PV-Wave.
I needed to use "communicate" to wait for the process to complete.

How to programmatically count the number of files in an archive using python

In the program I maintain it is done as in:
# count the files in the archive
length = 0
command = ur'"%s" l -slt "%s"' % (u'path/to/7z.exe', srcFile)
ins, err = Popen(command, stdout=PIPE, stdin=PIPE,
startupinfo=startupinfo).communicate()
ins = StringIO.StringIO(ins)
for line in ins: length += 1
ins.close()
Is it really the only way ? I can't seem to find any other command but it seems a bit odd that I can't just ask for the number of files
What about error checking ? Would it be enough to modify this to:
proc = Popen(command, stdout=PIPE, stdin=PIPE,
startupinfo=startupinfo)
out = proc.stdout
# ... count
returncode = proc.wait()
if returncode:
raise Exception(u'Failed reading number of files from ' + srcFile)
or should I actually parse the output of Popen ?
EDIT: interested in 7z, rar, zip archives (that are supported by 7z.exe) - but 7z and zip would be enough for starters

To count the number of archive members in a zip archive in Python:
#!/usr/bin/env python
import sys
from contextlib import closing
from zipfile import ZipFile
with closing(ZipFile(sys.argv[1])) as archive:
count = len(archive.infolist())
print(count)
It may use zlib, bz2, lzma modules if available, to decompress the archive.
To count the number of regular files in a tar archive:
#!/usr/bin/env python
import sys
import tarfile
with tarfile.open(sys.argv[1]) as archive:
count = sum(1 for member in archive if member.isreg())
print(count)
It may support gzip, bz2 and lzma compression depending on version of Python.
You could find a 3rd-party module that would provide a similar functionality for 7z archives.
To get the number of files in an archive using 7z utility:
import os
import subprocess
def count_files_7z(archive):
s = subprocess.check_output(["7z", "l", archive], env=dict(os.environ, LC_ALL="C"))
return int(re.search(br'(\d+)\s+files,\s+\d+\s+folders$', s).group(1))
Here's version that may use less memory if there are many files in the archive:
import os
import re
from subprocess import Popen, PIPE, CalledProcessError
def count_files_7z(archive):
command = ["7z", "l", archive]
p = Popen(command, stdout=PIPE, bufsize=1, env=dict(os.environ, LC_ALL="C"))
with p.stdout:
for line in p.stdout:
if line.startswith(b'Error:'): # found error
error = line + b"".join(p.stdout)
raise CalledProcessError(p.wait(), command, error)
returncode = p.wait()
assert returncode == 0
return int(re.search(br'(\d+)\s+files,\s+\d+\s+folders', line).group(1))
Example:
import sys
try:
print(count_files_7z(sys.argv[1]))
except CalledProcessError as e:
getattr(sys.stderr, 'buffer', sys.stderr).write(e.output)
sys.exit(e.returncode)
To count the number of lines in the output of a generic subprocess:
from functools import partial
from subprocess import Popen, PIPE, CalledProcessError
p = Popen(command, stdout=PIPE, bufsize=-1)
with p.stdout:
read_chunk = partial(p.stdout.read, 1 << 15)
count = sum(chunk.count(b'\n') for chunk in iter(read_chunk, b''))
if p.wait() != 0:
raise CalledProcessError(p.returncode, command)
print(count)
It supports unlimited output.
Could you explain why buffsize=-1 (as opposed to buffsize=1 in your previous answer: stackoverflow.com/a/30984882/281545)
bufsize=-1 means use the default I/O buffer size instead of bufsize=0 (unbuffered) on Python 2. It is a performance boost on Python 2. It is default on the recent Python 3 versions. You might get a short read (lose data) if on some earlier Python 3 versions where bufsize is not changed to bufsize=-1.
This answer reads in chunks and therefore the stream is fully buffered for efficiency. The solution you've linked is line-oriented. bufsize=1 means "line buffered". There is minimal difference from bufsize=-1 otherwise.
and also what the read_chunk = partial(p.stdout.read, 1 << 15) buys us ?
It is equivalent to read_chunk = lambda: p.stdout.read(1<<15) but provides more introspection in general. It is used to implement wc -l in Python efficiently.

Since I already have 7z.exe bundled with the app and I surely want to avoid a third party lib, while I do need to parse rar and 7z archives I think I will go with:
regErrMatch = re.compile(u'Error:', re.U).match # needs more testing
r"""7z list command output is of the form:
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2015-06-29 21:14:04 ....A <size> <filename>
where ....A is the attribute value for normal files, ....D for directories
"""
reFileMatch = re.compile(ur'(\d|:|-|\s)*\.\.\.\.A', re.U).match
def countFilesInArchive(srcArch, listFilePath=None):
"""Count all regular files in srcArch (or only the subset in
listFilePath)."""
# https://stackoverflow.com/q/31124670/281545
command = ur'"%s" l -scsUTF-8 -sccUTF-8 "%s"' % ('compiled/7z.exe', srcArch)
if listFilePath: command += u' #"%s"' % listFilePath
proc = Popen(command, stdout=PIPE, startupinfo=startupinfo, bufsize=-1)
length, errorLine = 0, []
with proc.stdout as out:
for line in iter(out.readline, b''):
line = unicode(line, 'utf8')
if errorLine or regErrMatch(line):
errorLine.append(line)
elif reFileMatch(line):
length += 1
returncode = proc.wait()
if returncode or errorLine: raise StateError(u'%s: Listing failed\n' +
srcArch + u'7z.exe return value: ' + str(returncode) +
u'\n' + u'\n'.join([x.strip() for x in errorLine if x.strip()]))
return length
Error checking as in Python Popen - wait vs communicate vs CalledProcessError by #JFSebastien
My final(ish) based on accepted answer - unicode may not be needed, kept it for now as I use it everywhere. Also kept regex (which I may expand, I have seen things like re.compile(u'^(Error:.+|.+ Data Error?|Sub items Errors:.+)',re.U). Will have to look into check_output and CalledProcessError.
def countFilesInArchive(srcArch, listFilePath=None):
"""Count all regular files in srcArch (or only the subset in
listFilePath)."""
command = [exe7z, u'l', u'-scsUTF-8', u'-sccUTF-8', srcArch]
if listFilePath: command += [u'#%s' % listFilePath]
proc = Popen(command, stdout=PIPE, stdin=PIPE, # stdin needed if listFilePath
startupinfo=startupinfo, bufsize=1)
errorLine = line = u''
with proc.stdout as out:
for line in iter(out.readline, b''): # consider io.TextIOWrapper
line = unicode(line, 'utf8')
if regErrMatch(line):
errorLine = line + u''.join(out)
break
returncode = proc.wait()
msg = u'%s: Listing failed\n' % srcArch.s
if returncode or errorLine:
msg += u'7z.exe return value: ' + str(returncode) + u'\n' + errorLine
elif not line: # should not happen
msg += u'Empty output'
else: msg = u''
if msg: raise StateError(msg) # consider using CalledProcessError
# number of files is reported in the last line - example:
# 3534900 325332 75 files, 29 folders
return int(re.search(ur'(\d+)\s+files,\s+\d+\s+folders', line).group(1))
Will edit this with my findings.

Python subprocess: capture output of ffmpeg and run regular expression against it

I have the following code
import subprocess
import re
from itertools import *
command = ['ffprobe', '-i', '/media/some_file.mp4']
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
text = p.stderr.read()
retcode = p.wait()
text = text.decode('utf-8')
p = re.compile("Duration(.*)")
num = 0 #for debugging
for line in iter(text.splitlines()):
print(str(num) + line) #for debugging
m = p.match(str(line))
if m != None:
print(m.group(1))
When I look at the output there is a line that says "Duration" on it, however it is not captured, print(m.group(1)) is never reached. If I change the text variable to a hardcoded string of "Duration blahblah" I get " blahblah", which is what I expect. It seems like the regex doesn't recognize the text coming back from stderr. How can I get the text into a format that the regex will recognize and match on?
I have come up with the following solution, should it help anyone else attempting to capture duration from ffmpeg using python
import subprocess
import re
command = ['ffprobe', '-i', '/media/some_file.mp4']
p = subprocess.Popen(command, stderr=subprocess.PIPE)
text = p.stderr.read()
retcode = p.wait()
text = text.decode('utf-8')
p = re.compile(".*Duration:\s([0-9:\.]*),", re.MULTILINE|re.DOTALL)
m = p.match(text)
print(m.group(1))

p = re.compile(r".*?Duration(.*)")
Try this.match starts from the begining while there may might be something before duration.

Execute Shell Script from python with arguments

I want to execute a shell script with 3 arguments from a python script. (as described here: Python: executing shell script with arguments(variable), but argument is not read in shell script)
Here is my code:
subprocess.call('/root/bin/xen-limit %s %s %s' % (str(dom),str(result),str('--nosave'),), shell=True)
variables dom and result are containing strings.
And here is the output:
/bin/sh: --nosave: not found
UPDATE:
That is the variable "result":
c1 = ['/bin/cat', '/etc/xen/%s.cfg' % (str(dom))]
p1 = subprocess.Popen(c1, stdout=subprocess.PIPE)
c2 = ['grep', 'limited']
p2 = subprocess.Popen(c2, stdin=p1.stdout, stdout=subprocess.PIPE)
c3 = ['cut', '-d=', '-f2']
p3 = subprocess.Popen(c3, stdin=p2.stdout, stdout=subprocess.PIPE)
c4 = ['tr', '-d', '\"']
p4 = subprocess.Popen(c4, stdin=p3.stdout, stdout=subprocess.PIPE)
result = p4.stdout.read()
After that, the variable result is containing a number with mbit (for example 16mbit)
And dom is a string like "myserver"

from subprocess import Popen, STDOUT, PIPE
print('Executing: /root/bin/xen-limit ' + str(dom) + ' ' + str(result) + ' --nosave')
handle = Popen('/root/bin/xen-limit ' + str(dom) + ' ' + str(result) + ' --nosave', shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE)
print(handle.stdout.read())
If this doesn't work i honestly don't know what would.
This is the most basic but yet error describing way of opening a 3:d party application or script while still giving you the debug you need.

Why not you save --nosave to a variable and pass the variable in subprocess

It's simpler (and safer) to pass a list consisting of the command name and its arguments.
subprocess.call(['/root/bin/xen-limit]',
str(dom),
str(result),
str('--nosave')
])
str('--nosave') is a no-op, as '--nosave' is already a string. The same may be true for dom and result as well.

python unicode implementation (using external programs: cygnative plink ssh rsync)

I have a backup applications in python that needs to work on Windows. It needs UTF compatibility (to be able to backup directories that contain UTF characters like italian accents). The problem is it uses external programs (plink, cygwin, ssh and rsync) and I can't get them working. The prototype is 32 lines long, please take a look:
# -*- coding: utf-8 -*-
import subprocess
def safestr(obj, encoding='utf-8'):
r"""Converts any given object to utf-8 encoded string.
>>> safestr('hello')
'hello'
>>> safestr(u'\u1234')
'\xe1\x88\xb4'
>>> safestr(2)
'2'
"""
if isinstance(obj, unicode):
return obj.encode("utf-8")
elif isinstance(obj, str):
return obj.encode
else:
return str(obj)
def execute(command):
pipe = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
out, errs = pipe.communicate()
retcode = pipe.poll()
print "OUT: " + repr(out)
print "ERRS: " + repr(errs)
print "RET: " + str(retcode)
command = u'rsync --stats -az --numeric-ids --delete --blocking-io --modify-window=2 --no-group --chmod=u=rwX,g=,o= -e \'cygnative plink -ssh -2 -batch -pw test \' "/cygdrive/c/κόσμε" vaidab#192.168.1.86:/volatile/backup/vaidab/2010-03-03.15_41_56/ --link-dest=../2010-03-03.15_00_57'.encode('utf-8')
execute(command)
Still doesn't work with nosklo's version, check the result:
python prototype_unicode_new.py
'rsync.exe --stats -az --numeric-ids --delete --blocking-io --modify-window=2 --
no-group --chmod=u=rwX,g=,o= -e "cygnative plink -ssh -2 -batch -pw test" /cygdr
ive/c/\xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xb5 vaidab#192.168.1.86:/volatile/bac
kup/vaidab/2010-03-03.15_41_56/'
OUT: '\nNumber of files: 0\nNumber of files transferred: 0\nTotal file size: 0 b
ytes\nTotal transferred file size: 0 bytes\nLiteral data: 0 bytes\nMatched data:
0 bytes\nFile list size: 9\nFile list generation time: 0.001 seconds\nFile list
transfer time: 0.000 seconds\nTotal bytes sent: 22\nTotal bytes received: 12\n\
nsent 22 bytes received 12 bytes 68.00 bytes/sec\ntotal size is 0 speedup is
0.00\n'
ERRS: 'rsync: link_stat "/cygdrive/c/\xc3\x8e\xc2\xba\xc3\x8f\xc5\x92\xc3\x8f\xc
6\x92\xc3\x8e\xc2\xbc\xc3\x8e\xc2\xb5" failed: No such file or directory (2)\nrs
ync error: some files/attrs were not transferred (see previous errors) (code 23)
at /home/lapo/packaging/rsync-3.0.6-1/src/rsync-3.0.6/main.c(1039) [sender=3.0.
6]\n'
RET: 23

Don't use shell=True. EVER. It needlessy invokes a shell to call your program.
Pass the parameters as a list instead of a string.
This example should work, provided the parameters are right and the rsync.exe is in current folder (or PATH):
# -*- coding: utf-8 -*-
import subprocess
def execute(command):
pipe = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, errs = pipe.communicate()
retcode = pipe.poll()
print "OUT: " + repr(out)
print "ERRS: " + repr(errs)
print "RET: " + str(retcode)
return out
command = ['rsync.exe', '--stats', '-az', '--numeric-ids', '--delete',
'--blocking-io', '--modify-window=2', '--no-group',
'--chmod=u=rwX,g=,o=', '-e',
'cygnative plink -ssh -2 -batch -pw test',
u'/cygdrive/c/κόσμε'.encode('utf-8'),
'vaidab#192.168.1.86:/volatile/backup/vaidab/2010-03-03.15_41_56/',
'--link-dest=../2010-03-03.15_00_57']
execute(command)

A piece of code that passeth all understanding:
if isinstance(obj, unicode):
return obj.encode("utf-8")
elif isinstance(obj, str):
return obj.encode
# the above is returning a METHOD ***************************
else:
return str(obj)
What's the point of doctests if you don't run them?