Develop Reference

Call 7zip from Python on Windows

I'm trying to figure out how to invoke 7zip, on Windows, from a Python program.
I'm trying:
stdout = subprocess.run(['C:\\Program Files\\7-Zip\\7z.exe', "a -t7z -mx0", "C:\\Users\\IanWo\\test.7z", "C:\\Users\\IanWo\\test.txt", "C:\\Users\\IanWo\\test2.txt"], shell=True, check=True, capture_output=True, text=True).stdout
print(stdout)
but am getting:
Traceback (most recent call last):
File "D:\Normal Backup\Code\ProcessRetrospectBackups\process.py", line 93, in <module>
stdout = subprocess.run(['C:\\Program Files\\7-Zip\\7z.exe', "a -t7z -mx0", "C:\\Users\\IanWo\\test.7z", "C:\\Users\\IanWo\\test.txt", "C:\\Users\\IanWo\\test2.txt"], shell=True, check=True, capture_output=True, text=True).stdout
File "C:\Users\IanWo\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Program Files\\7-Zip\\7z.exe', 'a -t7z -mx0', 'C:\\Users\\IanWo\\test.7z', 'C:\\Users\\IanWo\\test.txt', 'C:\\Users\\IanWo\\test2.txt']' returned non-zero exit status 7.
>>>
It invokes 7z.exe fine as long as I don't have any arguments. I've tried with and without shell=True with no change.

#ThiefMaster is right. Here's the correct call:
stdout = subprocess.run(['C:\\Program Files\\7-Zip\\7z.exe', "a", "-t7z", "-mx0", "C:\\Users\\IanWo\\test.7z", "C:\\Users\\IanWo\\test.txt", "C:\\Users\\IanWo\\test2.txt"], shell=True, check=True, capture_output=True, text=True).stdout
print(stdout)

Running a list command strings with subprocess popen and getting the output

I'm trying to run multiple UNIX commands in a python script like this
import subprocess
cmds = ['sleep 3', 'uptime','time ls -l /']
p = subprocess.Popen(cmds,stdout=subprocess.PIPE,shell=True)
while p.poll() is None:
time.sleep(0.5)
tempdata = p.stdout.read()
print(tempdata)
However my output does not contain all output and doesn't seem to run all the commands. Setting shell=False also causes an error.
Traceback (most recent call last):
File "task1.py", line 32, in ?
p = subprocess.Popen(commands,stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python36/subprocess.py", line 550, in __init__
errread, errwrite)
File "/usr/lib64/python36/subprocess.py", line 996, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

When you create a new process, you don't pass it a list of commands to run; rather, you pass it a single command -- either as a string (with shell=True) or as a list of args (with shell=False).
import subprocess
cmds = ['sleep 1', 'uptime', 'ls -l /']
for cmd in cmds:
stdout = subprocess.check_output(cmd, shell=True)
print('\n# {}'.format(cmd))
print(stdout)
If you just want to collect stdout, subprocess.check_output() might be simpler than Popen() -- but either approach will work, depending on what you need to do with the process.

Your problem is 'sleep 3' causes the error you get from the traceback, when I removed that it worked.

To run all for these:
cmds = ['sleep 3', 'uptime','time ls -l /']
You have to call popen for each of them:
for cmd in cmds:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
while p.poll() is None:
time.sleep(0.5)
output = p.stdout.read()
Or simpler:
for cmd in cmds:
output = subprocess.check_output(cmd, stdout=subprocess.PIPE, shell=True)
Second question: This captures all output written to stdout. To capture also stderr, redirect that into subprocess.PIPE as well.

input directly in process.communicate() in subprocess library

I want to use both multiline input for command 'cat -' and single line input in 'pwd'.
For that i am trying to get input directly in process.communicate() , I am getting broken pipe error. What shall i replace in the argument passed?
command = 'cat -'
stdout_value=''
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
while True:
if process.poll() is not None:
break
stdout_value = stdout_value + process.communicate(process.stdin)[0]
print(repr(stdout_value))
This code gets error:
Traceback (most recent call last):
File "try.py", line 67, in <module>
stdout_value = stdout_value + process.communicate(process.stdin)[0]
File "/usr/lib64/python3.5/subprocess.py", line 1068, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib64/python3.5/subprocess.py", line 1687, in _communicate
input_view = memoryview(self._input)
TypeError: memoryview: a bytes-like object is required, not '_io.BufferedWriter'
[vibhcool#localhost try]$ Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

process.communicate() expects string which you have encoded() to bytes and it returns bytes which you have to decode() to string.
You use process.stdin which is not string. It even doesn't have method read() to do process.communicate(process.stdin.read()) (besides using process.stdin with the same process makes no sense). It could be rather process.communicate(other_process.stdout.read()) (if you have other process)
Working example - I send text with numbers to command sort and it returns text with sorted numbers.
input: 3\n2\n5\n-1
output: -1\n2\n3\n5\n
Code
import subprocess
import shlex
command = 'sort'
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
stdout_value = ''
while True:
if process.poll() is not None:
break
result = process.communicate("3\n2\n5\n-1".encode('utf-8'))[0]
stdout_value += result.decode('utf-8')
print(repr(stdout_value))

Analyze output of subprocess.Popen when piping to file

How can I analyze the output of my command, which I pipe to file, in real-time while the file is written?
This is what I have so far:
with open('output.log', 'w') as out:
command = ['pg_dump', 'myDB']
p = subprocess.Popen(cmd, stdout=out, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
sys.stdout.flush()
print(">>> " + line.rstrip())
But this generates the following error:
Traceback (most recent call last):
File "pipe-to-file.py", line 95, in <module>
for line in iter(p.stdout.readline, b''):
AttributeError: 'NoneType' object has no attribute 'readline'
Why is p.stdout equal to None here?

You have to use subprocess.PIPE for the stdout argument in order to get a file object, else it'll be None. That's why p.stdout equals to None in your code.
From the DOC
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
If you want to write stdout to a file while analyzing the output then you can use something like this.
with open('log', 'ab+') as out:
p = subprocess.Popen('ls', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
std_out, std_error = p.communicate()
# Do something with std_out
# ...
# Write to the file
out.write( std_out )
# You can use `splitlines()` to iterate over the lines.
for line in std_out.splitlines():
print line

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,error=p.communicate()
Now you have your output,error.

Broken Pipe from subprocess.Popen.communciate() with stdin

I'm having a strange issue when using subprocess.Popen.communicate(). For background, I want to execute an application from my python script. When I run the program from the command line, I do it like this (UNIX):
$ echo "input text" | /path/to/myapp
From my script, I also want to pipe the input into the application. So, I tried the following. But I get a "broken pipe" error when I try to send the input with communicate():
>>> cmd = ['/path/to/myapp']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('input text')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/subprocess.py", line 670, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1223, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 1003, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
To make matters stranger, if I leave out the input data, I don't get any errors. However, this isn't really a good workaround because the application needs input to work.
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate()
>>> print out
[error from myapp regarding lack of input]
Any idea what I'm missing?

Your observation suggests that myapp is terminating without reading (all of the) input. Not knowing anything about myapp, that's hard to confirm, but consider for example
$ echo 'hello world' | tr 'l' 'L'
heLLo worLd
now...:
>>> cmd = ['/usr/bin/tr']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/subprocess.py", line 668, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1218, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 997, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
because...:
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> /usr/bin/tr: missing operand
Try `/usr/bin/tr --help' for more information.
and if we fix the bug:
>>> cmd = ['/usr/bin/tr', 'l', 'L']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')>>> print out
heLLo worLd
>>> print err
None
...it fixes everything. What happens if you omit the stderr redirection -- do you perchance see any error messages from myapp...?