Develop Reference

How to fix Variables send not the good Values [duplicate]

This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to assign the output of a command I run using os.system to a variable and prevent it from being output to the screen. But, in the below code ,the output is sent to the screen and the value printed for var is 0, which I guess signifies whether the command ran successfully or not. Is there any way to assign the command output to the variable and also stop it from being displayed on the screen?
var = os.system("cat /etc/services")
print var #Prints 0

From this question which I asked a long time ago, what you may want to use is popen:
os.popen('cat /etc/services').read()
From the docs for Python 3.6,
This is implemented using subprocess.Popen; see that class’s
documentation for more powerful ways to manage and communicate with
subprocesses.
Here's the corresponding code for subprocess:
import subprocess
proc = subprocess.Popen(["cat", "/etc/services"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
print("program output:", out)

You might also want to look at the subprocess module, which was built to replace the whole family of Python popen-type calls.
import subprocess
output = subprocess.check_output("cat /etc/services", shell=True)
The advantage it has is that there is a ton of flexibility with how you invoke commands, where the standard in/out/error streams are connected, etc.

The commands module is a reasonably high-level way to do this:
import commands
status, output = commands.getstatusoutput("cat /etc/services")
status is 0, output is the contents of /etc/services.

For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code. Since you are only interested in the output, you can write a utility wrapper like this.
from subprocess import PIPE, run
def out(command):
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
return result.stdout
my_output = out("echo hello world")
# Or
my_output = out(["echo", "hello world"])

I know this has already been answered, but I wanted to share a potentially better looking way to call Popen via the use of from x import x and functions:
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
print cmdline("cat /etc/services")
print cmdline('ls')
print cmdline('rpm -qa | grep "php"')
print cmdline('nslookup google.com')

I do it with os.system temp file:
import tempfile, os
def readcmd(cmd):
ftmp = tempfile.NamedTemporaryFile(suffix='.out', prefix='tmp', delete=False)
fpath = ftmp.name
if os.name=="nt":
fpath = fpath.replace("/","\\") # forwin
ftmp.close()
os.system(cmd + " > " + fpath)
data = ""
with open(fpath, 'r') as file:
data = file.read()
file.close()
os.remove(fpath)
return data

Python 2.6 and 3 specifically say to avoid using PIPE for stdout and stderr.
The correct way is
import subprocess
# must create a file object to store the output. Here we are getting
# the ssid we are connected to
outfile = open('/tmp/ssid', 'w');
status = subprocess.Popen(["iwgetid"], bufsize=0, stdout=outfile)
outfile.close()
# now operate on the file

from os import system, remove
from uuid import uuid4
def bash_(shell_command: str) -> tuple:
"""
:param shell_command: your shell command
:return: ( 1 | 0, stdout)
"""
logfile: str = '/tmp/%s' % uuid4().hex
err: int = system('%s &> %s' % (shell_command, logfile))
out: str = open(logfile, 'r').read()
remove(logfile)
return err, out
# Example:
print(bash_('cat /usr/bin/vi | wc -l'))
>>> (0, '3296\n')```

How to use Subprocess in Python

I would like to execute following shell command in python: grep 'string' file | tail -1 | cut -c 1-3
I tried:
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
subprocess.call(grep 'string' file | tail -1 | cut -c 1-3)
i = i + 1
Any help would be appreciated. Thanks.

First of all, whatever you pass into the subprocess.call should be a string. Names grep, file, tail and cut are not defined in your code and you need to turn the whole expression into a string. Since the search string for the grep command should be dynamic, you need to construct the final string before passing it as argument into the function.
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
subprocess.call(command_string)
i = i + 1
You probably want to pass in an additional argument to subprocess.call: shell=True. The argument will make sure the command is executed through the shell.
Your command is using cut. You might want to retrieve the output of the subprocess, so a better option would be to create a new process object and use subprocess.communicate with turned out output capturing:
import subprocess
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
p = subprocess.Popen(command_string, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutdata, stderrdata = p.communicate()
# stdoutdata now contains the output of the shell commands and you can use it
# in your program
i = i + 1
EDIT: Here is the information on how to store the data into a text file, as requested in the comment.
import subprocess
outputs = []
i = 1
while i < 1070:
file = "sorted." + str(i) + ".txt"
string = "2x"
command_string = 'grep {0} {1} | tail -1 | cut -c 1-3'.format(string, file)
p = subprocess.Popen(command_string, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
stdoutdata, stderrdata = p.communicate()
# stdoutdata now contains the output of the shell commands and you can use it
# in your program, like writing the output to a file.
outputs.append(stdoutdata)
i = i + 1
with open('output.txt', 'w') as f:
f.write('\n'.join(outputs))

Your command should be provided as a string.
In addition, if you want to get the output of your command, you can use the following:
subprocess.run("grep 'string' file | tail -1 | cut -c 1-3", shell=True, capture_output=True, check=True)
where capture_output (works in Python3.7+) returns object with returncode, stdout and stderr and the check flag will raise exception if your command fails.

Subprocess expects the arguments as a string or array:
subprocess.call("grep '{}' {} | tail -1 | cut -c 1-3".format(string, file), shell=True)
shell=True is nececairy because you are using shell-specific commands like the pipe.
However, in this case it might be a lot easier to implement the entire program in pure python.
Note that if either string or file contain any special characters including spaces or quotation marks, the command will not work, and could in fact do a variety of unwanted things to your system. If you need it to work on more than these simple values, consider either a pure-python solution, setting shell=False and using the array syntax with manual piping, or some form of escaping.

Using ffmpeg to obtain video durations in python

I've installed ffprobe using the pip ffprobe command on my PC, and installed ffmpeg from here.
However, I'm still having trouble running the code listed here.
I try to use the following code unsuccessfully.
SyntaxError: Non-ASCII character '\xe2' in file GetVideoDurations.py
on line 12, but no encoding declared; see
http://python.org/dev/peps/pep-0263/ for details
Does anyone know what's wrong? Am I not referencing the directories correctly? Do I need to make sure the .py and video files are in a specific location?
import subprocess
def getLength(filename):
result = subprocess.Popen(["ffprobe", "filename"],
stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
return [x for x in result.stdout.readlines() if "Duration" in x]
fileToWorkWith = ‪'C:\Users\PC\Desktop\Video.mkv'
getLength(fileToWorkWith)
Apologies if the question is somewhat basic. All I need is to be able to iterate over a group of video files and get their start time and end time.
Thank you!

There is no need to iterate though the output of FFprobe. There is one simple command which returns only the duration of the input file:
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 <input_video>
You can use the following method instead to get the duration:
def get_length(input_video):
result = subprocess.run(['ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'default=noprint_wrappers=1:nokey=1', input_video], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return float(result.stdout)

I'd suggest using FFprobe (comes with FFmpeg).
The answer Chamath gave was pretty close, but ultimately failed for me.
Just as a note, I'm using Python 3.5 and 3.6 and this is what worked for me.
import subprocess
def get_duration(file):
"""Get the duration of a video using ffprobe."""
cmd = 'ffprobe -i {} -show_entries format=duration -v quiet -of csv="p=0"'.format(file)
output = subprocess.check_output(
cmd,
shell=True, # Let this run in the shell
stderr=subprocess.STDOUT
)
# return round(float(output)) # ugly, but rounds your seconds up or down
return float(output)
If you want to throw this function into a class and use it in Django (1.8 - 1.11), just change one line and put this function into your class, like so:
def get_duration(file):
to:
def get_duration(self, file):
Note: Using a relative path worked for me locally, but the production server required an absolute path. You can use os.path.abspath(os.path.dirname(file)) to get the path to your video or audio file.

Using the python ffmpeg package (https://pypi.org/project/python-ffmpeg)
import ffmpeg
duration = ffmpeg.probe(local_file_path)["format"]["duration"]
where local_file_path is a relative or absolute path to your file.

I think Chamath's second comment answers the question: you have a strange character somewhere in your script, either because you are using a ` instead of a ' or you have a word with non-english accents, something like this.
As a remark, for what you are doing you can also try MoviePy which parses the ffmpeg output like you do (but maybe in the future I'll use Chamath's ffprobe method it looks cleaner):
import moviepy.editor as mp
duration = mp.VideoFileClip("my_video.mp4").duration

Updated solution using ffprobe based on #llogan guidance with the pointed link:
import subprocess
def get_duration(input_video):
cmd = ["ffprobe", "-i", input_video, "-show_entries", "format=duration",
"-v", "quiet", "-sexagesimal", "-of", "csv=p=0"]
return subprocess.check_output(cmd).decode("utf-8").strip()
Fragile Solution due to stderr output:
the stderr output from ffmpeg is not intended for machine parsing and
is considered fragile.
I get help from the following documentation (https://codingwithcody.com/2014/05/14/get-video-duration-with-ffmpeg-and-python/) and https://stackoverflow.com/a/6239379/2402577
Actually, sed is unnecessary: ffmpeg -i file.mp4 2>&1 | grep -o -P "(?<=Duration: ).*?(?=,)"
You can use the following method to get the duration in HH:MM:SS format:
import subprocess
def get_duration(input_video):
# cmd: ffmpeg -i file.mkv 2>&1 | grep -o -P "(?<=Duration: ).*?(?=,)"
p1 = subprocess.Popen(['ffmpeg', '-i', input_video], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
p2 = subprocess.Popen(["grep", "-o", "-P", "(?<=Duration: ).*?(?=,)"], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
return p2.communicate()[0].decode("utf-8").strip()
Example output for both: 01:37:11.83

Have you tried adding the encoding? That error is typical of that, as Chamath said.
Add the utf-8 encoding to your script header:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

I like to build a shared library with ffmpeg, and load it in python.
C++ code:
#ifdef __WIN32__
#define LIB_CLASS __declspec(dllexport)
#else
#define LIB_CLASS
#endif
extern "C" {
#define __STDC_CONSTANT_MACROS
#include "libavformat/avformat.h"
}
extern "C" LIB_CLASS int64_t getDur(const char* url) {
AVFormatContext* pFormatContext = avformat_alloc_context();
if (avformat_open_input(&pFormatContext, url, NULL, NULL)) {
avformat_free_context(pFormatContext);
return -1;
}
int64_t t = pFormatContext->duration;
avformat_close_input(&pFormatContext);
avformat_free_context(pFormatContext);
return t;
}
Then use gcc to compile it and get a shared library.
Python code:
from ctypes import *
lib = CDLL('/the/path/to/your/library')
getDur = lib.getDur
getDur.restype = c_longlong
duration = getDur('the path/URL to your file')
It works well in my python program.

Python Code
<code>
cmnd = ['/root/bin/ffmpeg', '-i', videopath]
process = subprocess.Popen(cmnd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = process.communicate()
#This matches regex to get the time in H:M:S format
matches = re.search(r"Duration:\s{1}(?P<hours>\d+?):(?P<minutes>\d+?):(?P<seconds>\d+\.\d+?),", stdout, re.DOTALL).groupdict()
t_hour = matches['hours']
t_min = matches['minutes']
t_sec = matches['seconds']
t_hour_sec = int(t_hour) * 3600
t_min_sec = int(t_min) * 60
t_s_sec = int(round(float(t_sec)))
total_sec = t_hour_sec + t_min_sec + t_s_sec
#This matches1 is to get the frame rate of a video
matches1 = re.search(r'(\d+) fps', stdout)
frame_rate = matches1.group(0) // This will give 20fps
frame_rate = matches1.group(1) //It will give 20
</code>

we can also use ffmpeg to get the duration of any video or audio files.
To install ffmpeg follow this link
import subprocess
import re
process = subprocess.Popen(['ffmpeg', '-i', path_of_video_file], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = process.communicate()
matches = re.search(r"Duration:\s{1}(?P<hours>\d+?):(?P<minutes>\d+?):(?P<seconds>\d+\.\d+?),", stdout, re.DOTALL).groupdict()
print (matches['hours'])
print (matches['minutes'])
print (matches['seconds'])

Python 3.4.3 subprocess.Popen get output of command without piping?

I am trying to assign the output of a command to a variable without the command thinking that it is being piped. The reason for this is that the command in question gives unformatted text as output if it is being piped, but it gives color formatted text if it is being run from the terminal. I need to get this color formatted text.
So far I've tried a few things. I've tried Popen like so:
output = subprocess.Popen(command, stdout=subprocess.PIPE)
output = output.communicate()[0]
output = output.decode()
print(output)
This will let me print the output, but it gives me the unformatted output that I get when the command is piped. That makes sense, as I'm piping it here in the Python code. But I am curious if there is a way to assign the output of this command, directly to a variable, without the command running the piped version of itself.
I have also tried the following version that relies on check_output instead:
output = subprocess.check_output(command)
output = output.decode()
print(output)
And again I get the same unformatted output that the command returns when the command is piped.
Is there a way to get the formatted output, the output the command would normally give from the terminal, when it is not being piped?

Using pexpect:
2.py:
import sys
if sys.stdout.isatty():
print('hello')
else:
print('goodbye')
subprocess:
import subprocess
p = subprocess.Popen(
['python3.4', '2.py'],
stdout=subprocess.PIPE
)
print(p.stdout.read())
--output:--
goodbye
pexpect:
import pexpect
child = pexpect.spawn('python3.4 2.py')
child.expect(pexpect.EOF)
print(child.before) #Print all the output before the expectation.
--output:--
hello
Here it is with grep --colour=auto:
import subprocess
p = subprocess.Popen(
['grep', '--colour=auto', 'hello', 'data.txt'],
stdout=subprocess.PIPE
)
print(p.stdout.read())
import pexpect
child = pexpect.spawn('grep --colour=auto hello data.txt')
child.expect(pexpect.EOF)
print(child.before)
--output:--
b'hello world\n'
b'\x1b[01;31mhello\x1b[00m world\r\n'

Yes, you can use the pty module.
>>> import subprocess
>>> p = subprocess.Popen(["ls", "--color=auto"], stdout=subprocess.PIPE)
>>> p.communicate()[0]
# Output does not appear in colour
With pty:
import subprocess
import pty
import os
master, slave = pty.openpty()
p = subprocess.Popen(["ls", "--color=auto"], stdout=slave)
p.communicate()
print(os.read(master, 100)) # Print 100 bytes
# Prints with colour formatting info
Note from the docs:
Because pseudo-terminal handling is highly platform dependent, there
is code to do it only for Linux. (The Linux code is supposed to work
on other platforms, but hasn’t been tested yet.)
A less than beautiful way of reading the whole output to the end in one go:
def num_bytes_readable(fd):
import array
import fcntl
import termios
buf = array.array('i', [0])
if fcntl.ioctl(fd, termios.FIONREAD, buf, 1) == -1:
raise Exception("We really should have had data")
return buf[0]
print(os.read(master, num_bytes_readable(master)))
Edit: nicer way of getting the content at once thanks to #Antti Haapala:
os.close(slave)
f = os.fdopen(master)
print(f.read())
Edit: people are right to point out that this will deadlock if the process generates a large output, so #Antti Haapala's answer is better.

A working polyglot example (works the same for Python 2 and Python 3), using pty.
import subprocess
import pty
import os
import sys
master, slave = pty.openpty()
# direct stderr also to the pty!
process = subprocess.Popen(
['ls', '-al', '--color=auto'],
stdout=slave,
stderr=subprocess.STDOUT
)
# close the slave descriptor! otherwise we will
# hang forever waiting for input
os.close(slave)
def reader(fd):
try:
while True:
buffer = os.read(fd, 1024)
if not buffer:
return
yield buffer
# Unfortunately with a pty, an
# IOError will be thrown at EOF
# On Python 2, OSError will be thrown instead.
except (IOError, OSError) as e:
pass
# read chunks (yields bytes)
for i in reader(master):
# and write them to stdout file descriptor
os.write(1, b'<chunk>' + i + b'</chunk>')

Many programs automatically turn off colour printing codes when they detect they are not connected directly to a terminal. Many programs will have a flag so you can force colour output. You could add this flag to your process call. For example:
grep "search term" inputfile.txt
# prints colour to the terminal in most OSes
grep "search term" inputfile.txt | less
# output goes to less rather than terminal, so colour is turned off
grep "search term" inputfile.txt --color | less
# forces colour output even when not connected to terminal
Be warned though. The actual colour output is done by the terminal. The terminal interprets special character espace codes and changes the text colour and background color accordingly. Without the terminal to interpret the colour codes you will just see the text in black with these escape codes interspersed throughout.

Python string as file argument to subprocess

I am trying to pass a file to a program (MolPro) that I start as subprocess with Python.
It most commonly takes a file as argument, like this in console:
path/molpro filename.ext
Where filename.ex contains the code to execute. Alternatively a bash script (what I'm trying to do but in Python):
#!/usr/bin/env bash
path/molpro << EOF
# MolPro code
EOF
I'm trying to do the above in Python. I have tried this:
from subprocess import Popen, STDOUT, PIPE
DEVNULL = open('/dev/null', 'w') # I'm using Python 2 so I can't use subprocess.DEVNULL
StdinCommand = '''
MolPro code
'''
# Method 1 (stdout will be a file)
Popen(['path/molpro', StdinCommand], shell = False, stdout = None, stderr = STDOUT, stdin = DEVNULL)
# ERROR: more than 1 input file not allowed
# Method 2
p = Popen(['path/molpro', StdinCommand], shell = False, stdout = None, stderr = STDOUT, stdin = PIPE)
p.communicate(input = StdinCommand)
# ERROR: more than 1 input file not allowed
So I am pretty sure the input doesn't look enough like a file, but even looking at Python - How do I pass a string into subprocess.Popen (using the stdin argument)? I can't find what Im doing wrong.
I prefer not to:
Write the string to an actual file
set shell to True
(And I can't change MolPro code)
Thanks a lot for any help!
Update: if anyone is trying to do the same thing, if you don't want to wait for the job to finish (as it doesn't return anything, either way), use p.stdin.write(StdinCommand) instead.

It seems like your second method should work if you remove StdinCommand from the Popen() arguments:
p = Popen(['/vol/thchem/x86_64-linux/bin/molpro'], shell = False, stdout = None, stderr = STDOUT, stdin = PIPE)
p.communicate(input = StdinCommand)