This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 7 years ago.
I am trying to filter out first 3 line of /proc/meminfo using pipe and head command.
so basically i need to run this in Python:
cat /proc/meminfo | head -3
I am using below line in my code :
subprocess.call(["cat", "/proc/meminfo", "|", "head", "-3"])
While just using subprocess.call(["cat", "/proc/meminfo"]) I am getting whole list but I am just interested in first 3 line.
Using above command is giving me below error:
cat: invalid option -- '3'
Try `cat --help' for more information.
Any suggestions?
/proc/meminfo is just a file. You don't need a subprocess to read it. Simply open and read it as a file. Here is all you need:
fh = open('/proc/meminfo', 'r')
lines = fh.readlines()
fh.close()
first_lines = lines[:3]
The first_lines list will contain the first three lines (including trailing newline characters).
To use pip you have to enable shell as shell=True, however it's not advisable specifically because of security reason . You can do this alternative,
import subprocess
ps = subprocess.Popen(('cat', '/proc/meminfo'),stdout=subprocess.PIPE)
output = subprocess.check_output(('head', '-3'), stdin=ps.stdout)
print output
The pipe is a shell syntax element. You need to run the code in a shell to use a pipe:
subprocess.call(["cat /proc/meminfo | head -3"], shell=True)
From the manual:
If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
Well head actually accepts an argument, so the pipe is not actually necessary. The following should give the expected result.
subprocess.call(["head", "-3", "/proc/meminfo"])
following this document
In default, subprocess.call with shell=False will disables all shell based features including pipe. When using shell=True, pipes.quote() can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.
you can use this code
subprocess.call("cat /proc/meminfo | head -3", shell=True)
Related
I'm currently having a major issue with a python script. The script runs arbitrary commands through a handler to convert incorrect error reporting into correct error reporting.
The issue I'm having is getting the script to work correctly on windows with a command that contains ampersands in it's path. I've attempted quoting the command, escaping the ampersand with ^ and neither works. I'm now out of ideas. Any suggestions?
To clarify from current responses:
I am using the subprocess module
I am passing the command line + arguments in as a list
The issue is with the path to the command itself, not any of the arguments
I've tried quoting the command. It causes a [Error 123] The filename, directory name, or volume label syntax is incorrect error
I'm using no shell argument (so shell=false)
In case it matters, I'm grabbing a pipe to stderr for processing it, but ignoring stdout and stdin
It is only for use on Windows currently, and works as expected in all other cases that I've tested so far.
The command that is failing is:
p = subprocess.Popen(prog, stderr = subprocess.PIPE, bufsize=-1)
when the first element of the list 'prog' contains any ampersands. Quoting this first string does not work.
Make sure you are using lists and no shell expansion:
subprocess.Popen(['command', 'argument1', 'argument2'], shell=False)
Try quoting the argument that contains the &
wget "http://foo.com/?bar=baz&baz=bar"
Is usually what has to be done in a Linux shell
To answer my own question:
Quoting the actual command when passing the parameters as a list doesn't work correctly (command is first item of list) so to solve the issue I turned the list into a space separated string and passed that into subprocess instead.
Better solutions still welcomed.
"escaping the ampersand with ^"
Are you sure ^ is an escape character to Windows? Shouldn't you use \?
I try a situation as following:
exe = 'C:/Program Files (x86)/VideoLAN/VLC/VLC.exe'
url = 'http://translate.google.com/translate_tts?tl=en&q=hello+world'
subprocess.Popen([exe, url.replace("&","^&")],shell=True)
This does work.
I would like to be able to call some executables that take in parameters and then dump the output to a file. I've attempted to use both os.system and subprocess calls to no avail. Here is a sample of what I'd like python to execute for me...
c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt
Notice the absolute paths as I will be traversing hundreds of various directories to act on files etc..
Many thanks ahead of time!
Some examples that I've tried:
subprocess.run(['c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt']
subprocess.call(r'"c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt"']
subprocess.call(r'"c:\directory\executable_program.exe" -f "w:\directory\input_file.txt > Z\directory\output_file.txt"']
Your attempts contain various amounts of quoting errors.
subprocess.run(r'c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt', shell=True)
should work, where the r prefix protects the backslashes from being interpreted and removed by Python before the subprocess runs, and the absence of [...] around the value passes it verbatim to the shell (hence, shell=True).
On Windows you could get away with putting the command in square brackets even though it's not a list, and omitting shell=True in some circumstances.
If you wanted to avoid the shell, try
with open(r'Z\directory\output_file.txt', 'wb') as dest:
subprocess.run(
[r'c:\directory\executable_program.exe', '-f', r'w:\directory\input_file.txt'],
stdout=dest)
which also illustrates how to properly pass a list of strings in square brackets as the first argument to subprocess.run.
I have a command which I use for deployment from windows command line. Now I need to run the same from an external python3.4 script.
The command is C:\Program Files (x86)\MSBuild\12.0\Bin\msbuild "D:\WebService\WebService.sln" /p:DeployOnBuild=true /p:PublishProfile="D:\WebService\Properties\PublishProfiles\MyDeployment.pubxml" /p:AllowUntrustedCertificate=true /p:UserName=name /p:Password=PASSWORD.
How can I achieve this. I tried subprocess . But it's not working.Please help me out.
Your problems appear to be the \ and " characters, so use raw strings. Also, it is safer to use a list:
proc = subprocess.Popen(
[r"C:\Program Files (x86)\MSBuild\12.0\Bin\msbuild",
r"D:\WebService\WebService.sln",
r"/p:DeployOnBuild=true",
r"/p:PublishProfile=D:\WebService\Properties\PublishProfiles\MyDeployment.pubxml",
r"/p:AllowUntrustedCertificate=true",
r"/p:UserName=name",
r"/p:Password=PASSWORD"])
proc.wait()
Strictly speaking you don't need raw strings for all those parameters, but it is safer to do so with Windows paths. You only need the internal double quotes if you have embedded whitespace (as the first parameter). Here we are not using a shell, set shell=True as a parameter if you need one. A reason to use a shell on Windows is for filename association, but you don't appear to be using that here.
Can you post some code with what you have tried so far?
The subprocess module should be able to handle that, with something like
theproc = subprocess.Popen(["COMMAND HERE"])
theproc.communicate()
or you could try with the shell flag
theproc = subprocess.Popen(["COMMAND HERE"], shell=True)
I would like to start out by saying any help is greatly appreciated. I'm new to Python and scripting in general. I am trying to use a program called samtools view to convert a file from .sam to a .bam I need to be able do what this BASH command is doing in Python:
samtools view -bS aln.sam > aln.bam
I understand that BASH commands like | > < are done using the subprocess stdin, stdout and stderr in Python. I have tried a few different methods and still can't get my BASH script converted correctly. I have tried:
cmd = subprocess.call(["samtools view","-bS"], stdin=open(aln.sam,'r'), stdout=open(aln.bam,'w'), shell=True)
and
from subprocess import Popen
with open(SAMPLE+ "."+ TARGET+ ".sam",'wb',0) as input_file:
with open(SAMPLE+ "."+ TARGET+ ".bam",'wb',0) as output_file:
cmd = Popen([Dir+ "samtools-1.1/samtools view",'-bS'],
stdin=(input_file), stdout=(output_file), shell=True)
in Python and am still not getting samtools to convert a .sam to a .bam file. What am I doing wrong?
Abukamel is right, but in case you (or others) are wondering about your specific examples....
You're not too far off with your first attempt, just a few minor items:
Filenames should be in quotes
samtools reads from a named input file, not from stdin
You don't need "shell=True" since you're not using shell tricks like redirection
So you can do:
import subprocess
subprocess.call(["samtools", "view", "-bS", "aln.sam"],
stdout=open('aln.bam','w'))
Your second example has more or less the same issues, so would need to be changed to something like:
from subprocess import Popen
with open('aln.bam', 'wb',0) as output_file:
cmd = Popen(["samtools", "view",'-bS','aln.sam'],
stdout=(output_file))
You can pass execution to the shell by kwarg 'shell=True'
subprocess.call('samtools view -bS aln.sam > aln.bam', shell=True)
I'm currently having a major issue with a python script. The script runs arbitrary commands through a handler to convert incorrect error reporting into correct error reporting.
The issue I'm having is getting the script to work correctly on windows with a command that contains ampersands in it's path. I've attempted quoting the command, escaping the ampersand with ^ and neither works. I'm now out of ideas. Any suggestions?
To clarify from current responses:
I am using the subprocess module
I am passing the command line + arguments in as a list
The issue is with the path to the command itself, not any of the arguments
I've tried quoting the command. It causes a [Error 123] The filename, directory name, or volume label syntax is incorrect error
I'm using no shell argument (so shell=false)
In case it matters, I'm grabbing a pipe to stderr for processing it, but ignoring stdout and stdin
It is only for use on Windows currently, and works as expected in all other cases that I've tested so far.
The command that is failing is:
p = subprocess.Popen(prog, stderr = subprocess.PIPE, bufsize=-1)
when the first element of the list 'prog' contains any ampersands. Quoting this first string does not work.
Make sure you are using lists and no shell expansion:
subprocess.Popen(['command', 'argument1', 'argument2'], shell=False)
Try quoting the argument that contains the &
wget "http://foo.com/?bar=baz&baz=bar"
Is usually what has to be done in a Linux shell
To answer my own question:
Quoting the actual command when passing the parameters as a list doesn't work correctly (command is first item of list) so to solve the issue I turned the list into a space separated string and passed that into subprocess instead.
Better solutions still welcomed.
"escaping the ampersand with ^"
Are you sure ^ is an escape character to Windows? Shouldn't you use \?
I try a situation as following:
exe = 'C:/Program Files (x86)/VideoLAN/VLC/VLC.exe'
url = 'http://translate.google.com/translate_tts?tl=en&q=hello+world'
subprocess.Popen([exe, url.replace("&","^&")],shell=True)
This does work.