On my Mac, I am using the following command to call g++ in python:
subprocess.run(['g++', './rmc-output/*.cpp', '-w', '-o', 'executable'], check=True,
stderr=PIPE, stdout=PIPE, shell=True)
however, I get the following error while the rmc-output folder is not empty.
clang: error: no such file or directory: './rmc-output/*.cpp'
Am I missing something?
shell=True won't expand wildcards when arguments are put in a list. Quickfix: use a string:
subprocess.run('g++ ./rmc-output/*.cpp -w -o executable', check=True,
stderr=PIPE, stdout=PIPE, shell=True)
quick but dirty, so a better solution:
drop shell=True (avoid whenever possible, security issues, lazy command lines...)
use glob to compute files using python, not the shell
like this:
import glob
subprocess.run(['g++'] + glob.glob('./rmc-output/*.cpp') +['-w', '-o', 'executable'], check=True,
stderr=PIPE, stdout=PIPE)
note that Windows versions of g++ have internal wildcard expansion to make up for the fact that Windows "shell" hasn't. Would have worked on Windows probably.
*.cpp is a shell wildcard pattern. Since you're specifying the command arguments directly, wildcard filename expansion never happens, and it's using *.cpp as a literal filename.
See this answer https://stackoverflow.com/a/9997093/494134 for more details.
Related
I am using Python 3.10.6 and Miniconda 3
What I'm trying to do is write a python script which will open a .cmd file. simple enough, but the problem is it has to be opened in anaconda. I've been able to have my script open the .cmd through the standard command line, and I've been able to have the script open anaconda. But I'm unable to do both in sequence, open the anaconda terminal and then pass a command to it, instructing it to open the .cmd
opening anaconda:
path = 'C:\\Users\...\shortcuts\\conda.lnk'
cmds = 'C:\\Users\\...test.cmd'
cmd_result = subprocess.run(path, shell=True, check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
text=True )
print(cmd_result.args)
print(cmd_result.stdout)
this works properly when i open either anaconda path or test.cmd directly using the subprocess.run function. but when I attempt to open test.cmd as a command argument in the anaconda terminal, it doesn't work
path = 'C:\\Users\...\shortcuts\\conda.lnk'
cmds = [path,'/C','C:\\Users\\...test.cmd']
cmd_result = subprocess.run(cmds, shell=True, check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
text=True )
I believe this is the relevant part of the error code it returns
File "C:\Users\...dream_restart_module.py", line 17, in dream_restart
cmd_result = subprocess.run(cmds, shell=True, check=True,
File "C:\Users\...Python310\lib\subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\...shortcuts\\conda.lnk', '/C', 'C:\\Users\\...test.cmd']' returned non-zero exit status 1.
I feel like I am missing something obvious here, perhaps about the syntax or limitations of subprocess. Any guidance would be appreciated
When you use subprocess.run() with shell=True, the arguments you give it as part of your cmds list are passed directly to the shell, rather than as arguments to the initial command.
So if you intend to pass '/C' and 'C:\\Users\\...test.cmd' as arguments to path, you should either not declare shell=True (preferred) or provide your cmds as a single, space-separated string (less preferred), like path+' /C C:\\Users\\...test.cmd' or ' '.join(cmds).
I'm struggling to understand why this fails with a wget: missing URL error:
import shlex
import subprocess
copy_command = "wget -O - 'http://example.com/somepath/somefile.txt?someparam=test' | sshpass -p pass ssh user#localhost -p 2222 \"cat - > /upload/somefile.txt\""
cmd = shlex.split(copy_command, posix=False)
with subprocess.Popen(
cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True
) as proc:
output, error = proc.communicate()
What am I missing here? If I just give subprocess the copy_command string directly, then it works without issues.
To set up a pipeline requires the parent process to spawn all the programs involved and the connect (pipe) the stdio of one to another.
The Python documentation for subprocess explains how to do this.
It works with string argument andshell=True because then it just hands off the command line to a sub shell, and that shell handles all those details.
When executing this command from the terminal:
\grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo
It outputs the correct results, where it excludes the --exclude-dir directories.
The following Python script should theoretically perform the exact same operation:
#!/usr/bin/env python3
from subprocess import Popen, PIPE
cmd = 'grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo'
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for l in p.stdout:
print(l)
But the --exclude-dir flag seems to be ignored, that is, it also grep from node_modules, .meteor and .git.
Question: Why is the above code outputting a different result from just executing the command?
Note that this is more of a theory question than me looking for an alternative fix, since the Python code is basically from a plugin that I'm using where I can only supply flags for the command, and just that, to my surprise, passing in --exclude-dir does nothing. If there's something wrong with the code above, do point it out though.
System info:
OS: macOS 10.13.6
Python: 3.7.0
grep: (BSD grep) 2.5.1-FreeBSD (with --exclude-dir support)
--exclude-dir={dir1,dir2} is expanded to
--exclude-dir=dir1 --exclude-dir=dir2
by the shell, not by grep. Popen uses shell=False by default.
So instead use
from subprocess import Popen, PIPE
cmd = '''grep -inH -r --exclude-dir=node_modules --exclude-dir=.meteor
--exclude-dir=.git -e test -- /Users/nelsyeung/Sites/foo'''
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for l in p.stdout:
print(l)
(Note that while using shell=True might be another option, it is not preferred because of security issues.)
For whatever reason,
from subprocess import Popen, PIPE, STDOUT
handle = Popen('echo %CD%', shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE, cwd=r'C:\test')
print 'STDOUT::',handle.stdout.readline().strip()
Gives me:
STDOUT:: C:\Documents and Settings\myusername\Desktop
Windows XP (for whatever reason)
Python 2.6.6
Does this perhaps have any similarities to how windows executes commands in row such as:
C:\> cd "C:\test\" && echo %CD%
C:\
Which, is wrong.
If you are just trying to change the current working directory, you could use:
os.chdir('c:\\test')
Note the double backslash as the backslash is a special character in Python that needs to be escaped.
Also, to check your current working directory, you can use:
os.getcwd()
Tested with Win7 and Python 2.6 and 2.7, works as expected (STDOUT:: C:\test).
So this seems to be related to WinXP.
I am writing a small python script that needs to execute git commands from inside a given directory
The code is as follows:
import subprocess, os
pr = subprocess.Popen(['/usr/bin/git', 'status'],
cwd=os.path.dirname('/path/to/dir/'),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
(out, error) = pr.communicate()
print out
But it shows git usage as the output.
If the command doesn't involve git for eg. ['ls'] then it shows the correct output.
Is there anything I am missing ?
python version - 2.6.6
Thanks.
subprocess.Popen:
On Unix, with shell=True: […] If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
You don't want shell=True and also a list of arguments. Set shell=False.