Language: Python v2.6.2
OS: AIX 5.3
I'm using Python to restore some files from a backup to a test system - all commands are called in the manner below, however some just plain don't want to work.
#!/usr/bin/python
import subprocess, shlex
cmd = 'sudo rm -rf /work/TEST/*'
arg = shlex.split(cmd)
# This does not work
p = subprocess.Popen(arg)
# This, however, works just fine
p = subprocess.Popen(cmd, shell=True)
If I remove the *'s from the commands they work fine (well, they work as they should without the wildcards, which is unfortauntely not what I want).
I really do not want to use shell=True for obvious security reasons, however there are a couple of other commands that basically do the same thing. If there is a wildcard in the command it just won't work - it executes without error, just doesn't do anything.
Interestingly the following command (parsed through shlex):
sudo mv /work/testrestore/production/* /work/TESTC
Produces the following:
mv: 0653-401 Cannot rename /work/testrestore/production/* to /work/TESTC/*: A file or directory in the path name does not exist.
It's as if unix is now trying to move a file named * rather then using * as a wildcard. Is this typical behaviour of shlex?
Edit: I have tried escaping the * with a \, also tried changing from single quotes to double.. not that I expected that to do anything.
For replacing the * with what it means, you either need the shell or you need the glob module. So the easiest way would be shell=True (if the command is constant, I do not see any security holes).
Another approach would be
#!/usr/bin/python
import subprocess
import shlex
import glob
cmd = 'sudo rm -rf /work/TEST/*'
arg = shlex.split(cmd)
arg = arg[:-1] + glob.glob(arg[-1])
# This should work now
p = subprocess.Popen(arg)
or, if you would nevertheless append the path by yourself,
cmd = 'sudo rm -rf'
basearg = shlex.split(cmd)
arg = basearg + glob.glob(path+"/*")
Related
I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.
subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])
Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.
The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE
At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)
For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])
import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass
I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");
For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])
Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py
No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)
I have the following paths:
airflow_home = os.path.join("opt", "airflow")
airflow_docs = os.path.join(airflow_home, "docs")
And I want airflow_docs path to be used within a bash command. For that, I have used the following code:
subprocess.run([f"sphinx-apidoc -o ./ ../plugins"],
shell=True,
cwd=airflow_docs)
And I get an error FileNotFoundError.
However, this does work:
subprocess.run([f"sphinx-apidoc -o ./ ../{doc_module}"],
shell=True,
cwd="/opt/airflow/docs")
So it seems that a missing leading slash is causing the problem. I have searched in google about adding a leading slash to a path with no success. So, is it possible to use os.path package for subprocess.run, or do I have to use a hardcoded string?
If you want a slash, put a slash.
airflow_home = os.path.join("/opt", "airflow")
But of course, having Python glue together the strings is not really useful. Indeed, the result of os.path.join is simply a string, equivalent to a hard-coded string. So just write it out:
airflow_home = "/opt/airflow"
Or if you want to do this in Python, perhaps prefer pathlib:
airflow_home = pathlib.Path("/opt") / "airflow"
As an aside, your subprocess code is broken; you want to pass either a string, with shell=True, or a list of tokens, without shell=True. (Windows "helpfully" hides this error but it's still wrong.)
subprocess.run(
["sphinx-apidoc", "-o", "./", "../plugins"],
cwd=airflow_docs)
subprocess conveniently allows you to pass in a pathlib.Path object as the value of cwd, though this might not always have been the case, if you need to support older versions of Python.
You probably want to add check=True to have Python raise an error if the subprocess fails. Perhaps see also Running Bash commands in Python
I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.
subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])
Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.
The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE
At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)
For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])
import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass
I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");
For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])
Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py
No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)
When I run in my Ubuntu terminal:
sudo dd if=/dev/sda of=~/file bs=8k count=200k; rm -f ~/file
it works fine.
If I run it through Pythons subprocess.Popen():
output, err = subprocess.Popen(['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k'], stderr=subprocess.PIPE).communicate()
print err
it doesn't work. The Error I get is:
dd: failed to open '~/disk_benchmark_file': No such file or directory
If I change in the Popen() call the tilde ~ to /home/user, then it works!
Why is it like that? And more important to me: How can I make it work?
I don't know what the user name will be in production.
You need to wrap those pathnames with os.path.expanduser():
>>> import os
>>> os.path.expanduser('~/disk_benchmark_file')
'/home/dan/disk_benchmark_file'
In your code the occurrence of:
['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k']
should be replaced with:
['sudo', 'dd', 'if=/dev/' + disk, 'of=' + os.path.expanduser('~/disk_benchmark_file'), 'bs=8k', 'count=200k']
import os
import shlex
outfile = os.path.expanduser('~/file')
cmd_string = 'sudo dd if=/dev/sda of=%s bs=8k count=200k; rm -f %s' % (outfile, outfile)
cmd_list = shlex.split(cmd_string)
# Then use cmd_list as argument for Popen
shlex.split is the standard and safest way to produce the list that must be used as command in subprocess. It is able to handle all the exception and make your code easier to read
You can find the home using os.path.expanduser('~').
~ is a shortcut in the shell for home. In order for your command to be interpreted by the shell you need to set shell=True in your Popen.
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence
https://docs.python.org/2/library/subprocess.html
Note, there are some warnings about doing this though.
You need to expand the leading ~ in the path to the user's home directory before passing it to Popen. You can use pathlib for this:
from pathlib import Path
Path('~/disk_benchmark_file').expanduser()
Alternatively, you can use Path.home() to get the home directory:
Path(Path.home(), 'disk_benchmark_file')
I was trying to use subprocess calls to perform a copy operation (code below):
import subprocess
pr1 = subprocess.call(['cp','-r','./testdir1/*','./testdir2/'], shell = True)
and I got an error saying:
cp: missing file operand
Try `cp --help' for more information.
When I try with shell=False , I get
cp: cannot stat `./testdir1/*': No such file or directory
How do I get around this problem?
I'm using RedHat Linux GNOME Deskop version 2.16.0 and bash shell and Python 2.6
P.S. I read the question posted in Problems with issuing cp command with Popen in Python, and it suggested using shell = True option, which is not working for me as I mentioned :(
When using shell=True, pass a string, not a list to subprocess.call:
subprocess.call('cp -r ./testdir1/* ./testdir2/', shell=True)
The docs say:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
So (on Unix), when a list is passed to subprocess.Popen (or subprocess.call), the first element of the list is interpreted as the command, all the other elements in the list are interpreted as arguments for the shell. Since in your case you do not need to pass arguments to the shell, you can just pass a string as the first argument.
This is an old thread now, but I was just having the same problem.
The problem you were having with this call:
subprocess.call(['cp','-r','./testdir1/*','./testdir2/'], shell = False)
was that each of the parameters after the first one are quoted. So to the shell sees the command like this:
cp '-r' './testdir1/*' './testdir2/'
The problem with that is the wildcard character (*). The filesystem looks for a file with the literal name '*' in the testdir1 directory, which of course, is not there.
The solution is to make the call like the selected answer using the shell = True option and none of the parameters quoted.
I know that the option of shell=True may be tempting but it's always inadvisable due to security issues. Instead, you can use a combination of the subprocess and glob modules.
For Python 3.5 or higher:
import subprocess
import glob
subprocess.run(['cp', '-r'] + glob.glob('./testdir1/*') + ['./testdir2/'])
For Python 3.4 or lower:
import subprocess
import glob
subprocess.call(['cp', '-r'] + glob.glob('./testdir1/*') + ['./testdir2/'])