When I run in my Ubuntu terminal:
sudo dd if=/dev/sda of=~/file bs=8k count=200k; rm -f ~/file
it works fine.
If I run it through Pythons subprocess.Popen():
output, err = subprocess.Popen(['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k'], stderr=subprocess.PIPE).communicate()
print err
it doesn't work. The Error I get is:
dd: failed to open '~/disk_benchmark_file': No such file or directory
If I change in the Popen() call the tilde ~ to /home/user, then it works!
Why is it like that? And more important to me: How can I make it work?
I don't know what the user name will be in production.
You need to wrap those pathnames with os.path.expanduser():
>>> import os
>>> os.path.expanduser('~/disk_benchmark_file')
'/home/dan/disk_benchmark_file'
In your code the occurrence of:
['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k']
should be replaced with:
['sudo', 'dd', 'if=/dev/' + disk, 'of=' + os.path.expanduser('~/disk_benchmark_file'), 'bs=8k', 'count=200k']
import os
import shlex
outfile = os.path.expanduser('~/file')
cmd_string = 'sudo dd if=/dev/sda of=%s bs=8k count=200k; rm -f %s' % (outfile, outfile)
cmd_list = shlex.split(cmd_string)
# Then use cmd_list as argument for Popen
shlex.split is the standard and safest way to produce the list that must be used as command in subprocess. It is able to handle all the exception and make your code easier to read
You can find the home using os.path.expanduser('~').
~ is a shortcut in the shell for home. In order for your command to be interpreted by the shell you need to set shell=True in your Popen.
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence
https://docs.python.org/2/library/subprocess.html
Note, there are some warnings about doing this though.
You need to expand the leading ~ in the path to the user's home directory before passing it to Popen. You can use pathlib for this:
from pathlib import Path
Path('~/disk_benchmark_file').expanduser()
Alternatively, you can use Path.home() to get the home directory:
Path(Path.home(), 'disk_benchmark_file')
Related
I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.
subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])
Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.
The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE
At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)
For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])
import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass
I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");
For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])
Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py
No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)
I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.
subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])
Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.
The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE
At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)
For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])
import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass
I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");
For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])
Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py
No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)
I want to call a command like
scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir
from Python to copy many files in one command.
The command works perfectly if entered directly into the shell.
But if I use
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/\{a,b,c\}',
'/target/dir'])
sts = os.waitpid(p.pid, 0)
I get the error
scp: /dir/to/files/{a,b,c}: No such file or directory
Obviously, the backslashes are missing. And if I use double backslashes in the Popen arguments like
\\{a,b,c,d\\}
I get the error
scp: /dir/to/target/\a: No such file or directory
scp: /dir/to/target/\b: No such file or directory
scp: /dir/to/target/\c\: No such file or directory
Nothing changes if I use raw strings like r'\{' + r'\}'
How can I call the scp command from Python with the correctly escaped curly braces '\\{a,b,c\\}'?
cannot test, but I would remove all blackslashes altogether since they're just here to protect the expansion from the shell on the local machine:
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/{a,b,c}',
'/target/dir'])
sts = p.wait()
also note that p.wait() is way better than the wait command you performed (more portable!)
Jean-Francois Fabre got me on the right track:
import subprocess
p = subprocess.Popen('scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir',
shell=True)
sts = p.wait()
The
shell=True
argument was the missing bit. It is a solution that is not recommended, but at least it's working.
Don't use a shell feature unnecessarily in a script; you have your text editor to make typing easier. Just pass the three file names individually:
p = subprocess.Popen(['scp',
'username#hostname:/dir/to/files/a',
'username#hostname:/dir/to/files/b',
'username#hostname:/dir/to/files/c',
'/target/dir'])
Alternatively, let Python build the list of files for you.
file_list = ['username#hostname:/dir/to/files/%s' % (s,)
for f in ['a', 'b', 'c']]
p = subprocess.Popen(['scp'] + file_list + ['/target/dir'])
If I'm getting it correctly, you need to have the "\" and the "{". Since you need to escape both of them, what about 'username#hostname:/dir/to/files/\\\{a,b,c\\\}'
For some reason, no matter how many variations I've tried, I can't seem to execute a bash script I've written. The command words 100% fine in Terminal, but when I try calling it with a subprocess, it returns nothing.
from os import listdir
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for i in listdir(moviefolder):
title = i.split('.')
formatted_title = title[0].replace(' ', '\ ')
if string.lower() == title[0].lower():
command = 'vlc {}/{}.{}'.format(moviefolder, formatted_title, title[1])
subprocess.call(["/usr/local/bin",'-i','-c', command], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
else:
continue
The bash executable file looks like this:
#/bin/bash
func() {
open -a /Applications/VLC.app/Contents/MacOS/VLC $1
}
Where have I gone wrong?
You should call open directly:
import os
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for filename in os.listdir(moviefolder):
title = filename.split('.')
if string.lower() == title[0].lower():
subprocess.call(['open', '-a', '/Applications/VLC.app/Contents/MacOS/VLC', os.path.join(moviefolder, filename)])
Since you are using shell=True, the command must be a string:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. (docs)
Like you even mentioned in a comment, you get /usr/local/bin: is a directory when you properly capture the error from the shell (and take out the erroneous shell=True; or correspondingly refactor the command line to be suitable for this usage, i.e. pass a string instead of a list).
Just to spell this out, you are attempting to run the command /usr/local/bin with some options; but of course, it's not a valid command; so this fails.
The actual script you seem to want to run will declare a function and then exit, which results in the function's definition being lost again, because the subprocess which ran the shell in which this function declaration was executed has now terminated and released all its resources back to the system.
Perhaps you should take more than just a few steps back and explain what you actually want to accomplish; but really, that should be a new, separate question.
Assuming you are actually trying to run vlc, and guessing some other things, too, perhaps you actually want
subprocess.call(['vlc','{}/{}.{}'.format(moviefolder, formatted_title, title[1]),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If your PATH is correct, you should not need to specify /usr/local/bin/ explicitly (and if your PATH is wrong, correct it in the code before, instead of hardcoding a directory for the executable you want to call).
/usr/local/bin is a directory. You can't run a directory as if it were a command.
Anyhow, there's no point to having /usr/local/bin anywhere in your command at all. Leave out the shell=True, and explicitly call vlc:
subprocess.call([
'vlc',
'{}/{}.{}'.format(moviefolder, formatted_title, title[1])
])
When shell=True is used in subprocess.call, if the command arguments is a sequence, then the first element of the sequence needs to be the command, and the rest are treated as argument(s) to the shell itself.
So, this should do:
subprocess.call(["/usr/local/bin/{}".format(command), '-i','-c'], shell=True, ...)
Otherwise, you can make the command a string.
Example:
In [20]: subprocess.call(["cat spamegg", "-i", "-c"], shell=True)
foobar
Language: Python v2.6.2
OS: AIX 5.3
I'm using Python to restore some files from a backup to a test system - all commands are called in the manner below, however some just plain don't want to work.
#!/usr/bin/python
import subprocess, shlex
cmd = 'sudo rm -rf /work/TEST/*'
arg = shlex.split(cmd)
# This does not work
p = subprocess.Popen(arg)
# This, however, works just fine
p = subprocess.Popen(cmd, shell=True)
If I remove the *'s from the commands they work fine (well, they work as they should without the wildcards, which is unfortauntely not what I want).
I really do not want to use shell=True for obvious security reasons, however there are a couple of other commands that basically do the same thing. If there is a wildcard in the command it just won't work - it executes without error, just doesn't do anything.
Interestingly the following command (parsed through shlex):
sudo mv /work/testrestore/production/* /work/TESTC
Produces the following:
mv: 0653-401 Cannot rename /work/testrestore/production/* to /work/TESTC/*: A file or directory in the path name does not exist.
It's as if unix is now trying to move a file named * rather then using * as a wildcard. Is this typical behaviour of shlex?
Edit: I have tried escaping the * with a \, also tried changing from single quotes to double.. not that I expected that to do anything.
For replacing the * with what it means, you either need the shell or you need the glob module. So the easiest way would be shell=True (if the command is constant, I do not see any security holes).
Another approach would be
#!/usr/bin/python
import subprocess
import shlex
import glob
cmd = 'sudo rm -rf /work/TEST/*'
arg = shlex.split(cmd)
arg = arg[:-1] + glob.glob(arg[-1])
# This should work now
p = subprocess.Popen(arg)
or, if you would nevertheless append the path by yourself,
cmd = 'sudo rm -rf'
basearg = shlex.split(cmd)
arg = basearg + glob.glob(path+"/*")