Develop Reference

Uncomplete path recognition (FFmpeg) [duplicate]

I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.

subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])

Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.

The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE

At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)

For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])

import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass

I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");

For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])

Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py

No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)

How to run batch command with spaces from python? [duplicate]

I have a Python script that needs to execute an external program, but for some reason fails.
If I have the following script:
import os;
os.system("C:\\Temp\\a b c\\Notepad.exe");
raw_input();
Then it fails with the following error:
'C:\Temp\a' is not recognized as an internal or external command, operable program or batch file.
If I escape the program with quotes:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe"');
raw_input();
Then it works. However, if I add a parameter, it stops working again:
import os;
os.system('"C:\\Temp\\a b c\\Notepad.exe" "C:\\test.txt"');
raw_input();
What is the right way to execute a program and wait for it to complete? I do not need to read output from it, as it is a visual program that does a job and then just exits, but I need to wait for it to complete.
Also note, moving the program to a non-spaced path is not an option either.
This does not work either:
import os;
os.system("'C:\\Temp\\a b c\\Notepad.exe'");
raw_input();
Note the swapped single/double quotes.
With or without a parameter to Notepad here, it fails with the error message
The filename, directory name, or volume label syntax is incorrect.

subprocess.call will avoid problems with having to deal with quoting conventions of various shells. It accepts a list, rather than a string, so arguments are more easily delimited. i.e.
import subprocess
subprocess.call(['C:\\Temp\\a b c\\Notepad.exe', 'C:\\test.txt'])

Here's a different way of doing it.
If you're using Windows the following acts like double-clicking the file in Explorer, or giving the file name as an argument to the DOS "start" command: the file is opened with whatever application (if any) its extension is associated with.
filepath = 'textfile.txt'
import os
os.startfile(filepath)
Example:
import os
os.startfile('textfile.txt')
This will open textfile.txt with Notepad if Notepad is associated with .txt files.

The outermost quotes are consumed by Python itself, and the Windows shell doesn't see it. As mentioned above, Windows only understands double-quotes.
Python will convert forward-slashed to backslashes on Windows, so you can use
os.system('"C://Temp/a b c/Notepad.exe"')
The ' is consumed by Python, which then passes "C://Temp/a b c/Notepad.exe" (as a Windows path, no double-backslashes needed) to CMD.EXE

At least in Windows 7 and Python 3.1, os.system in Windows wants the command line double-quoted if there are spaces in path to the command. For example:
TheCommand = '\"\"C:\\Temp\\a b c\\Notepad.exe\"\"'
os.system(TheCommand)
A real-world example that was stumping me was cloning a drive in VirtualBox. The subprocess.call solution above didn't work because of some access rights issue, but when I double-quoted the command, os.system became happy:
TheCommand = '\"\"C:\\Program Files\\Sun\\VirtualBox\\VBoxManage.exe\" ' \
+ ' clonehd \"' + OrigFile + '\" \"' + NewFile + '\"\"'
os.system(TheCommand)

For python >= 3.5 subprocess.run should be used in place of subprocess.call
https://docs.python.org/3/library/subprocess.html#older-high-level-api
import subprocess
subprocess.run(['notepad.exe', 'test.txt'])

import win32api # if active state python is installed or install pywin32 package seperately
try: win32api.WinExec('NOTEPAD.exe') # Works seamlessly
except: pass

I suspect it's the same problem as when you use shortcuts in Windows... Try this:
import os;
os.system("\"C:\\Temp\\a b c\\Notepad.exe\" C:\\test.txt");

For Python 3.7, use subprocess.call. Use raw string to simplify the Windows paths:
import subprocess
subprocess.call([r'C:\Temp\Example\Notepad.exe', 'C:\test.txt'])

Suppose we want to run your Django web server (in Linux) that there is space between your path (path='/home/<you>/<first-path-section> <second-path-section>'), so do the following:
import subprocess
args = ['{}/manage.py'.format('/home/<you>/<first-path-section> <second-path-section>'), 'runserver']
res = subprocess.Popen(args, stdout=subprocess.PIPE)
output, error_ = res.communicate()
if not error_:
print(output)
else:
print(error_)
[Note]:
Do not forget accessing permission: chmod 755 -R <'yor path'>
manage.py is exceutable: chmod +x manage.py

No need for sub-process, It can be simply achieved by
GitPath="C:\\Program Files\\Git\\git-bash.exe"# Application File Path in mycase its GITBASH
os.startfile(GitPath)

Import bash variables from a python script

I have seen plenty examples of running a python script from inside a bash script and either passing in variables as arguments or using export to give the child shell access, I am trying to do the opposite here though.
I am running a python script and have a separate file, lets call it myGlobalVariables.bash
myGlobalVariables.bash:
foo_1="var1"
foo_2="var2"
foo_3="var3"
My python script needs to use these variables.
For a very simple example:
myPythonScript.py:
print "foo_1: {}".format(foo_1)
Is there a way I can import them directly? Also, I do not want to alter the bash script if possible since it is a common file referenced many times elsewhere.

If your .bash file is formatted as you indicated - you might be able to just import it direct as a Python module via the imp module.
import imp
bash_module = imp.load_source("bash_module, "/path/to/myGlobalVariables.bash")
print bash_module.foo_1

You can also use os.environ:
Bash:
#!/bin/bash
# works without export as well
export testtest=one
Python:
#!/usr/bin/python
import os
os.environ['testtest'] # 'one'

I am very new to python, so I would welcome suggestions for more idiomatic ways to do this, but the following code uses bash itself to tell us which values get set by first calling bash with an empty environment (env -i bash) to tell us what variables are set as a baseline, then I call it again and tell bash to source your "variables" file, and then tell us what variables are now set. After removing some false-positives and an apparently-blank line, I loop through the "additional" output, looking for variables that were not in the baseline. Newly-seen variables get split (carefully) and put into the bash dictionary. I've left here (but commented-out) my previous idea for using exec to set the variables natively in python, but I ran into quoting/escaping issues, so I switched gears to using a dict.
If the exact call (path, etc) to your "variables" file is different than mine, then you'll need to change all of the instances of that value -- in the subprocess.check_output() call, in the list.remove() calls.
Here's the sample variable file I was using, just to demonstrate some of the things that could happen:
foo_1="var1"
foo_2="var2"
foo_3="var3"
if [[ -z $foo_3 ]]; then
foo_4="test"
else
foo_4="testing"
fi
foo_5="O'Neil"
foo_6='I love" quotes'
foo_7="embedded
newline"
... and here's the python script:
#!/usr/bin/env python
import subprocess
output = subprocess.check_output(['env', '-i', 'bash', '-c', 'set'])
baseline = output.split("\n")
output = subprocess.check_output(['env', '-i', 'bash', '-c', '. myGlobalVariables.bash; set'])
additional = output.split("\n")
# these get set when ". myGlobal..." runs and so are false positives
additional.remove("BASH_EXECUTION_STRING='. myGlobalVariables.bash; set'")
additional.remove('PIPESTATUS=([0]="0")')
additional.remove('_=myGlobalVariables.bash')
# I get an empty item at the end (blank line from subprocess?)
additional.remove('')
bash = {}
for assign in additional:
if not assign in baseline:
name, value = assign.split("=", 1)
bash[name]=value
#exec(name + '="' + value + '"')
print "New values:"
for key in bash:
print "Key: ", key, " = ", bash[key]
Another way to do it:
Inspired by Marat's answer, I came up with this two-stage hack. Start with a python program, let's call it "stage 1", which uses subprocess to call bash to source the variable file, as my above answer does, but it then tells bash to export all of the variables, and then exec the rest of your python program, which is in "stage 2".
Stage 1 python program:
#!/usr/bin/env python
import subprocess
status = subprocess.call(
['bash', '-c',
'. myGlobalVariables.bash; export $(compgen -v); exec ./stage2.py'
]);
Stage 2 python program:
#!/usr/bin/env python
# anything you want! for example,
import os
for key in os.environ:
print key, " = ", os.environ[key]

As stated in #theorifice answer, the trick here may be that such formatted file may be interpreted by both as bash and as python code. But his answer is outdated. imp module is deprecated in favour of importlib.
As your file has extension other than ".py", you can use the following approach:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("foobar", SourceFileLoader("foobar", "myGlobalVariables.bash"))
foobar = module_from_spec(spec)
spec.loader.exec_module(foobar)
I do not completely understand how this code works (where there are these foobar parameters), however, it worked for me. Found it here.

Tilde (~) isn't working in subprocess.Popen()

When I run in my Ubuntu terminal:
sudo dd if=/dev/sda of=~/file bs=8k count=200k; rm -f ~/file
it works fine.
If I run it through Pythons subprocess.Popen():
output, err = subprocess.Popen(['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k'], stderr=subprocess.PIPE).communicate()
print err
it doesn't work. The Error I get is:
dd: failed to open '~/disk_benchmark_file': No such file or directory
If I change in the Popen() call the tilde ~ to /home/user, then it works!
Why is it like that? And more important to me: How can I make it work?
I don't know what the user name will be in production.

You need to wrap those pathnames with os.path.expanduser():
>>> import os
>>> os.path.expanduser('~/disk_benchmark_file')
'/home/dan/disk_benchmark_file'
In your code the occurrence of:
['sudo', 'dd', 'if=/dev/' + disk, 'of=~/disk_benchmark_file', 'bs=8k', 'count=200k']
should be replaced with:
['sudo', 'dd', 'if=/dev/' + disk, 'of=' + os.path.expanduser('~/disk_benchmark_file'), 'bs=8k', 'count=200k']

import os
import shlex
outfile = os.path.expanduser('~/file')
cmd_string = 'sudo dd if=/dev/sda of=%s bs=8k count=200k; rm -f %s' % (outfile, outfile)
cmd_list = shlex.split(cmd_string)
# Then use cmd_list as argument for Popen
shlex.split is the standard and safest way to produce the list that must be used as command in subprocess. It is able to handle all the exception and make your code easier to read
You can find the home using os.path.expanduser('~').

~ is a shortcut in the shell for home. In order for your command to be interpreted by the shell you need to set shell=True in your Popen.
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence
https://docs.python.org/2/library/subprocess.html
Note, there are some warnings about doing this though.

You need to expand the leading ~ in the path to the user's home directory before passing it to Popen. You can use pathlib for this:
from pathlib import Path
Path('~/disk_benchmark_file').expanduser()
Alternatively, you can use Path.home() to get the home directory:
Path(Path.home(), 'disk_benchmark_file')

How do I change my current directory from a python script?

I'm trying to implement my own version of the 'cd' command that presents the user with a list of hard-coded directories to choose from, and the user has to enter a number corresponding to an entry in the list. The program, named my_cd.py for now, should then effectively 'cd' the user to the chosen directory. Example of how this should work:
/some/directory
$ my_cd.py
1) ~
2) /bin/
3) /usr
Enter menu selection, or q to quit: 2
/bin
$
Currently, I'm trying to 'cd' using os.chdir('dir'). However, this doesn't work, probably because my_cd.py is kicked off in its own child process. I tried wrapping the call to my_cd.py in a sourced bash script named my_cd.sh:
#! /bin/bash
function my_cd() {
/path/to/my_cd.py
}
/some/directory
$ . my_cd.sh
$ my_cd
... shows list of dirs, but doesn't 'cd' in the interactive shell
Any ideas on how I can get this to work? Is it possible to change my interactive shell's current directory from a python script?

Change your sourced bash code to:
#! /bin/bash
function my_cd() {
cd `/path/to/my_cd.py`
}
and your Python code to do all of its cosmetic output (messages to the users, menus, etc) on sys.stderr, and, at the end, instead of os.chdir, just print (to sys.stdout) the path to which the directory should be changed.

my_cd.py:
#!/usr/bin/env python
import sys
dirs = ['/usr/bin', '/bin', '~']
for n, dir in enumerate(dirs):
sys.stderr.write('%d) %s\n' % (n+1, dir))
sys.stderr.write('Choice: ')
n = int(raw_input())
print dirs[n-1]
Usage:
nosklo:/tmp$ alias mcd="cd \$(/path/to/my_cd.py)"
nosklo:/tmp$ mcd
1) /usr/bin
2) /bin
3) ~
Choice: 1
nosklo:/usr/bin$

This can't be done. Changes to the working directory are not visible to parent processes. At best you could have the Python script print the directory to change to, then have the sourced script actually change to that directory.

For what its worth, since this question is also tagged "bash", here is a simple bash-only solution:
$ cat select_cd
#!/bin/bash
PS3="Number: "
dir_choices="/home/klittle /local_home/oracle"
select CHOICE in $dir_choices; do
break
done
[[ "$CHOICE" != "" ]] && eval 'cd '$CHOICE
Now, this script must be source'd, not executed:
$ pwd
/home/klittle/bin
$ source select_cd
1) /home/klittle
2) /local_home/oracle
Number: 2
$ pwd
/local_home/oracle
So,
$ alias mycd='source /home/klittle/bin/select_cd'
$ mycd
1) /home/klittle
2) /local_home/oracle
Number:
To solve your case, you could have the command the user runs be an alias that sources a bash script, which does the dir selection first, then dives into a python program after the cd has been done.

Contrary to what was said, you can do this by replacing the process image, twice.
In bash, replace your my_cd function with:
function my_cd() {
exec /path/to/my_cd.py "$BASH" "$0"
}
Then your python script has to finish with:
os.execl(sys.argv[1], sys.argv[2])
Remember to import os, sys at the beginning of the script.
But note that this is borderline hack. Your shell dies, replacing itself with the python script, running in the same process. The python script makes changes to the environment and replaces itself with the shell, back again, still in the same process. This means that if you have some other local unsaved and unexported data or environment in the previous shell session, it will not persist to the new one. It also means that rc and profile scripts will run again (not usually a problem).