Develop Reference

Command gives different behaviour in terminal or in program (Python and C++)

I am trying to execute the following command programatically:
~$ obabel -:"ccco" -O /home/user/output.png
obabel is a chemistry library, and basically if the string in the "" is complete nonsense chemically, it won't be able to generate the PNG file, and if it is a legitimate chemical structure the PNG file will be generated. This works in the terminal.
However, if I call the same command with Python, PNG files are generated for complete nonsense input strings which don't generate a PNG when the command is executed in the terminal.
I'm using subprocess like this:
cmd = 'obabel -:"ccco" -O /home/user/output.png'
proc = sub.Popen([cmd], shell=True, stderr=sub.PIPE)
res = proc.communicate()
I have also tried this:
os.system(cmd)
And tried Python2 and Python3. This happens when running scripts from the terminal or iPython.
I have also tried using C++ and running the cmd like this:
std::string cmd = "obabel -:\"ccco\" -O /home/user/output.png";
system(cmd.c_str());

By default Popen expects a list of string arguments, but if you pass shell=True, you can supply the command as a simple string (it will be executed in a shell). Currently you are passing in a list with one string that contains the entirety of the command, instead you can use either of these:
proc = subprocess.Popen('obabel -:"ccco" -O output.png', shell=True, stderr=subprocess.PIPE)
proc = subprocess.Popen(['obabel', '-:ccco', '-O', 'output.png'], stderr=subprocess.PIPE)
Escaping the SMILES string with quotes seems to be done to protect it from the shell, and you don't need it when passing the input directly (otherwise the " characters will be a part of the string and cause invalid syntax).

Calling scp from Python's subprocess not working with filelist \{a,b,c\}

I want to call a command like
scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir
from Python to copy many files in one command.
The command works perfectly if entered directly into the shell.
But if I use
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/\{a,b,c\}',
'/target/dir'])
sts = os.waitpid(p.pid, 0)
I get the error
scp: /dir/to/files/{a,b,c}: No such file or directory
Obviously, the backslashes are missing. And if I use double backslashes in the Popen arguments like
\\{a,b,c,d\\}
I get the error
scp: /dir/to/target/\a: No such file or directory
scp: /dir/to/target/\b: No such file or directory
scp: /dir/to/target/\c\: No such file or directory
Nothing changes if I use raw strings like r'\{' + r'\}'
How can I call the scp command from Python with the correctly escaped curly braces '\\{a,b,c\\}'?

cannot test, but I would remove all blackslashes altogether since they're just here to protect the expansion from the shell on the local machine:
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/{a,b,c}',
'/target/dir'])
sts = p.wait()
also note that p.wait() is way better than the wait command you performed (more portable!)

Jean-Francois Fabre got me on the right track:
import subprocess
p = subprocess.Popen('scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir',
shell=True)
sts = p.wait()
The
shell=True
argument was the missing bit. It is a solution that is not recommended, but at least it's working.

Don't use a shell feature unnecessarily in a script; you have your text editor to make typing easier. Just pass the three file names individually:
p = subprocess.Popen(['scp',
'username#hostname:/dir/to/files/a',
'username#hostname:/dir/to/files/b',
'username#hostname:/dir/to/files/c',
'/target/dir'])
Alternatively, let Python build the list of files for you.
file_list = ['username#hostname:/dir/to/files/%s' % (s,)
for f in ['a', 'b', 'c']]
p = subprocess.Popen(['scp'] + file_list + ['/target/dir'])

If I'm getting it correctly, you need to have the "\" and the "{". Since you need to escape both of them, what about 'username#hostname:/dir/to/files/\\\{a,b,c\\\}'

Bash script will not run with subprocess in Python

For some reason, no matter how many variations I've tried, I can't seem to execute a bash script I've written. The command words 100% fine in Terminal, but when I try calling it with a subprocess, it returns nothing.
from os import listdir
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for i in listdir(moviefolder):
title = i.split('.')
formatted_title = title[0].replace(' ', '\ ')
if string.lower() == title[0].lower():
command = 'vlc {}/{}.{}'.format(moviefolder, formatted_title, title[1])
subprocess.call(["/usr/local/bin",'-i','-c', command], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
else:
continue
The bash executable file looks like this:
#/bin/bash
func() {
open -a /Applications/VLC.app/Contents/MacOS/VLC $1
}
Where have I gone wrong?

You should call open directly:
import os
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for filename in os.listdir(moviefolder):
title = filename.split('.')
if string.lower() == title[0].lower():
subprocess.call(['open', '-a', '/Applications/VLC.app/Contents/MacOS/VLC', os.path.join(moviefolder, filename)])

Since you are using shell=True, the command must be a string:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. (docs)

Like you even mentioned in a comment, you get /usr/local/bin: is a directory when you properly capture the error from the shell (and take out the erroneous shell=True; or correspondingly refactor the command line to be suitable for this usage, i.e. pass a string instead of a list).
Just to spell this out, you are attempting to run the command /usr/local/bin with some options; but of course, it's not a valid command; so this fails.
The actual script you seem to want to run will declare a function and then exit, which results in the function's definition being lost again, because the subprocess which ran the shell in which this function declaration was executed has now terminated and released all its resources back to the system.
Perhaps you should take more than just a few steps back and explain what you actually want to accomplish; but really, that should be a new, separate question.
Assuming you are actually trying to run vlc, and guessing some other things, too, perhaps you actually want
subprocess.call(['vlc','{}/{}.{}'.format(moviefolder, formatted_title, title[1]),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If your PATH is correct, you should not need to specify /usr/local/bin/ explicitly (and if your PATH is wrong, correct it in the code before, instead of hardcoding a directory for the executable you want to call).

/usr/local/bin is a directory. You can't run a directory as if it were a command.
Anyhow, there's no point to having /usr/local/bin anywhere in your command at all. Leave out the shell=True, and explicitly call vlc:
subprocess.call([
'vlc',
'{}/{}.{}'.format(moviefolder, formatted_title, title[1])
])

When shell=True is used in subprocess.call, if the command arguments is a sequence, then the first element of the sequence needs to be the command, and the rest are treated as argument(s) to the shell itself.
So, this should do:
subprocess.call(["/usr/local/bin/{}".format(command), '-i','-c'], shell=True, ...)
Otherwise, you can make the command a string.
Example:
In [20]: subprocess.call(["cat spamegg", "-i", "-c"], shell=True)
foobar

In subprocess, some commands work but not others from Mac terminal

I am trying to create a python script that runs a perl script on the Mac terminal. The popular 3D printer slicing engine, Slic3r, has the ability to use command line usage, which is written in Perl. I want to write a python script to automate some processes, which is the language I know best. If I type the commands I want to use directly into the terminal, it works as it should, however, if I try to use python's subprocess, it works for some commands but not others.
For example if I use my script to fetch the Slic3r version using the syntax outlined in the docs, it works correctly. This script works:
import os
import subprocess
os.system("cd Slic3r")
perl = "perl"
perl_script = "/Users/path/to/Slic3r/slic3r.pl"
params = "--version"
pl_script = subprocess.Popen([perl, perl_script, params], stdout=sys.stdout)
pl_script.communicate()
print 'done'
This returns:
1.3.0-dev
done
If I use a command such as --info (see Slic3r docs under repairing models for more info) using the same script I get:
In:
import os
import subprocess
os.system("cd Slic3r")
perl = "perl"
perl_script = "/Users/path/to/Slic3r/slic3r.pl"
params = "--info /Users/path/to/Desktop/STL_Files/GEAR.stl"
pl_script = subprocess.Popen([perl, perl_script, params], stdout=sys.stdout)
pl_script.communicate()
print 'done'
Out:
Unknown option: info /Users/path/to/Desktop/STL_Files/GEAR.stl
Slic3r 1.3.0-dev is a STL-to-GCODE translator for RepRap 3D printers
written by Alessandro Ranellucci <aar#cpan.org> - http://slic3r.org/
Usage: slic3r.pl [ OPTIONS ] [ file.stl ] [ file2.stl ] ...
From what I have researched, I suspect that there is some issue with the whitespace of a string being used as a argument. I have never used subprocess until attempting this project, so a simple syntax error could be likely.
I know that the Slic3r syntax is correct because it works perfectly if I type it directly into the terminal. Can anybody see what I am doing wrong?

subprocess.Popen accepts args as the first parameter. This can be either a string with the complete command (including parameters):
args = "perl /Users/path/to/Slic3r/slic3r.pl --info /Users/path/to/Desktop/STL_Files/GEAR.stl"
pl_script = subprocess.Popen(args, stdout=sys.stdout)
or a list consisting of the actual command and all its parameters (the actual command in your case is perl):
args = ["perl",
"/Users/path/to/Slic3r/slic3r.pl",
"--info",
"/Users/path/to/Desktop/STL_Files/GEAR.stl"]
pl_script = subprocess.Popen(args, stdout=sys.stdout)
The latter is preferred because it bypasses the shell and directly executes perl. From the docs:
args should be a sequence of program arguments or else a single
string. By default, the program to execute is the first item in args
if args is a sequence. If args is a string, the interpretation is
platform-dependent and described below. See the shell and executable
arguments for additional differences from the default behavior. Unless
otherwise stated, it is recommended to pass args as a sequence.
(emphasis mine)
The args list may of course be built with Python's standard list operations:
base_args = ["perl",
"/Users/path/to/Slic3r/slic3r.pl"]
options = ["--info",
"/Users/path/to/Desktop/STL_Files/GEAR.stl"]
args = base_args + options
args.append("--verbose")
pl_script = subprocess.Popen(args, stdout=sys.stdout)
Sidenote: You wrote os.system("cd Slic3r"). This will open a shell, change the directory in that shell, and then exit. Your Python script will still operate in the original working directory. To change it, use os.chdir("Slic3r") instead. (See here.)

you can also use shlex to break down the complex arguments expecially in mac or unix
more information here
https://docs.python.org/2/library/shlex.html#shlex.split
e.g.
import shlex, subprocess
args = "perl /Users/path/to/Slic3r/slic3r.pl --info /Users/path/to/Desktop/STL_Files/GEAR.stl"
#using shlex to break down the arguments
mac_arg=shlex.split(args)
#shlex.split will return all the arguments in a list
output
['perl', '/Users/path/to/Slic3r/slic3r.pl', '--info', '/Users/path/to/Desktop/STL_Files/GEAR.stl']
This can then further be used with Popen
p1=Popen(mac_arg)
Shlex main adavantage is that you dont need to worry about the commands , it will always split them in a manner accepted by Popen

Getting console output of a Perl script through Python

There are a variety of posts and resources explaining how to use Python to get output of an outside call. I am familiar with using these--I've used Python to get output of jars and exec several times, when it was not realistic or economical to re-implement the functionality of that jar/exec inside Python itself.
I am trying to call a Perl script via Python's subprocess module, but I have had no success with this particular Perl script. I carefully followed the answers here, Call Perl script from Python, but had no results.
I was able to get the output of this test Perl script from this question/answer: How to call a Perl script from Python, piping input to it?
#!/usr/bin/perl
use strict;
use warnings;
my $name = shift;
print "Hello $name!\n";
Using this block of Python code:
import subprocess
var = "world"
args_test = ['perl', 'perl/test.prl', var]
pipe = subprocess.Popen(args_test, stdout=subprocess.PIPE)
out, err = pipe.communicate()
print out, err
However, if I swap out the arguments and the Perl script with the one I need output from, I get no output at all.
args = ['perl', 'perl/my-script.prl', '-a', 'perl/file-a.txt',
'-t', 'perl/file-t.txt', 'input.txt']
which runs correctly when entered on the command line, e.g.
>perl perl/my-script.prl -a perl/file-a.txt -t perl/file-t.txt input.txt
but this produces no output when called via subprocess:
pipe = subprocess.Popen(args, stdout=subprocess.PIPE)
out, err = pipe.communicate()
print out, err
I've done another sanity check as well. This correctly outputs the help message of Perl as a string:
import subprocess
pipe = subprocess.Popen(['perl', '-h'], stdout=subprocess.PIPE)
out, err = pipe.communicate()
print out, err
As shown here:
>>> ================================ RESTART ================================
>>>
Usage: perl [switches] [--] [programfile] [arguments]
-0[octal] specify record separator (\0, if no argument)
-a autosplit mode with -n or -p (splits $_ into #F)
-C[number/list] enables the listed Unicode features
-c check syntax only (runs BEGIN and CHECK blocks)
-d[:debugger] run program under debugger
-D[number/list] set debugging flags (argument is a bit mask or alphabets)
-e program one line of program (several -e's allowed, omit programfile)
-f don't do $sitelib/sitecustomize.pl at startup
-F/pattern/ split() pattern for -a switch (//'s are optional)
-i[extension] edit <> files in place (makes backup if extension supplied)
-Idirectory specify #INC/#include directory (several -I's allowed)
-l[octal] enable line ending processing, specifies line terminator
-[mM][-]module execute "use/no module..." before executing program
-n assume "while (<>) { ... }" loop around program
-p assume loop like -n but print line also, like sed
-P run program through C preprocessor before compilation
-s enable rudimentary parsing for switches after programfile
-S look for programfile using PATH environment variable
-t enable tainting warnings
-T enable tainting checks
-u dump core after parsing program
-U allow unsafe operations
-v print version, subversion (includes VERY IMPORTANT perl info)
-V[:variable] print configuration summary (or a single Config.pm variable)
-w enable many useful warnings (RECOMMENDED)
-W enable all warnings
-x[directory] strip off text before #!perl line and perhaps cd to directory
-X disable all warnings
None