Develop Reference

how to chain 2 commands on terminal

I am trying to use subprocess in my python script to open Julia and then run a script.
To run on my machine, I enter this in terminal:
$ julia
$ include(test.jl); func("in.csv", "out.csv")
How do I replicate this process and chain both of these commands so that I can run from subprocess in a single call?
I've tried julia; include(test.jl); func("in.csv", "out.csv") and julia && include(test.jl) && func("in.csv", "out.csv")
but both result in
-bash: syntax error near unexpected token `"test.jl"`

The key here is that you're not really chaining two commands from the standpoint of Python's subprocess. There's just one command: julia. You want to pass a somewhat complicated argument to Julia that will execute multiple Julia expressions.
In short, you just want to do:
subprocess.run(['julia','-e','include("test.jl"); func("in.csv", "out.csv")'])
What's happening here is that you're just executing one subprocess, julia, started up with the -e command line flag that just runs whatever comes next in Julia. You can optionally use the capitalized -E flag instead which will print out whatever func (your last expression there) returns.
It's worth pointing out, though, that there are better ways of getting Julia and Python interoperating — especially if you need to transfer data back and forth.

#!/bin/sh vs #!/usr/local/bin/python in executables

In the pip program, the She-bang is
#!/usr/local/bin/python
if __name__ == "__main__":
# Python program body
while in the Install Certificates.command that Python Launcher offers:
#!/bin/sh
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 << "EOF"
# python program body
EOF
Are there any differences between those two approaches? And is there any reason to prefer one to another?
It seems to me they are all the same, except for the second one has one more bash subroutine. Is this right?

In the general case, you simply want to specify the interpreter you actually want.
Outside of this, you sometimes see workarounds like this as portability hacks. On POSIX systems, /usr/bin/env covers the majority of scenarios quite nicely; but if you need portability to older or otherwise peculiar systems, falling back to the lowest common denominator and then working your way back up to a place where you can reliably run e.g. Python on a variety of systems may require all kinds of unobvious constructs. (The previous - upvoted! - answer by Dan D. is a good example.)
There are also cases where you want sh to set something up (fetch some environment variables which are specified in a file which uses sh syntax, for example) and then hand over execution to Python;
#!/bin/sh
# source some variables
. /etc/defaults/myenv.sh
# Then run Python
exec env python -c '
# ... Your Python script here
' "$#"

There is a line length limit on the #! line. Perhaps they did that to get around that.
The options are the path to the program but only if it is short enough. Use of env python which uses the path. Or chain loading like this.

This specific code for the Install Certificates.command script was introduced in Python Issue #17128. As far as I can tell, the author hasn't explained why he wrote the code this way.
Note that .command files are Shell scripts on Mac OS X that can be executed by double-clicking on them in Finder.
I believe the likely explanation is that the author simply wanted to honour Mac OS X's expectation that .command files should be Shell scripts.
You could test this by placing the following content in a file ~/Desktop/test.command:
#!/usr/bin/env python
print "Hello world"
Then view the Desktop folder in Finder, and note that it is reported as a "shell" file:
(Although it is reported incorrectly as a Shell file, this Python script can still be executed by double-clicking on it. It doesn't break Finder or anything.)
To answer the specific question, one reason for preferring this pattern might be, as Dan D. said, to avoid a Shebang line limit.
In general, you would prefer to use #!/usr/bin/env python as your Shebang line. Creating a Bash Heredoc (i.e. the python3.6 << EOF pattern) would create all sorts of problems, such as your syntax highlighting won't work, you have to watch out for Bash variable interpolation inside the Heredoc, etc.

How to mix bash with python

I enjoy using unix commands very much, but I came to the point, where I would find embedded python parts useful. This is my code:
#!/bin/bash -
echo "hello!";
exec python <<END_OF_PYTHON
#!/usr/bin/env python
import sys
print ("xyzzy")
sys.exit(0)
END_OF_PYTHON
echo "goodbye!";
However, only "hello" gets printed.
$ ./script.sh
hello!
xyzzy
How can I modify the bash script to fully embedd python? And would it then be possible to pass values from python variables into bash variables? Thanks a lot.

On the exec python ... line, you're exec()ing the Python interpreter on your PATH, so the python image will replace the bash image, and there is absolutely no hope of the echo "goodbye!" ever being executed. If that's what you want, that's fine, but otherwise, just omit the exec.
The shebang (“#!”) line in the python code is completely unnecessary. When you try to run an ordinary file, the kernel sees the “#!”, runs whatever follows it (/usr/bin/env python), and feeds the rest of the file to the stdin of whatever has been run. This is a general facility used to invoke interpreters. Since you are invoking the python interpreter yourself, not asking the kernel to do it, this is neither needed nor useful.
The sys.exit(0) is also unnecessary, since the Python interpreter will naturally exit when it gets to the end of its input (at END_OF_PYTHON) anyway. This means that the import sys is also unnecessary.
In summary, the following is what I would write to achieve what you appear to want to achieve:
#!/bin/bash
echo "hello!";
python <<END_OF_PYTHON
print ("xyzzy")
END_OF_PYTHON
echo "goodbye!";

Don't use exec. That replaces the shell process with the program you're running, so the rest of the script doesn't execute.
#!/bin/bash -
echo "hello!";
python <<END_OF_PYTHON
#!/usr/bin/env python
import sys
print ("xyzzy")
sys.exit(0)
END_OF_PYTHON
echo "goodbye!";

Don't use exec python, just use python.
The exec tells the shell to replace itself with the Python interpreter, so it's no longer running after that point.

Others have answered your specific issue, but in answer to the general question "How to mix bash with python", Xonsh may be useful to you. It's a special shell that allows you to use python and bash side-by-side. There's also sultan if you want to be able to easily call bash from python.

Or maybe utilizing the commenting and quoting feature of both language:
''':'
# bash code below
echo 'hello world (I am bash) !'
python $0
exit 0 # 'exit' is necessary.
#'''
# python code below
import os, sys
print("hello world (I am python) !")
Output:
bash-3.1$ ./bash-with-python
hello world (I am bash) !
hello world (I am python) !

embedding Bash Script in python without using subprocess call

I have been able to use subprocess to embed bash script into python. I happen to navigate through a python code today and stumbled across this line of code below, which also embed bash script into python - using construct analogous to docstring.
#!/bin/bash -
''''echo -n
if [[ $0 == "file" ]]; then
..
fi
'''
Can someone throw light on this approach. What is this approach called, and perhaps the benefits associated. I can obviously see simplicity but I think there's more to this than that.

This is a somewhat clever way to make the file both a valid Python script and a valid bash script. Note that it does not cause a subprocess to magically be spawned. Rather, if the file is evaluated by bash, the bash script will be run, and if it is evaluated by Python, the bash script will be ignored.
It's clever, but probably not a good software engineering practice in general. It usually makes more sense to have separate scripts.
To give a more concrete example (say this file is called "polyglot"):
''''echo hello from bash
exit
'''
print('hello from python')
As you note, bash will ignore the initial quotes, and print "hello from bash", and then exit before reaching the triple quote. And Python will treat the bash script as a string, and ignore it, running the Python script below.
$ python polyglot
hello from python
$ bash polyglot
hello from bash
But naturally, this can usually (and more clearly) be refactored into two scripts, one in each language.

no, that's not embedded into python, the shebang says it's a bash script
the '''' is '' twice, which is just an empty string, it doesn't have any effect.
the ''' is invalid, as the last ' is not closed.

Why embed repo command in a bash script?

I was investigating repo (from Android project) source code.
It start with the following :
#!/bin/sh
magic='--calling-python-from-/bin/sh--'
"""exec" python -E "$0" "$#" """#$magic"
If I understand it well, it means that the script is recalling itself with python.
So there is my question, why do not directly use python.
For example I usually use something like :
#!/usr/bin/env python
I think there is a valuable reason, but I can't figure it out.
Thanks

Answer from the repo people: Purpose of embedding Repo python code into bash script

Google developer Shawn Pearce gives the reason in this discussion:
We need to pass the -E flag, but env on some platforms wasn't taking
it. So I cooked up this work around. It mostly had to do with our
internal desktops at Google; they have a lot of PYTHON environment
flags that we didn't want to inherit into the repo process (because
they are there for normal Google engineers, not Android Google
engineers), and at least at the time env on either Mac OS or Linux (I
can't remember which) was rejecting a shbang line of "#!/usr/bin/env
python -E".

Perl and Ruby have a '-x' command-line switch, used to do something like setting up shell environment variables before starting the interpeter itself -- mixing shell commands and perl/ruby in the same file:
#!/bin/sh
export PERL5LIB=/some/perl/lib/path:$PERL5LIB
export FOO=1
exec perl -x $0 $#
# ^^^^ ---- shell commands above this line ---- ^^^^
#!perl
# vvvv ---- perl script below this line ---- vvvv
use strict;
print "Hello world\n":
The 'magic' bit in repo is the author's solution to this problem -- but less flexible and far more obtuse. It's sadly a missing feature in python.