In Linux, we usually add a shebang in a script to invoke the respective interpreter. I tried the following example.
I wrote a shell script without a shebang and with executable permission. I was able to execute it using ./. But if I write a similar python program, without shebang, I am not able to execute it.
Why is this so? As far as my understanding, shebang is required to find the interpreter. So how does shell scripts work, but not a python script?
My assumption is that a script without a shebang is executed in the current environment, which at the command line is your default shell, e.g. /bin/bash.
shell scripts will only work if you are in the shell you targeted ... there is not python shell ... as such python will never work without explicity calling python (via shebang or command line)
By default the shell will try to execute the script. The #! notation came later
There’a subtle distinction here. If the target is a binary or begins with a #! shebang line, then the shell calls execv successfully. If the target is a text file without a shebang, then the call to execv will fail, and the shell is free to try launching it under /bin/sh or something else.
http://en.wikipedia.org/wiki/Shebang_(Unix)
Under Unix-like operating systems, when a script with a shebang is run as a program, the program loader parses the rest of the script's initial line as an interpreter directive; the specified interpreter program is run instead, passing to it as an argument the path that was initially used when attempting to run the script.
Now when the #! is not found, every line is interpreted as native shell command. And hence if you write a bash script and run it under bash shell it will work. If you run the same bash script in say a tcsh shell it will not work without the initial #!/usr/bin/tcsh
Related
Background
Below, I detail two different ways of running Python files - I am confused about the difference between them.
Running the Python file as an executable.
To run a Python file as an executable, I must first set a shebang in my file (# /usr/bin/env python3), then run $ chmod +x filename.py at the command line, then run $ ./filename.py at the command line to execute the file.
Running the Python file through the python3 command line command.
To run a Python file through the python3 command, I open my command line and run $ python3 filename.py.
My Question
I understand that, when running the Python file as an executable, the shebang directs the computer to launch the (in this case) python3 interpreter which will interpret the subsequent code in the file and therefore run the file. When running the file through the python3 command, I understand that this is just another way of directing the computer to launch python3 to interpret the code in the file. To me, these two techniques therefore seem identical.
Am I missing something? What's the difference, if any, between these two ways of running a Python file?
In practice, they're identical.
The shebang is just a convention that tells the OS what to do with the script file that you've chmod-ed to be executable -- i.e., executable with what. Without it, the OS just treats the file as a text file and will try to execute it as a shell script.
From the point of view of the system the shebang line is necessary when the file is run as an executable. After checking the permission bits the OS sends the file to the program loader which determines how to run the program by parsing the first line as an interpreter directive. Based on this the loader executes python3 based on the specified path to the executable. If instead /usr/bin/env was used in shebang line then the env command finds the executable based on PATH ( env can also be used to make other modifications to the environment using NAME=VALUE pairs). The loader then passes to the program the path that was used when the user tried to run the script as an argument.
In the second case the OS just loads python3 and passes the script as an argument, it doesn't care at all about the file and its permissions.
From the point of view of the user (which IMO is more important) the shebang line is just another level of abstraction that hides the details of implementation (in this case what program is used to run the script). This means the line can be changed (perhaps by modifying the environment or using a different executable) without the user having to change the way they invoke the script. Also, if the user puts the script in a location that is in PATH then they can invoke the script from anywhere without first navigating to the directory or remembering the location.
On the other hand invoking python3 directly allows the user to pass additional flags to python3 for example -i for interactive runs and -m to use additional modules such as pdb for debugging.
Edit: Based on #alaniwi's comment below explained the role of env in finding the path of the python3 executable in more detail.
Nope, you have pretty much captured it.
A practical consequence is that the shebang relieves you from having to remember whether it's python3 frobnicate or python frobnicate or sh frobnicate or bash frobnicate or awk frobnicate or perl frobnicate or...
This also makes it easy down the line to change your mind. Many tools of mine have started life as simple shell scripts, then been rewritten in Python or something else; but the calling interface doesn't change.
Before Unix, there was an unbridgable gap between system utilities (which you invoke simply by name) and user scripts (which before the introduction of the shebang always had to be called with an explicit interpreter).You still see remnants of this division in lesser systems. An important consequence was that users were able to easily and transparently wrap or replace standard commands with their own versions. This in some sense democratized the system, and empowered users to try out and evaluate improvement ideas for the system on their own. (Figuring out why your brilliant theory wasn't so great in practice is also an excellent way to learn and improve.) I don't think the importance of this versatility and flexibility can be overstated; it's one of those things which converted us from mere users to enthusiasts.
My book states that:
The first line of all your Python programs should be a shebang line, which tells your computer that you want Python to execute this program. The shebang line for OSX is #! /usr/bin/env python3.
But my program works perfectly fine without the shebang line in the Terminal. Then should I use it in the future? Also is there a fullstop(.) at the end of the shebang line in OSX or not?
There should not be a full stop at the end of the line.
Whether to add the shebang or not depends on how you want to run it. If you always invoke the interpreter explicitly then it is optional, i.e.
$ python3 script.py
does not require the shebang, nor does it require executable permission on the file. You can add the shebang, and the code will still run, in which case it might serve as documentation.
However, if you want to execute it like this:
$ ./script.py
or
$ /path/to/script/script.py
then you will need to add the shebang and set executable permission on the file (see chmod).
Why running a python file doesn't require the x permission when running it like this:
python script.py
But it does when it's being run as:
./script.py
Because what you are running with python script.py is the python program; then, it loads and runs the script that you specified in parameters, that is script.py (basically a text file). The script file doesn't need to be an executable because what is executed here is the python interpreter (the python binary itself, that should obviously have the x permission).
With .\script.py, you try to run directly your script (still the same text file) as a program. When you do that, you want it to be parsed with the interpreter that you specified in the first line of your script code, the "shebang", e.g. #!/usr/bin/env python. If it's not set with the x permission, the OS doesn't try to "execute" your file (though it might try to open it with the default program, where applicable), so, it will not care about the shebang.
The file itself it interpreted (read) rather than actually executed in your first example. The python application is what needs execute rights.
In the second example the file itself is being executed, so needs those rights in order to proceed.
When we run a script as python script.py we actually invoke the python interpreter which is generally located at /usr/bin/python (The output of which python will tell you where exactly).
The interpreter in turn reads the scripts and executes its code.
It is the interpreter that has the execute permission.
When a script is executed as ./script.py then the script is executed directly and hence the script requires execute permission. The interpreter used is specified by shebang line.
When the kernel finds that the first two bytes are #! then it uses the rest of the line as interpreter and passes the file as argument. Note that to do this the file needs to have execute permission. In the former case we are indirectly doing what the kernel would do had we executed the script as ./script.py
In short for executing by method1 the interpreter needs only read permission but for later it needs to execute it directly
One difference is that "./script.py" only works if script.py is executable (as in file permissions), but "python script.py" works regardless. However, I strongly suspect there are more differences, and I want to know what they are.
I have a Django website, and "python manage.py syncdb" works just fine, but "./manage.py syncdb" creates a broken database for some reason that remains a mystery to me. Maybe it has to do with the fact that syncdb prompts for a superuser name and password from the command line, and maybe using "./manage.py syncdb" changes the way it interacts with the command line, thus mangling the password. Maybe? I am just baffled by this bug. "python manage.py syncdb" totally fixes it, so this is just curiosity.
Thanks.
Edit: Right, right, I forgot about the necessity of the shebang line #!/usr/bin/python. But I just checked, "python manage.py syncdb" and "./manage.py syncdb" are using the same Python interpreter (2.7.2, the only one installed, on Linux Mint 12). Yet the former works and the latter does not.
Could the environment variables seen by the Python code be different? My code does require $LD_LOADER_PATH and $PYTHON_PATH to be set special for each shell.
Calling ./script.py uses the "shebang line" in the script to determine which interpreter to use to run the script. Such a line might look like
#!/usr/bin/env python
or
#!/usr/bin/python2.7
or whatever path to the python interpreter is used. If it resolves to the same Python interpreter that is called by just
python
from the shell command line, there is no difference between ./script.py and python script.py, but the two version can end up using different Python interpreters.
./script.py = "Attempt to execute a file called script.py in the current shell"
python script.py = "Send script.py as an argument to the first python executable in the current $PATH"
The first only works if the file has the execute bit set for the user attempting to execute the file and it has the so-called shebang line, which tells the shell how to run it.
In Linux using terminal you can execute any file -if the user has execute permission- by typing
./fileName. When the OS sees a valid header like #! /usr/bin/python (or for perl #! /usr/bin/python), It will call the python or perl (appropriate) interpreter to execute the program. You can use the command python script.py directly because, python is a executable program located at /usr/bin (or somewhere else) which is in a environmental variable $PATH,
that corresponding to directory of executables.
./script.py runs the interpreter defined in the #! at the beginning of the file. For example, the first line might be #! /usr/bin/env python or #! /usr/bin/python or something else like that. If you look at what interpreter is invoked, you might be able to fix that problem.
I've a shell script with two shebangs, the first one tells #!/bin/sh and after a few lines the other one is #!/usr/bin/env python.
When this script is given executable permission and ran as ./script.sh, the script works fine, uses /bin/sh in first part and uses python interpreter in latter part.
But when the script is run as sh script.sh, the second shebang is not recognized and the script fails. Is there anyway I can force to change the interpreter if the script is run explicitly as sh script.sh.
The reason I need this is because I need to run the scripts through a tool which runs as sh script.sh
As far as I know you cannot have two shebang lines in one script. The shebang works only when -
it is on the first line
it starts in column one
If you need to run python code then have it in another script and then call the script by doing
python path/to/the/script.py
A better way to do this would be to use a shell here document. Something like this:
#!/bin/sh
curdir=`pwd`
/usr/bin/env python <<EOF
import os
print os.listdir("$curdir")
EOF
This way, you won't need to distribute the code on two separate files.
As you see, you can even access shell variables from the python code.
have your script.sh script invoke the python interpreter directly:
#!/bin/sh
# include full path if needed
python (your python interpreter arguments)