Develop Reference

Difference Between Running Python File as Executable vs. Running from Command Line?

Background
Below, I detail two different ways of running Python files - I am confused about the difference between them.
Running the Python file as an executable.
To run a Python file as an executable, I must first set a shebang in my file (# /usr/bin/env python3), then run $ chmod +x filename.py at the command line, then run $ ./filename.py at the command line to execute the file.
Running the Python file through the python3 command line command.
To run a Python file through the python3 command, I open my command line and run $ python3 filename.py.
My Question
I understand that, when running the Python file as an executable, the shebang directs the computer to launch the (in this case) python3 interpreter which will interpret the subsequent code in the file and therefore run the file. When running the file through the python3 command, I understand that this is just another way of directing the computer to launch python3 to interpret the code in the file. To me, these two techniques therefore seem identical.
Am I missing something? What's the difference, if any, between these two ways of running a Python file?

In practice, they're identical.
The shebang is just a convention that tells the OS what to do with the script file that you've chmod-ed to be executable -- i.e., executable with what. Without it, the OS just treats the file as a text file and will try to execute it as a shell script.

From the point of view of the system the shebang line is necessary when the file is run as an executable. After checking the permission bits the OS sends the file to the program loader which determines how to run the program by parsing the first line as an interpreter directive. Based on this the loader executes python3 based on the specified path to the executable. If instead /usr/bin/env was used in shebang line then the env command finds the executable based on PATH ( env can also be used to make other modifications to the environment using NAME=VALUE pairs). The loader then passes to the program the path that was used when the user tried to run the script as an argument.
In the second case the OS just loads python3 and passes the script as an argument, it doesn't care at all about the file and its permissions.
From the point of view of the user (which IMO is more important) the shebang line is just another level of abstraction that hides the details of implementation (in this case what program is used to run the script). This means the line can be changed (perhaps by modifying the environment or using a different executable) without the user having to change the way they invoke the script. Also, if the user puts the script in a location that is in PATH then they can invoke the script from anywhere without first navigating to the directory or remembering the location.
On the other hand invoking python3 directly allows the user to pass additional flags to python3 for example -i for interactive runs and -m to use additional modules such as pdb for debugging.
Edit: Based on #alaniwi's comment below explained the role of env in finding the path of the python3 executable in more detail.

Nope, you have pretty much captured it.
A practical consequence is that the shebang relieves you from having to remember whether it's python3 frobnicate or python frobnicate or sh frobnicate or bash frobnicate or awk frobnicate or perl frobnicate or...
This also makes it easy down the line to change your mind. Many tools of mine have started life as simple shell scripts, then been rewritten in Python or something else; but the calling interface doesn't change.
Before Unix, there was an unbridgable gap between system utilities (which you invoke simply by name) and user scripts (which before the introduction of the shebang always had to be called with an explicit interpreter).You still see remnants of this division in lesser systems. An important consequence was that users were able to easily and transparently wrap or replace standard commands with their own versions. This in some sense democratized the system, and empowered users to try out and evaluate improvement ideas for the system on their own. (Figuring out why your brilliant theory wasn't so great in practice is also an excellent way to learn and improve.) I don't think the importance of this versatility and flexibility can be overstated; it's one of those things which converted us from mere users to enthusiasts.

shebang/hashbang, (that is, "#!") at the beginning of a script [duplicate]

Which of these is better or faster to use as the shebang line for a Perl script?
#! perl
#! perl.exe
#! fullpath/perl(/perl.exe)
#! partialpath/perl(/perl.exe)
And, when using #!perl, when it works on a particular system, how do I find out in the script which perl interpreter I'm using so I can put that one into the shebang line?
And, if using a /path/path/perl, are * or ... allowed to be used for the folders?

If you have to hard code #!, use #!/usr/bin/env perl. Why? What you want is for the Perl program to run with the user's preferred Perl. That's going to be the first on in their PATH. #!perl doesn't do what I mean, it doesn't search the user's PATH, #!/usr/bin/env perl is how you pull that off. /usr/bin/env will always be there on Unix systems.
If the user is using Windows, as others have pointed out, it doesn't matter. Windows doesn't use #! it uses file extension associations. Make sure your program is called foo.pl or something and it'll work. But include the #! line anyway as some utilities and editors make use of it.
If you're shipping code, let the installer take care of it. Both MakeMaker/Makefile.PL and Module::Build/Build.PL will change your #! line to match the perl the user used to install with. They will take care of this problem for you.
If you are installing code for your own production use, you should use the full path to a particular copy of perl. Which copy of perl? One specific to your project. Does this mean you need to compile perl for every project? No, you can make a symlink. Project foo might have /usr/local/bin/fooperl point at /usr/bin/perl5.18. Use #!/usr/local/bin/fooperl. Now if you decide to upgrade perl you can do it per project by changing the symlink.

If you are running CGI via Apache on Windows, the SHEBANG IS USED. You will need the fullpath to perl.

A Windows she-bang (deduced from the perl.exe bit) seems irrelevant since your (ahem) "shell" probably does not even parse it (correct me if I am wrong, could have been changed lately).
Some command line flags may still be picked up by Perl itself though (according to this thread).

As ChristopheD noted, I can confirm from practice (ActivePerl on XP) that the shebang line is not really necessary on Windows.
A shebang line tells a Unix shell which interpreter to pass the script to.
On Windows, the program to pass the script to will be determined by associations based on the extension.
On Unix, the third option (full path to perl executable) is best.
And yes, you can use ".." in theory (shell doesn't care) but you should not really use relative path - you never know what your current working directory when executing a script will be.

If you're developing in Unix using Perl and you use "perlbrew" to easily switch between different versions of Perl, then the "#!/usr/bin/env perl" shebang line works well.
I originally had the first 2 characters in the shebang line reversed. Just fixed/edited that.

The first line stands for shebang. It basically tells the program where Perl interpreter is located since Perl is interpreted language. On Linux you can type in terminal:
whereis perl
which will give you exact location of it. Usually it's inside /usr/bin/perl. This means that you want to make shebang regarding to /usr/bin/perl
#! /usr/bin/perl
use strict;
use warnings;
use v5.10.1;
.
.
.
This is just some good practice, hence it's obviously fastest solution.
I hope you find this useful,
Thanks.

And, when using "#! perl", when it works on a particular system, what is the print() for showing the full path to perl.exe, that could be included into the Shebang Line ?
Well, if you're using the print statement you are already executing perl code, so...

This is one of the things which I dislike about Perl.
On Windows, if you are using ActiveState Perl at least, if the file ends with .pl then the Windows registry will run the Perl interpreter for you, regardless of the shebang line. On Cygwin, I am not sure why but #! perl works too. On Unix you should put the full path to your Perl executable in the shebang line. Schwern's idea of using env is convenient, but has some danger, as I pointed out in a comment.
This is why I suggest to you that the best solution is to package your Perl scripts as CPAN modules. CPAN installers like Module::Build will then change the shebang line to the full path to your Perl interpreter. (I am not sure whether Schwern's installer, ExtUtils::MakeMaker, does this or uses env, since I don't use it.)

Python script gives `: No such file or directory`

I have several python scripts which work just fine but one script has (as of this morning) started giving me this error if I try to run it from the bash:
: No such file or directory
I am able to run the 'broken' script by doing python script_name.py and after looking around a bit the general idea that I picked up was that maybe my line ending of the hashbang got changed (silently) so I looked at the line ending of a working script and a broken script via the :set list option in VI as indicated in this question -> View line-endings in a text file
Both files appear to end using the same character (a $) so I am kind of at a loss on how to proceed from here. Specifically, how to actually 'see' the line ending in case the set list was not the right method.
PS: The script is executable and the shebang is in there, I stated that it's just this 1 script that was working fine before the weekend but it started giving me this error as of this morning.
-- edit: --
Running the script through dos2unix does get it working again but I would like to know of any way to visualize the line ending somehow in VI(M) or why Geany somehow converted the line endings in the first place (as I never work on a dos/windows system anyhow).

From the comments above it looks like you have dos line endings, and so the hashbang line is not properly processed.
Line ending style are not shown with :set list in Vim because that option is only used when reading/writing the file. In memory line endings are always that, line-endings. The line ending style used for a file is kept in a Vim per-file option, weirdly called fileformat.
To see/change the line ending style from Vim, you can use the following commands:
:set fileformat
:set ff
It will show dos or unix. You want unix, of course ;-).
To change it quickly you can save the file with:
:w ++ff=unix
Or if you prefer:
:set ff=unix
And then save the file normally.
So see all the gory details just do :help fileformat, :help file-formats and :help fileformats

You can also use the dos2unix command to convert the file format
dos2unix
This helped me to run the python scripts
This normally happens when we open files in windows do changes and save it.
if you open the file locate the ^M characters at the end of every line
Thanks

Personally, I find it kinda wrong using direct path to python interpreter. As you dont use windows platform, you should have program env, usually in /usr/bin (/usr/bin/env). Try using following shebang:
#!/usr/bin/env python
Different distros store python binary in /bin or /usr/bin (or some weird locations), and this one makes your script config-independent (as far as possible, here we have possibility that env is stored elsewhere; still - it is less possible that env is not in /usr/bin than that python is mislocated).
I had similiar problem (if not exactly the same) and that worked for me.
Also, I have both python interpreters (2.7.x and 3.x) installed, so I need to use "python3" argument for env. AFAIR usually distros link different names to different binaries, so "env python" will run python2.7 on my system, "env python3" (also python33, or smth like that) will run p3k, and "env python2" (also python27, etc) will run python 2.7.x. Declaring which version of interpreter should be used seems like a good idea too.

I came across this problem editing my code on Windows, checking it in with git, and checking out and running it on Linux.
My solution was: tell git to Do The Right Thing. I issued this command on the Windows box:
git config --global core.autocrlf true
Modified the files and checked them in; voila, no such problem any more.
As discussed on the Git documentation.

How can I start the python shell and automatically initialize it with some commands?

I find that when I start the python shell I have a bunch of commands I always type to get into the state I want. It is tiresome to keep re-typing these commands, so I have bundled them into a script. Now I just type:
execfile('script.py')
as soon as I enter the shell, and it goes through all the steps to get me to the state I need to be in.
Now I'd like to take it one step further. How can I get the python shell to automatically run script.py every time I start the shell, so I don't have to keep re-typing even that one line?

Here's a way without having to mess with environment variables:
For example, if I had a script with the following in it called script.py:
#!/usr/bin/env python
print("example")
I could tell python to run this before bringing me to the interpreter with the -i flag.
$ python -i script.py
example
>>>

I think you're looking for the PYTHONSTARTUP environment variable

I'd suggest you to use IPython, if possible. It gives tones of great features, and autoexec is only one of them. But of course, correct answer is mentioned by #mgilston

Create a file called usercustomize.py, and place it in your USER_SITE directory (which you can find as import site; site._script(). This file will be executed every time you start an interpreter.
There's a similar file called sitecustomize.py which is executed anytime anyone starts Python.

This is a Windows solution, but I'm sure a Unix equivalent could be done. I created a file \MyPythonCode\autoexec.py (the last line of which is a print("Autoexec Complete")) and a BAT file MyPython.Bat which has:
cd \myPythonCode
python -i autoexec.py
I just use the command: mypython
to start the Python environment every time, now.

What does it mean in linux scripts? #!/usr/bin/python -tt

I know that in the begining of .sh bash scripts is
#!/bin/bash
which points to the command interpeter executable.
But during watching Google Python Class http://www.youtube.com/watch?v=tKTZoB2Vjuk I noticed that for python they use
#!/usr/bin/python -tt
. Surfing the Internet I also have found such styles of this notation:
#!/usr/local/bin/python
and even
#!/usr/bin/env python
.
So, I'm new with Python and I'm ordinary Linux user and I have a few questions about this "magic" line:
First of all, what is the right form of this line? and why?
What does -tt key means in #!/usr/bin/python -tt ?
What program is parsing this line in Linux?
What syntax of this line for any script?
Why this line is so necessary if each file have it's extension?
And what about that in each computer interpreter for some kind of scripts will be stored in different place than in another? And script couldn't be run.
It's really interesting to me.
What's this line? Why this line? How to write this line? Why in such a way?...

Question #1) The line is called a shebang, and there's no right form that works universally. e.g.
#!python
#!/usr/bin/python
#!/usr/local/bin/python
#!/usr/bin/python -t
are all valid/acceptable forms, but may not work on all systems:
#!python will work only if the python executable is somewhere in your shell's PATH
#!/usr/bin/python only works if the python binary is actually in /usr/bin
#!/usr/local/bin/python also only works if python is in /usr/local/bin
Question #2)
#!/usr/bin/python -tt is passing the -tt option to python, as if you'd done:
$ python -t somescript.py
at the shell prompt. You can pass arbitary command line arguments to the interpreter on the shebang line.
Question #3)
The line is interpreted by the OS kernel and the shell you're currently using. The stuff after the #! simply tells the OS which program should be fired up to "execute" the rest of the script.
Question #4)
The script syntax depends on the language you're using. E.g. a PHP shell script must take the form of
#!/usr/bin/php
<?php
... php code here ...
A #!/usr/bin/perl perl script must use Perl syntax, etc... If you put PHP code with a Perl shebang, you'll just have Perl barf up the script with syntax errors, as PHP code is NOT perl code
Question #5)
Shebangs are for Unix systems, where file extensions were never really used to identify file types to the OS. A .c file was understood to be a C language source code file, but that's merely a convention. You could put a Bash shell script into a .c file, make it executable, and with the #!/bin/bash shebang, it would execute as a Bash script.
Determining executable types by file extension is more of a Windows thing.
Question #6)
That goes back the stuff in question #1 - if the shebang claims the interpreter is at some OTHER path than where it is, this particular script can't be executed until the shebang is fixed, or the interpreter is moved. Shebangs are very handy, but not infallible.
Thankfully, most interpreters are installed in fairly standard locations these days, so it'd be somewhat unusual to find (say) Perl installed at /some/wonky/weird/path instead of /usr/bin

From the manpage:
-t Issue a warning when a source file mixes tabs and spaces
for indentation in a way that makes it depend on the worth of a tab
expressed in spaces. Issue an
error when the option is given twice.
The right form of the line is the one you want to use.
It's the interpreter that reads this line known as shebang. If you write a python script with first line as "#!/usr/bin/python" & invoke it using bash, it's the /bin/sh interpreter that reads first line and starts the proper interpreter.
It's a shebang. The syntax of feature consists of the character sequence #!, i.e. the number sign and an exclamation point character
File extensions are not relevant in linux generally. You can have a python script that doesn't have a .py extension.
For ex.
shadyabhi#archlinux ~ $ cat a
print "Hello World"
shadyabhi#archlinux ~ $ python2 a
Hello World
shadyabhi#archlinux ~ $
Even the shebangs are only necessary if you want to start a script using $./script as in this case you didn't mention the interpreter you want to use.

#!/usr/bin/env python
issue errors about inconsistent tab usage
Kernel
#!/path_to_the_interpreter or /usr/bin/env
*nix does not check extensinon at all(except some DE could do that)
This is why you should use #!/usr/bin/env
More info at wiki

The different paths are to where the python interpreter has been installed. Different flavours of Linux install it in different places.
Linux doesn't care for extensions its a Windows thing.
The bash session uses the line to call the correct interpreter for the script your running.
The different places to where the files are stored, called and used are all based on defined places where files should be and located by software. Dev for devices, home for user stored area, bin for programs. But as time has gone by, different systems require different locations.
I would suggest getting a book on Linux/Unix and learning the basics of the file system. It does help a lot.

This is called a shebang. It tells the system that it should pass the file as an argument to the specified program instead of trying to execute it per se.
First of all, what is the right form of this line? and why?
The correct path is wherever your python interpreter is installed. The arguments (-tt) will depend on what you want. Some people insist on #!/usr/bin/env in case the interpreter happens to be elewhere.
What does -tt key means in #!/usr/bin/python -tt ?
I don't use python, so someone else will have to answer this.
When I launch any script in Linux (not exact Python script) what program parses and uses this line? I consider that it's not bash because even for bash scripts this line is needed.
I've heard (and am pretty sure) it's the kernel. Even if it were bash, it would need the line to tell bash it is supposed to be a script it should interpret instead of passing to another program. /usr/bin/env is a command that searches the PATH for the specified argument and passes the script through the program it finds.
What syntax of this line for any script? And what's the name of interpeter that parses it?
The syntax is the same as a command line, #!command arguments, but command has to be an absolute path, the PATH doesn't get searched.
Why this line is so necessary if each file have it's extension?
Extensions mean nothing in *nix. I could name a bash script script.pl, script.exe, or even script without an extension. If the script has the right shebang line, it gets passed through the right interpreter, otherwise the kernel tries to execute it as an executable and fails. The system doesn't know about extensions. They're a convention for users, nothing more.
And what about that in each computer interpreter for some kind of scripts will be stored in different place than in another? And script couldn't be run.
If I understand this correctly, you're saying different systems / distributions keep the interpreters in different places (e.g. /usr/bin/python and /usr/local/bin/python), and asking how the system knows which to use?
The answer is, it uses the one which is at the absolute path you gave it. This is actually a slight problem with executable scripts, and the reason why /usr/bin/env has come into vogue. As I said, env searches the PATH for the correct interpreter, so as long as your system has a /usr/bin/env, you're set, you don't need to look up or guarantee the location of the interpreter.