Develop Reference

FileNotFound Exception Raised When Reading Using No Path [duplicate]

My book states:
Every program that runs on your computer has a current working directory, or cwd. Any filenames or paths that do not begin with the root folder are assumed to be under the current working directory
As I am on OSX, my root folder is /. When I type in os.getcwd() in my Python shell, I get /Users/apple/Documents. Why am I getting the Documents folder in my cwd? Is it saying that Python is using Documents folder? Isn't there any path heading to Python that begins with / (the root folder)? Also, does every program have a different cwd?

Every process has a current directory. When a process starts, it simply inherits the current directory from its parent process; and it's not, for example, set to the directory which contains the program you are running.
For a more detailed explanation, read on.
When disks became large enough that you did not want all your files in the same place, operating system vendors came up with a way to structure files in directories. So instead of saving everything in the same directory (or "folder" as beginners are now taught to call it) you could create new collections and other new collections inside of those (except in some early implementations directories could not contain other directories!)
Fundamentally, a directory is just a peculiar type of file, whose contents is a collection of other files, which can also include other directories.
On a primitive operating system, that was where the story ended. If you wanted to print a file called term_paper.txt which was in the directory spring_semester which in turn was in the directory 2021 which was in the directory studies in the directory mine, you would have to say
print mine/studies/2021/spring_semester/term_paper.txt
(except the command was probably something more arcane than print, and the directory separator might have been something crazy like square brackets and colons, or something;
lpr [mine:studies:2021:spring_semester]term_paper.txt
but this is unimportant for this exposition) and if you wanted to copy the file, you would have to spell out the whole enchilada twice:
copy mine/studies/2021/spring_semester/term_paper.txt mine/studies/2021/spring_semester/term_paper.backup
Then came the concept of a current working directory. What if you could say "from now on, until I say otherwise, all the files I am talking about will be in this particular directory". Thus was the cd command born (except on old systems like VMS it was called something clunkier, like SET DEFAULT).
cd mine/studies/2021/spring_semester
print term_paper.txt
copy term_paper.txt term_paper.backup
That's really all there is to it. When you cd (or, in Python, os.chdir()), you change your current working directory. It stays until you log out (or otherwise exit this process), or until you cd to a different working directory, or switch to a different process or window where you are running a separate command which has its own current working directory. Just like you can have your file browser (Explorer or Finder or Nautilus or whatever it's called) open with multiple windows in different directories, you can have multiple terminals open, and each one runs a shell which has its own independent current working directory.
So when you type pwd into a terminal (or cwd or whatever the command is called in your command language) the result will pretty much depend on what you happened to do in that window or process before, and probably depends on how you created that window or process. On many Unix-like systems, when you create a new terminal window with an associated shell process, it is originally opened in your home directory (/home/you on many Unix systems, /Users/you on a Mac, something more or less like C:\Users\you on recent Windows) though probably your terminal can be configured to open somewhere else (commonly Desktop or Documents inside your home directory on some ostensibly "modern" and "friendly" systems).
Many beginners have a vague and incomplete mental model of what happens when you run a program. Many will incessantly cd into whichever directory contains their script or program, and be genuinely scared and confused when you tell them that you don't have to. If frobozz is in /home/you/bin then you don't have to
cd /home/you/bin
./frobozz
because you can simply run it directly with
/home/you/bin/frobozz
and similarly if ls is in /bin you most definitely don't
cd /bin
./ls
just to get a directory listing.
Furthermore, like the ls (or on Windows, dir) example should readily convince you, any program you run will look in your current directory for files. Not the directory the program or script was saved in. Because if that were the case, ls could only produce a listing of the directory it's in (/bin) -- there is nothing special about the directory listing program, or the copy program, or the word processor program; they all, by design, look in the current working directory (though again, some GUI programs will start with e.g. your Documents directory as their current working directory, by design, at least if you don't tell them otherwise).
Many beginners write scripts which demand that the input and output files are in a particular directory inside a particular user's home directory, but this is just poor design; a well-written program will simply look in the current working directory for its input files unless instructed otherwise, and write output to the current directory (or perhaps create a new directory in the current directory for its output if it consists of multiple files).
Python, then, is no different from any other programs. If your current working directory is /Users/you/Documents when you run python then that directory is what os.getcwd() inside your Python script or interpreter will produce (unless you separately os.chdir() to a different directory during runtime; but again, this is probably unnecessary, and often a sign that a script was written by a beginner). And if your Python script accepts a file name parameter, it probably should simply get the operating system to open whatever the user passed in, which means relative file names are relative to the invoking user's current working directory.
python /home/you/bin/script.py file.txt
should simply open(sys.argv[1]) and fail with an error if file.txt does not exist in the current directory. Let's say that again; it doesn't look in /home/you/bin for file.txt -- unless of course that is also the current working directory of you, the invoking user, in which case of course you could simply write
python script.py file.txt
On a related note, many beginners needlessly try something like
with open(os.path.join(os.getcwd(), "input.txt")) as data:
...
which needlessly calls os.getcwd(). Why is it needless? If you have been following along, you know the answer already: the operating system will look for relative file names (like here, input.txt) in the current working directory anyway. So all you need is
with open("input.txt") as data:
...
One final remark. On Unix-like systems, all files are ultimately inside the root directory / which contains a number of other directories (and usually regular users are not allowed to write anything there, and system administrators with the privilege to do it typically don't want to). Every relative file name can be turned into an absolute file name by tracing the path from the root directory to the current directory. So if the file we want to access is in /home/you/Documents/file.txt it means that home is in the root directory, and contains you, which contains Documents, which contains file.txt. If your current working directory were /home you could refer to the same file by the relative path you/Documents/file.txt; and if your current directory was /home/you, the relative path to it would be Documents/file.txt (and if your current directory was /home/you/Music you could say ../Documents/file.txt but let's not take this example any further now).
Windows has a slightly different arrangement, with a number of drives with single-letter identifiers, each with its own root directory; so the root of the C: drive is C:\ and the root of the D: drive is D:\ etc. (and the directory separator is a backslash instead of a slash, although you can use a slash instead pretty much everywhere, which is often a good idea for preserving your sanity).

Your python interpreter location is based off of how you launched it, as well as subsequent actions taken after launching it like use of the os module to navigate your file system. Merely starting the interpreter will place you in the directory of your python installation (not the same on different operating systems). On the other hand, if you start by editing or running a file within a specific directory, your location will be the folder of the file you were editing. If you need to run the interpreter in a certain directory and you are using idle for example, it is easiest to start by creating a python file there one way or another and when you edit it you can start a shell with Run > Python Shell which will already be in that directory. If you are using the command line interpreter, navigate to the folder where you want to run your interpreter before running the python/python3/py command. If you need to navigate manually, you can of course use the following which has already been mentioned:
import os
os.chdir('full_path_to_your_directory')

This has nothing to do with osx in particular, it's more of a concept shared by all unix-based systems, and I believe Windows as well. os.getcwd() is the equivalent of the bash pwd command - it simply returns the full path of the current location in which you are in. In other words:
alex#suse:~> cd /
alex#suse:/> python
Python 2.7.12 (default, Jul 01 2016, 15:34:22) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd()
'/'
It depends from where you started the python shell/script.

Python is usually (except if you are working with virtual environments) accessible from any of your directory. You can check the variables in your path and Python should be available. So the directory you get when you ask Python is the one in which you started Python. Change directory in your shell before starting Python and you will see you will it.

os.getcwd() has nothing to do with OSX in particular. It simply returns the directory/location of the source-file. If my source-file is on my desktop it would return C:\Users\Dave\Desktop\ or let say the source-file is saved on an external storage device it could return something like G:\Programs\. It is the same for both unix-based and Windows systems.

Directory and path said to exist, but not appearing when using terminal to access directory (python)? [duplicate]

My book states:
Every program that runs on your computer has a current working directory, or cwd. Any filenames or paths that do not begin with the root folder are assumed to be under the current working directory
As I am on OSX, my root folder is /. When I type in os.getcwd() in my Python shell, I get /Users/apple/Documents. Why am I getting the Documents folder in my cwd? Is it saying that Python is using Documents folder? Isn't there any path heading to Python that begins with / (the root folder)? Also, does every program have a different cwd?

Every process has a current directory. When a process starts, it simply inherits the current directory from its parent process; and it's not, for example, set to the directory which contains the program you are running.
For a more detailed explanation, read on.
When disks became large enough that you did not want all your files in the same place, operating system vendors came up with a way to structure files in directories. So instead of saving everything in the same directory (or "folder" as beginners are now taught to call it) you could create new collections and other new collections inside of those (except in some early implementations directories could not contain other directories!)
Fundamentally, a directory is just a peculiar type of file, whose contents is a collection of other files, which can also include other directories.
On a primitive operating system, that was where the story ended. If you wanted to print a file called term_paper.txt which was in the directory spring_semester which in turn was in the directory 2021 which was in the directory studies in the directory mine, you would have to say
print mine/studies/2021/spring_semester/term_paper.txt
(except the command was probably something more arcane than print, and the directory separator might have been something crazy like square brackets and colons, or something;
lpr [mine:studies:2021:spring_semester]term_paper.txt
but this is unimportant for this exposition) and if you wanted to copy the file, you would have to spell out the whole enchilada twice:
copy mine/studies/2021/spring_semester/term_paper.txt mine/studies/2021/spring_semester/term_paper.backup
Then came the concept of a current working directory. What if you could say "from now on, until I say otherwise, all the files I am talking about will be in this particular directory". Thus was the cd command born (except on old systems like VMS it was called something clunkier, like SET DEFAULT).
cd mine/studies/2021/spring_semester
print term_paper.txt
copy term_paper.txt term_paper.backup
That's really all there is to it. When you cd (or, in Python, os.chdir()), you change your current working directory. It stays until you log out (or otherwise exit this process), or until you cd to a different working directory, or switch to a different process or window where you are running a separate command which has its own current working directory. Just like you can have your file browser (Explorer or Finder or Nautilus or whatever it's called) open with multiple windows in different directories, you can have multiple terminals open, and each one runs a shell which has its own independent current working directory.
So when you type pwd into a terminal (or cwd or whatever the command is called in your command language) the result will pretty much depend on what you happened to do in that window or process before, and probably depends on how you created that window or process. On many Unix-like systems, when you create a new terminal window with an associated shell process, it is originally opened in your home directory (/home/you on many Unix systems, /Users/you on a Mac, something more or less like C:\Users\you on recent Windows) though probably your terminal can be configured to open somewhere else (commonly Desktop or Documents inside your home directory on some ostensibly "modern" and "friendly" systems).
Many beginners have a vague and incomplete mental model of what happens when you run a program. Many will incessantly cd into whichever directory contains their script or program, and be genuinely scared and confused when you tell them that you don't have to. If frobozz is in /home/you/bin then you don't have to
cd /home/you/bin
./frobozz
because you can simply run it directly with
/home/you/bin/frobozz
and similarly if ls is in /bin you most definitely don't
cd /bin
./ls
just to get a directory listing.
Furthermore, like the ls (or on Windows, dir) example should readily convince you, any program you run will look in your current directory for files. Not the directory the program or script was saved in. Because if that were the case, ls could only produce a listing of the directory it's in (/bin) -- there is nothing special about the directory listing program, or the copy program, or the word processor program; they all, by design, look in the current working directory (though again, some GUI programs will start with e.g. your Documents directory as their current working directory, by design, at least if you don't tell them otherwise).
Many beginners write scripts which demand that the input and output files are in a particular directory inside a particular user's home directory, but this is just poor design; a well-written program will simply look in the current working directory for its input files unless instructed otherwise, and write output to the current directory (or perhaps create a new directory in the current directory for its output if it consists of multiple files).
Python, then, is no different from any other programs. If your current working directory is /Users/you/Documents when you run python then that directory is what os.getcwd() inside your Python script or interpreter will produce (unless you separately os.chdir() to a different directory during runtime; but again, this is probably unnecessary, and often a sign that a script was written by a beginner). And if your Python script accepts a file name parameter, it probably should simply get the operating system to open whatever the user passed in, which means relative file names are relative to the invoking user's current working directory.
python /home/you/bin/script.py file.txt
should simply open(sys.argv[1]) and fail with an error if file.txt does not exist in the current directory. Let's say that again; it doesn't look in /home/you/bin for file.txt -- unless of course that is also the current working directory of you, the invoking user, in which case of course you could simply write
python script.py file.txt
On a related note, many beginners needlessly try something like
with open(os.path.join(os.getcwd(), "input.txt")) as data:
...
which needlessly calls os.getcwd(). Why is it needless? If you have been following along, you know the answer already: the operating system will look for relative file names (like here, input.txt) in the current working directory anyway. So all you need is
with open("input.txt") as data:
...
One final remark. On Unix-like systems, all files are ultimately inside the root directory / which contains a number of other directories (and usually regular users are not allowed to write anything there, and system administrators with the privilege to do it typically don't want to). Every relative file name can be turned into an absolute file name by tracing the path from the root directory to the current directory. So if the file we want to access is in /home/you/Documents/file.txt it means that home is in the root directory, and contains you, which contains Documents, which contains file.txt. If your current working directory were /home you could refer to the same file by the relative path you/Documents/file.txt; and if your current directory was /home/you, the relative path to it would be Documents/file.txt (and if your current directory was /home/you/Music you could say ../Documents/file.txt but let's not take this example any further now).
Windows has a slightly different arrangement, with a number of drives with single-letter identifiers, each with its own root directory; so the root of the C: drive is C:\ and the root of the D: drive is D:\ etc. (and the directory separator is a backslash instead of a slash, although you can use a slash instead pretty much everywhere, which is often a good idea for preserving your sanity).

Your python interpreter location is based off of how you launched it, as well as subsequent actions taken after launching it like use of the os module to navigate your file system. Merely starting the interpreter will place you in the directory of your python installation (not the same on different operating systems). On the other hand, if you start by editing or running a file within a specific directory, your location will be the folder of the file you were editing. If you need to run the interpreter in a certain directory and you are using idle for example, it is easiest to start by creating a python file there one way or another and when you edit it you can start a shell with Run > Python Shell which will already be in that directory. If you are using the command line interpreter, navigate to the folder where you want to run your interpreter before running the python/python3/py command. If you need to navigate manually, you can of course use the following which has already been mentioned:
import os
os.chdir('full_path_to_your_directory')

This has nothing to do with osx in particular, it's more of a concept shared by all unix-based systems, and I believe Windows as well. os.getcwd() is the equivalent of the bash pwd command - it simply returns the full path of the current location in which you are in. In other words:
alex#suse:~> cd /
alex#suse:/> python
Python 2.7.12 (default, Jul 01 2016, 15:34:22) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd()
'/'
It depends from where you started the python shell/script.

Python is usually (except if you are working with virtual environments) accessible from any of your directory. You can check the variables in your path and Python should be available. So the directory you get when you ask Python is the one in which you started Python. Change directory in your shell before starting Python and you will see you will it.

os.getcwd() has nothing to do with OSX in particular. It simply returns the directory/location of the source-file. If my source-file is on my desktop it would return C:\Users\Dave\Desktop\ or let say the source-file is saved on an external storage device it could return something like G:\Programs\. It is the same for both unix-based and Windows systems.

Diference between os.getcwd() and os.path.dirname(__file__)

In a previous project, I used the first version of the following two lines. Now that I found getcwd() I thought this would be the shorter alternative.
print(os.path.dirname(__file__)) # D:/Personal_Software/my_project
print(os.getcwd()) # D:\Personal_Software\my_project
I already read this post, but the thing I'm curious about is the use of "/" vs. "\". I'm using Windows 10 if that's relevant.

__file__ is filename provided to Python, exactly as it was provided.
os.path.dirname(__file__) is usually the directory in which a script is located. (But not always, as you'll see below.)
os.path.dirname(os.path.realpath(__file__)) is the directory in which the script is located.
os.getcwd() is the current directory.
The current directory has nothing to do with where the script is located. It's a per-process path used to resolve relative paths.
/tmp/ikegami/a.py:
#!/usr/bin/python3
import os
print(__file__)
print(os.path.dirname(__file__))
print(os.path.dirname(os.path.realpath(__file__)))
print(os.getcwd())
[~]$ bin/a # A symlink to /tmp/ikegami/a.py
bin/a # The path provided to python3.
bin # NOT the directory in which the script is located.
/tmp/ikegami # The directory in which the script is located.
/home/ikegami # The current directory (~ = /home/ikegami)
[~]$ /tmp/ikegami/a.py
/tmp/ikegami/a.py # The path provided to python3.
/tmp/ikegami # The parent of the path provided to python3.
/tmp/ikegami # The directory in which the script is located.
/home/ikegami # The current directory (~ = /home/ikegami)
[~]$ cd /tmp/ikegami
[/tmp/ikegami]$ ./a.py
./a.py # The path provided to python3.
. # The parent of the path provided to python3.
/tmp/ikegami # The directory in which the script is located.
/tmp/ikegami # The current directory
(Those test were run on a Linux machine because I don't have python installed on this Windows machine. You'll get similar results on Windows.)
the thing I'm curious about is the use of "/" vs. "\".
Windows has always accepted both \ and / as the directory separator.[1][2], and so did DOS before that.[3] These are equivalent. \ is the canonical separator (the one that's preferred for consistency), but / works just as well.
It appears that your program was launched using python3 D:/Personal_Software/my_project/file.py. __file__ contains the exact path provided to Python, and dirname(__file__) simply removes the last bit.
os.getcwd(), on the other hand, receives the current directory from the OS, and the OS probably returns the canonical form of the current directory.
You can even use // for UNC paths (//server/share) and similar (//?/..., //./..., etc).
Contrary to claims that have been made, this is the case for system calls (i.e. the Win32 API), this is the case in Explorer ("My Computer", "File Explorer", "Save As" dialogs, etc), this is the case for the Power Shell (e.g. dir "c:/bin", c:/bin/myprog), and this is case in the Windows console (e.g. c:/bin/myprog).
Caveat: Some specific Windows console commands (e.g. dir) requires paths with / to be in quotes (e.g. dir "c:/") because / signals the start of an option to them.
Well, since directories were introduced in DOS 2.0.

There is a difference, though you wouldn't be able to tell from a single script.
__file__ is the full filename of a loaded module or script, so getting the parent directory of it with os.path.dirname(__file__) gets you the directory that script is in.
Note: on Linux (and similar OSes), such a filename can be a symbolic link to the actual file which may reside somewhere else. You can use os.path.realpath() to resolve through any such links, if needed, although you can typically use the symlink equivalently. On Windows these are less common, but similarly, you can resolve symbolic links through realpath().
os.getcwd() gets you the current working directory. If you start a script from the directory the script is in (which is common), the working directory will be the same as the result from the call from os.path.dirname(__file__).
But if you start the script from another directory (i.e. python d:\some\path\script.py), or if you change the working directory during the script (e.g. with os.chdir()), the current working directory has changed, but the directory part of the script filename has not.
So, it depends on what you need:
Need the directory your script file is in? Use os.path.dirname(__file__)
Need the directory your script is currently running in? use os.getcwd()
You'll see / in some results and \ in others. Sadly, MS Windows uses \ to separate parts of a path (e.g. C:\Program Files\App\), while pretty much all other operating systems use / (e.g. /home/user/script.py)
Python will often convert those automatically, so you can use paths like C:/Program Files/App in Python on Windows as well, but it tends to be a good idea to be safe and use os.path.sep.
Note: if you're on Python 3, you may be better off just using pathlib's Path instead of os.path. It automatically resolves symbolic links (although you can still resolve to the link if you prefer) and has other nice conveniences as well.

Launch Python script from windows command line using system PATH for file location

This may be more of a general Windows question than a Python question, I'm not sure.
I have a folder full of python files called GDAL (a geospatial library). The location of the GDAL library is stored in the windows system PATH, so when I type this in a windows command window to check PATH is configured correctly:
gdal_retile.py
I get notepad opening to show the code, as I would expect, as this is the default application for .py files on this pc.
If however I do this:
python gdal_retile.py
It doesn't work, it says
no such file or directory
Yet if I define the full path:
python "C:\Program Files\GDAL\gdal_retile.py"
It works fine. Can't PATH be used as part of an argument to the Python interpreter?

TL:DR; No.
PATH is used to search for the (executable) file you're trying to run. If a file is not executable (e.g., text files), windows will try to look up which program is registered to handle the file's extension (in your case notepad) an open that one, passing the file's path as argument to it.
Once the correct program has been found, all the following arguments are first checked for eventual %ENVIRONMENT_VARIABLE% placeholders to replace with the actual values, then treated as a list of space-separated strings and passed to the program starting. It's the program's task, then, to figure out what to do with them. PATH has no play in the arguments resolution.
Why is it so?
Arguments to a program can be anything. Imagine you're passing the filename of a file you want to create in the current folder. How can the OS know that the filename you're passing in is not actually an existing file to be searched for in PATH, but a file that will be created by the program? That's why the responsibility to handle arguments is entirely on the program that is being started.

Python doesn't consider system path in its arguments, not even PYTHONPATH...
You can simulate this using where to find the script in the path
where gdal_retile.py > %TEMP%\fullp
then use that to set a variable
set /P C=<%TEMP%\fullpath
then call python with the full path
python "%C%"
(there's no error checking here on the where command, which could return nothing or more than 1 path, so that solution is perfectible but convenient to launch another interpreter than the one associated with the .py extension on Windows)

How does python know to run from the command line?

Hi I noticed that whenever from the command line (using windows 8.1) I type
python file.py
It automatically knows that I meant to write python.exe file.py
How does it do this?
I installed Anaconda, and I understood I have an environment variable pointing to python.exe. But that doesn't explain why I need not type python.exe everytime.

This is not a python feature. The behavior to call executables without file extension is defined by the operation system and the PATH variable. Wikipedia has a good answer to your question
PATH (variable)
...
When a command is entered in a command shell or a system call is made by a program to execute a program, the system first searches the current working directory and then searches the path, examining each directory from left to right, looking for an executable filename that matches the command name given.
...
The file name after the executable could be everything. So if you want you can call python demo.txt. If the file content is readable for python it would also be executed.

Python is searched for using the path. Find the PATH in the list of environment variables.
If you do not write python and simply double-clicks the file, the registry is searched.
You can see what and associate files with program using ftype and assoc from the command line. See e.g. http://www.fileformat.info/tip/microsoft/assocftype.htm
If you omit the extension the registry is also searched. This cmd shell searches the environment variable PATHEXT and the registry for finding python.exe. After this, it is using the registry to find the location.
When you register your python distribution from within Spyder, these changes are made.
You can reveal this information by using ftype and assoc like I wrote in the beginning.