os.walk finds files that don't exist

os.walk finds files that don't exist - python

I was creating a program that checked for files in my Downloads folder and moved them in some subfolders depending on their extention.
I used os.walk("path") # path is the path to my downloads folder
and printed the files it found, just to see if it worked.
The problem is that the program finds the files in the folder, but it also finds some files that aren't there, most of them end with .zpa, and it also finds a desktop.ini file.
Is this normal or is there something wrong?

os.walk will find all files including those marked with hidden and system attributes. Those files are not displayed by default in the Windows Explorer or the command line dir utility.
If you want to skip hidden and system files in Python you'll have to check for the file attributes:
import os
import stat
for path,dirs,files in os.walk('Downloads'):
for file in files:
filename = os.path.join(path,file)
info = os.stat(filename)
if not info.st_file_attributes & (stat.FILE_ATTRIBUTE_SYSTEM | stat.FILE_ATTRIBUTE_HIDDEN):
print(filename)

See os.walk finds all files inside a directory. Those files which are hidden in Windows file explorer are also found out by this. Hidden files can include
Files hidden by you from within file explorer
Files hidden by OS itself, which are OS specific configuration files which we don't have to worry much about.
Files hidden by any application software intentionally
Probably there can be files from the above category which you got to see in the output.
As far as .zpa files are concerned, I don't know about those, but found a link which could help you.

Related

Errno 2 No such file or directory: 'user.txt' When there are files in the dictionary [duplicate]

I am trying to open the file recentlyUpdated.yaml from my Python script. But when I try using:
open('recentlyUpdated.yaml')
I get an error that says:
IOError: [Errno 2] No such file or directory: 'recentlyUpdated.yaml'
Why? How can I fix the problem?

Ensure the file exists (and has the right file extension): use os.listdir() to see the list of files in the current working directory.
Ensure you're in the expected directory using os.getcwd().
(If you launch your code from an IDE, you may be in a different directory.)
You can then either:
Call os.chdir(dir) where dir is the directory containing the file. Then, open the file using just its name, e.g. open("file.txt").
Specify an absolute path to the file in your open call.
Use a raw string (r"") if your path uses backslashes, like
so: dir = r'C:\Python32'
If you don't use raw string, you have to escape every backslash: 'C:\\User\\Bob\\...'
Forward-slashes also work on Windows 'C:/Python32' and do not need to be escaped.
Let me clarify how Python finds files:
An absolute path is a path that starts with your computer's root directory, for example C:\Python\scripts if you're on Windows.
A relative path is a path that does not start with your computer's root directory, and is instead relative to something called the working directory. You can view Python's current working directory by calling os.getcwd().
If you try to do open('sortedLists.yaml'), Python will see that you are passing it a relative path, so it will search for the file inside the current working directory.
Calling os.chdir() will change the current working directory.
Example: Let's say file.txt is found in C:\Folder.
To open it, you can do:
os.chdir(r'C:\Folder')
open('file.txt') # relative path, looks inside the current working directory
or
open(r'C:\Folder\file.txt') # absolute path

Most likely, the problem is that you're using a relative file path to open the file, but the current working directory isn't set to what you think it is.
It's a common misconception that relative paths are relative to the location of the python script, but this is untrue. Relative file paths are always relative to the current working directory, and the current working directory doesn't have to be the location of your python script.
You have three options:
Use an absolute path to open the file:
file = open(r'C:\path\to\your\file.yaml')
Generate the path to the file relative to your python script:
from pathlib import Path
script_location = Path(__file__).absolute().parent
file_location = script_location / 'file.yaml'
file = file_location.open()
(See also: How do I get the path and name of the file that is currently executing?)
Change the current working directory before opening the file:
import os
os.chdir(r'C:\path\to\your\file')
file = open('file.yaml')
Other common mistakes that could cause a "file not found" error include:
Accidentally using escape sequences in a file path:
path = 'C:\Users\newton\file.yaml'
# Incorrect! The '\n' in 'Users\newton' is a line break character!
To avoid making this mistake, remember to use raw string literals for file paths:
path = r'C:\Users\newton\file.yaml'
# Correct!
(See also: Windows path in Python)
Forgetting that Windows doesn't display file extensions:
Since Windows doesn't display known file extensions, sometimes when you think your file is named file.yaml, it's actually named file.yaml.yaml. Double-check your file's extension.

The file may be existing but may have a different path. Try writing the absolute path for the file.
Try os.listdir() function to check that atleast python sees the file.
Try it like this:
file1 = open(r'Drive:\Dir\recentlyUpdated.yaml')

Possibly, you closed the 'file1'.
Just use 'w' flag, that create new file:
file1 = open('recentlyUpdated.yaml', 'w')
mode is an optional string that specifies the mode in which the file
is opened. It defaults to 'r' which means open for reading in text
mode. Other common values are 'w' for writing (truncating the file if
it already exists)...
(see also https://docs.python.org/3/library/functions.html?highlight=open#open)

If is VSCode see the workspace. If you are in other workspace this error can rise

Understanding absolute and relative paths
The term path means exactly what it sounds like. It shows the steps that need to be taken, into and out of folders, to find a file. Each step on the path is either a folder name, the special name . (which means the current folder), or the special name .. (which means to go back/out into the parent folder).
The terms absolute and relative also have their usual English meaning. A relative path shows where something is relative to some start point; an absolute path is a location starting from the top.
Paths that start with a path separator, or a drive letter followed by a path separator (like C:/foo) on Windows, are absolute. (On Windows there are also UNC paths, which are necessarily absolute. Most people will never have to worry about these.)
Paths that directly start with a file or folder name, or a drive letter followed directly by the file or folder name (like C:foo) on Windows, are relative.
Understanding the "current working directory"
Relative paths are "relative to" the so-called current working directory (hereafter abbreviated CWD). At the command line, Linux and Mac use a common CWD across all drives. (The entire file system has a common "root", and may include multiple physical storage devices.) Windows is a bit different: it remembers the most recent CWD for each drive, and has separate functionality to switch between drives, restoring those old CWD values.
Each process (this includes terminal/command windows) has its own CWD. When a program is started from the command line, it will get the CWD that the terminal/command process was using. When a program is started from a GUI (by double-clicking a script, or dragging something onto the script, or dragging the script onto a Python executable) or by using an IDE, the CWD might be any number of things depending on the details.
Importantly, the CWD is not necessarily where the script is located.
The script's CWD can be checked using os.getcwd, and modified using os.chdir. Each IDE has its own rules that control the initial CWD; check the documentation for details.
To set the CWD to the folder that contains the current script, determine that path and then set it:
os.chdir(os.path.dirname(os.path.abspath(__file__)))
Verifying the actual file name and path
There are many reasons why the path to a file might not match expectations. For example, sometimes people expect C:/foo.txt on Windows to mean "the file named foo.txt on the desktop". This is wrong. That file is actually - normally - at C:/Users/name/Desktop/foo.txt (replacing name with the current user's username). It could instead be elsewhere, if Windows is configured to put it elsewhere. To find the path to the desktop in a portable way, see How to get Desktop location?.
It's also common to mis-count ..s in a relative path, or inappropriately repeat a folder name in a path. Take special care when constructing a path programmatically. Finally, keep in mind that .. will have no effect while already in a root directory (/ on Linux or Mac, or a drive root on Windows).
Take even more special care when constructing a path based on user input. If the input is not sanitized, bad things could happen (e.g. allowing the user to unzip a file into a folder where it will overwrite something important, or where the user ought not be allowed to write files).
Another common gotcha is that the special ~ shortcut for the current user's home directory does not work in an absolute path specified in a Python program. That part of the path must be explicitly converted to the actual path, using os.path.expanduser. See Why am I forced to os.path.expanduser in python? and os.makedirs doesn't understand "~" in my path.
Keep in mind that os.listdir will give only the file names, not paths. Trying to iterate over a directory listed this way will only work if that directory is the current working directory.
It's also important to make sure of the actual file name. Windows has an option to hide file name extensions in the GUI. If you see foo.txt in a window, it could be that the file's actual name is foo.txt.txt, or something else. You can disable this option in your settings. You can also verify the file name using the command line; dir will tell you the truth about what is in the folder. (The Linux/Mac equivalent is ls, of course; but the problem should not arise there in the first place.)
Backslashes in ordinary strings are escape sequences. This causes problems when trying to a backslash as the path separator on Windows. However, using backslashes for this is not necessary, and generally not advisable. See Windows path in Python.
When trying to create a new file using a file mode like w, the path to the new file still needs to exist - i.e., all the intervening folders. See for example Trying to use open(filename, 'w' ) gives IOError: [Errno 2] No such file or directory if directory doesn't exist. Also keep in mind that the new file name has to be valid. In particular, it will not work to try to insert a date in MM/DD/YYYY format into the file name, because the /s will be treated as path separators.

Check the path that has been mentioned, if it's absolute or relative.
If its something like-->/folder/subfolder/file -->Computer will search for folder in root directory.
If its something like--> ./folder/subfolder/file --> Computer will search for folder in current working directory.

If you are using IDE like VScode, make sure you have opened the IDE from the same directory where you have kept the file you want to access.
For example, if you want to access file.txt which is inside the Document, try opening the IDE from Document by right clicking in the directory and clicking "Open with "

Receiving back a FileNotFoundError and not quite sure why? [duplicate]

I am trying to open the file recentlyUpdated.yaml from my Python script. But when I try using:
open('recentlyUpdated.yaml')
I get an error that says:
IOError: [Errno 2] No such file or directory: 'recentlyUpdated.yaml'
Why? How can I fix the problem?

Ensure the file exists (and has the right file extension): use os.listdir() to see the list of files in the current working directory.
Ensure you're in the expected directory using os.getcwd().
(If you launch your code from an IDE, you may be in a different directory.)
You can then either:
Call os.chdir(dir) where dir is the directory containing the file. Then, open the file using just its name, e.g. open("file.txt").
Specify an absolute path to the file in your open call.
Use a raw string (r"") if your path uses backslashes, like
so: dir = r'C:\Python32'
If you don't use raw string, you have to escape every backslash: 'C:\\User\\Bob\\...'
Forward-slashes also work on Windows 'C:/Python32' and do not need to be escaped.
Let me clarify how Python finds files:
An absolute path is a path that starts with your computer's root directory, for example C:\Python\scripts if you're on Windows.
A relative path is a path that does not start with your computer's root directory, and is instead relative to something called the working directory. You can view Python's current working directory by calling os.getcwd().
If you try to do open('sortedLists.yaml'), Python will see that you are passing it a relative path, so it will search for the file inside the current working directory.
Calling os.chdir() will change the current working directory.
Example: Let's say file.txt is found in C:\Folder.
To open it, you can do:
os.chdir(r'C:\Folder')
open('file.txt') # relative path, looks inside the current working directory
or
open(r'C:\Folder\file.txt') # absolute path

Most likely, the problem is that you're using a relative file path to open the file, but the current working directory isn't set to what you think it is.
It's a common misconception that relative paths are relative to the location of the python script, but this is untrue. Relative file paths are always relative to the current working directory, and the current working directory doesn't have to be the location of your python script.
You have three options:
Use an absolute path to open the file:
file = open(r'C:\path\to\your\file.yaml')
Generate the path to the file relative to your python script:
from pathlib import Path
script_location = Path(__file__).absolute().parent
file_location = script_location / 'file.yaml'
file = file_location.open()
(See also: How do I get the path and name of the file that is currently executing?)
Change the current working directory before opening the file:
import os
os.chdir(r'C:\path\to\your\file')
file = open('file.yaml')
Other common mistakes that could cause a "file not found" error include:
Accidentally using escape sequences in a file path:
path = 'C:\Users\newton\file.yaml'
# Incorrect! The '\n' in 'Users\newton' is a line break character!
To avoid making this mistake, remember to use raw string literals for file paths:
path = r'C:\Users\newton\file.yaml'
# Correct!
(See also: Windows path in Python)
Forgetting that Windows doesn't display file extensions:
Since Windows doesn't display known file extensions, sometimes when you think your file is named file.yaml, it's actually named file.yaml.yaml. Double-check your file's extension.

The file may be existing but may have a different path. Try writing the absolute path for the file.
Try os.listdir() function to check that atleast python sees the file.
Try it like this:
file1 = open(r'Drive:\Dir\recentlyUpdated.yaml')

Possibly, you closed the 'file1'.
Just use 'w' flag, that create new file:
file1 = open('recentlyUpdated.yaml', 'w')
mode is an optional string that specifies the mode in which the file
is opened. It defaults to 'r' which means open for reading in text
mode. Other common values are 'w' for writing (truncating the file if
it already exists)...
(see also https://docs.python.org/3/library/functions.html?highlight=open#open)

If is VSCode see the workspace. If you are in other workspace this error can rise

Check the path that has been mentioned, if it's absolute or relative.
If its something like-->/folder/subfolder/file -->Computer will search for folder in root directory.
If its something like--> ./folder/subfolder/file --> Computer will search for folder in current working directory.

If you are using IDE like VScode, make sure you have opened the IDE from the same directory where you have kept the file you want to access.
For example, if you want to access file.txt which is inside the Document, try opening the IDE from Document by right clicking in the directory and clicking "Open with "

Specify a folder to save screenshots to in Desktopmagic

I'm using a module that saves screen shots into my Python34 folder as .png files.
I want the .png files to be relocated into a separate folder within the Python34 folder.
I want this to be done automatically. I was thinking maybe I could loop through the Python34 folder to find all the .png files and then save these files into a new "PNG" folder within the Python34 folder, but I don't know how I would do this. How should I approach this problem?
Some clarifications:
The module I am using is called "Desktopmagic", ( https://pypi.python.org/pypi/Desktopmagic/13.3.29 )
I am working with Windwows.

As I can see from the code of Lib\site-packages\desktopmagic\screengrab_win32.py:getDCAndBitMap (called from saveScreenToBmp), the module uses the provided file name argument as is.
I.e. if you specify it without a path, it saves into the current directory. So, if you wish to save elsewhere, either change it or specify a path (relative or absolute) in the argument. Btw, saving random files under the python installation dir is a very bad idea.
If you want to prepend a path by default, you'll need to use a wrapper or replace some of the module's machinery.

You can search a directory for a certain set of files using glob, for example
import glob
screenshots = glob.glob('*.png')
You can then copy the files by looping over the screenshots list and using shutil.copyfile

open() gives FileNotFoundError / IOError: '[Errno 2] No such file or directory'

I am trying to open the file recentlyUpdated.yaml from my Python script. But when I try using:
open('recentlyUpdated.yaml')
I get an error that says:
IOError: [Errno 2] No such file or directory: 'recentlyUpdated.yaml'
Why? How can I fix the problem?

Ensure the file exists (and has the right file extension): use os.listdir() to see the list of files in the current working directory.
Ensure you're in the expected directory using os.getcwd().
(If you launch your code from an IDE, you may be in a different directory.)
You can then either:
Call os.chdir(dir) where dir is the directory containing the file. Then, open the file using just its name, e.g. open("file.txt").
Specify an absolute path to the file in your open call.
Use a raw string (r"") if your path uses backslashes, like
so: dir = r'C:\Python32'
If you don't use raw string, you have to escape every backslash: 'C:\\User\\Bob\\...'
Forward-slashes also work on Windows 'C:/Python32' and do not need to be escaped.
Let me clarify how Python finds files:
An absolute path is a path that starts with your computer's root directory, for example C:\Python\scripts if you're on Windows.
A relative path is a path that does not start with your computer's root directory, and is instead relative to something called the working directory. You can view Python's current working directory by calling os.getcwd().
If you try to do open('sortedLists.yaml'), Python will see that you are passing it a relative path, so it will search for the file inside the current working directory.
Calling os.chdir() will change the current working directory.
Example: Let's say file.txt is found in C:\Folder.
To open it, you can do:
os.chdir(r'C:\Folder')
open('file.txt') # relative path, looks inside the current working directory
or
open(r'C:\Folder\file.txt') # absolute path

Most likely, the problem is that you're using a relative file path to open the file, but the current working directory isn't set to what you think it is.
It's a common misconception that relative paths are relative to the location of the python script, but this is untrue. Relative file paths are always relative to the current working directory, and the current working directory doesn't have to be the location of your python script.
You have three options:
Use an absolute path to open the file:
file = open(r'C:\path\to\your\file.yaml')
Generate the path to the file relative to your python script:
from pathlib import Path
script_location = Path(__file__).absolute().parent
file_location = script_location / 'file.yaml'
file = file_location.open()
(See also: How do I get the path and name of the file that is currently executing?)
Change the current working directory before opening the file:
import os
os.chdir(r'C:\path\to\your\file')
file = open('file.yaml')
Other common mistakes that could cause a "file not found" error include:
Accidentally using escape sequences in a file path:
path = 'C:\Users\newton\file.yaml'
# Incorrect! The '\n' in 'Users\newton' is a line break character!
To avoid making this mistake, remember to use raw string literals for file paths:
path = r'C:\Users\newton\file.yaml'
# Correct!
(See also: Windows path in Python)
Forgetting that Windows doesn't display file extensions:
Since Windows doesn't display known file extensions, sometimes when you think your file is named file.yaml, it's actually named file.yaml.yaml. Double-check your file's extension.

The file may be existing but may have a different path. Try writing the absolute path for the file.
Try os.listdir() function to check that atleast python sees the file.
Try it like this:
file1 = open(r'Drive:\Dir\recentlyUpdated.yaml')

Possibly, you closed the 'file1'.
Just use 'w' flag, that create new file:
file1 = open('recentlyUpdated.yaml', 'w')
mode is an optional string that specifies the mode in which the file
is opened. It defaults to 'r' which means open for reading in text
mode. Other common values are 'w' for writing (truncating the file if
it already exists)...
(see also https://docs.python.org/3/library/functions.html?highlight=open#open)

If is VSCode see the workspace. If you are in other workspace this error can rise

Check the path that has been mentioned, if it's absolute or relative.
If its something like-->/folder/subfolder/file -->Computer will search for folder in root directory.
If its something like--> ./folder/subfolder/file --> Computer will search for folder in current working directory.

If you are using IDE like VScode, make sure you have opened the IDE from the same directory where you have kept the file you want to access.
For example, if you want to access file.txt which is inside the Document, try opening the IDE from Document by right clicking in the directory and clicking "Open with "

Python: Move files from Folder and subfolders to another similar directory

I have two source file paths:
C:\Same\Path\To\File\unknown\
C:\Same\Path\To\File\unrecognized\
And one destination path:
C:\Same\Path\To\File\Import
The subfolders for all these paths are the same. They are 10-15 folders that have 3 digit names (233 for example).
If any files are in these subfolders of the source paths I want to CUT them from that directory and send them to the destination directory with the same path as the source. i am new to python so any help would be appreciated.

I think what you want is shutil.move (py3k docs):
Help on function move in module shutil:
move(src, dst)
Recursively move a file or directory to another location. This is
similar to the Unix "mv" command.
If the destination is a directory or a symlink to a directory, the source
is moved inside the directory. The destination path must not already
exist.
If the destination already exists but is not a directory, it may be
overwritten depending on os.rename() semantics.
If the destination is on our current filesystem, then rename() is used.
Otherwise, src is copied to the destination and then removed.
A lot more could be done here... A look at a mv.c shows a lot of
the issues this implementation glosses over.

Look into the shutil module.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.