The problem is the character limit for the path in windows when creating multiple directories using pythons os.makedirs()
I found this post when searching for my problem before posting this:
python win32 filename length workaround
Now the chosen answer suggests the prefix workaround but my question here is, is there a way to ensure functionality in Windows and UNIX?
The other approach I thought of was to create the folders one by one and then create the file so that you never exceed the path length, but I can't figure out the obvious bug in the code.
path = ['folder1/s1/s1/abc.txt',
'folder1/s1/s2/def.txt']
def makedirs(path):
explode = path.split('/')
for i in range(len(explode)-1):
os.mkdir(explode[i])
os.chdir(explode[i])
if i == len(explode) -2:
download_file(explode[i+1])
# something to go back here
os.chdir('../' * (len(explode)-3)) # ??
makedirs(path[0])
Now this works for only the first line because I can't figure out how to get back to the root or reset it. Without the 'reset' the folders are being under each other:
folder1/s1/s1/folder1/s1/s1/abc.txt (or something like that)
I could set the path from root to reset it but then we might run into the same issue of reaching the max length. Any help on how to get this working on both OS would be appreciated!
Please feel free to point out where I'm wrong.
you need to use unc path and unicode filenames, but not all python functions are aware of this, os.mkdir works while os.makedirs not
import os
path = u'\\\\?\\c:\\'
for i in xrange(1000):
path += u'subdir\\'
os.mkdir(path)
but it's better to give also the code to remove them, windows explorer is unable to delete
import os
path = u'\\\\?\\c:\\'
for i in xrange(1000, 0, -1):
try:
os.rmdir(path + (u'subdir\\' * i))
except:
pass
Per this stackoverflow answer: while chdir can go up one directory with os.chdir(".."), the platform-agnostic way is: os.chdir(os.pardir).
Either call this N times in a loop;
or try an unreadable one-liner like this (untested):
os.chdir(os.path.join(*([os.pardir] * NUM_TIMES)))
(Instead of path.split('/'), you could also use the method described here for it to work on all operating systems)
Related
lately I started working with the Os module in python . And I finally arrived to this Os.path method . So here is my question . I ran this method in one of my kivy project just for testing and it actually didn't returned the correct output.The method consisted of finding if any directory exist and return a list of folders in the directory . otherwise print Invalid Path and return -1 . I passed in an existing directory and it returned -1 but the weird path is that when I run similar program out of my kivy project using the same path present in thesame folder as my python file it return the desired output .here is the image with the python file and the directory name image I have tested which returns invalid path.
and here is my code snippet
def get_imgs(self, img_path):
if not os.path.exists(img_path):
print("Invalid Path...")
return -1
else:
all_files = os.listdir(img_path)
imgs = []
for f in all_files:
if (
f.endswith(".png")
or f.endswith(".PNG")
or f.endswith(".jpg")
or f.endswith(".JPG")
or f.endswith(".jpeg")
or f.endswith(".JPEG")
):
imgs.append("/".join([img_path, f]))
return imgs
It's tough to tell without seeing the code with your function call. Whatever argument you're passing must not be a valid path. I use the os module regularly and have slowly learned a lot of useful methods. I always print out paths that I'm reading or where I'm writing before doing it in case anything unexpected happens, I can see that img_path variable, for example. Copy and paste the path in file explorer up to the directory and make sure that's all good.
Some other useful os.path methods you will find useful, based on your code:
os.join(<directory>, <file_name.ext>) is much more intuitive than imgs.append("/".join([img_path, f]))
os.getcwd() gets your working directory (which I print at the start of scripts in dev to quickly address issues before debugging). I typically use full paths to play it safe because Python pathing can cause differences/issues when running from cmd vs. PyCharm
os.path.basename(f) gives you the file, while os.path.dirname(f) gives you the directory.
It seems like a better approach to this is to use pathlib and glob. You can iterate over directories and use wild cards.
Look at these:
iterating over directories: How can I iterate over files in a given directory?
different file types: Python glob multiple filetypes
Then you don't even need to check whether os.path.exists(img_path) because this will read the files directly from your file system. There's also more wild cards in the glob library such as * for anything/any length, ? for any character, [0-9] for any number, found here: https://docs.python.org/3/library/glob.html
I have a program which I designed both for myself and my colleague to use, with all the data being stored in a directories. However, I want to set up the loop so that it work both for me and him. I tried all of these:
file_location = glob.glob('/../*.nc')
file_location = glob('/../*.nc')
But none of them are picking up any files. How can I fix this?
You can get a directory relative to a user's home (called ~ in the function call) using os.path.expanduser(). In your case, the line would be
file_location = glob.glob(os.path.expanduser('~/Dropbox/Argo/Data/*.nc'))
Usually is a good practice not hardcoding paths if you're going to use your paths for other tasks which need well-formed paths (ie: subprocess, writing paths to shell scripts), I'd recommend to manage paths using the os.path module instead, for example:
import os, glob
home_path = os.path.expanduser("~")
dropbox_path = os.path.join(home_path, "Dropbox")
good_paths = glob.glob(os.path.join(dropbox_path,"Argo","Data","*.nc"))
bad_paths = glob.glob(dropbox_path+"/Argo\\Data/*.nc")
print len(good_paths)==len(bad_paths)
print all([os.path.exists(p) for p in good_paths])
print all([os.path.exists(p) for p in bad_paths])
The example shows a comparison between bad and well formed paths. Both of them will work, but good_paths will be more flexible and portable in the long term.
I'm trying to learn how to edit files, but I'm a bit of a python novice, and not all that bright, so when I get a FileNotFoundError I can't figure out how to fix it despite several searches on the interwebz.
import os
old = 'Users\My Name\Pictures\2013\182904_10201130467645938_341581100_n'
new = 'Users\My Name\Pictures\2013\Death_Valley_1'
os.rename(old, new)
'Users\My Name\Pictures\2013\182904_10201130467645938_341581100_n' is a relative path.
Unless you are running your code from the directory that contains the Users dir (which if you are using Windows would most probably be the root C: dir), Python isn't going to find that file.
You also have to make sure to include the file extension if it has any.
There are few ways to solve this, the easiest one will be to use the absolute paths in your code, ie 'C:\Users\My Name\Pictures\2013\182904_10201130467645938_341581100_n.jpg'.
You will also want to use r before the paths, so you want need to escape every \ character.
import os
old = r'C:\Users\My Name\Pictures\2013\182904_10201130467645938_341581100_n.jpg'
new = r'C:\Users\My Name\Pictures\2013\Death_Valley_1.jpg'
os.rename(old, new)
This of course assumes your drive letter is C.
python has recently added the pathlib module (which i like a lot!).
there is just one thing i'm struggling with: is it possible to normalize a path to a file or directory that does not exist? i can do that perfectly well with os.path.normpath. but wouldn't it be absurd to have to use something other than the library that should take care of path related stuff?
the functionality i would like to have is this:
from os.path import normpath
from pathlib import Path
pth = Path('/tmp/some_directory/../i_do_not_exist.txt')
pth = Path(normpath(str(pth)))
# -> /tmp/i_do_not_exist.txt
but without having to resort to os.path and without having to type-cast to str and back to Path. also pth.resolve() does not work for non-existing files.
is there a simple way to do that with just pathlib?
is it possible to normalize a path to a file or directory that does not exist?
Starting from 3.6, it's the default behavior. See https://docs.python.org/3.6/library/pathlib.html#pathlib.Path.resolve
Path.resolve(strict=False)
...
If strict is False, the path is resolved as far as possible and any remainder is appended without checking whether it exists
As of Python 3.5: No, there's not.
PEP 0428 states:
Path resolution
The resolve() method makes a path absolute, resolving
any symlink on the way (like the POSIX realpath() call). It is the
only operation which will remove " .. " path components. On Windows,
this method will also take care to return the canonical path (with the
right casing).
Since resolve() is the only operation to remove the ".." components, and it fails when the file doesn't exist, there won't be a simple means using just pathlib.
Also, the pathlib documentation gives a hint as to why:
Spurious slashes and single dots are collapsed, but double dots ('..')
are not, since this would change the meaning of a path in the face of
symbolic links:
PurePath('foo//bar') produces PurePosixPath('foo/bar')
PurePath('foo/./bar') produces PurePosixPath('foo/bar')
PurePath('foo/../bar') produces PurePosixPath('foo/../bar')
(a naïve approach would make PurePosixPath('foo/../bar') equivalent to PurePosixPath('bar'), which is wrong if foo is a symbolic link to another directory)
All that said, you could create a 0 byte file at the location of your path, and then it'd be possible to resolve the path (thus eliminating the ..). I'm not sure that's any simpler than your normpath approach, though.
If this fits you usecase(e.g. ifle's directory already exists) you might try to resolve path's parent and then re-append file name, e.g.:
from pathlib import Path
p = Path()/'hello.there'
print(p.parent.resolve()/p.name)
Old question, but here is another solution in particular if you want POSIX paths across the board (like nix paths on Windows too).
I found pathlib resolve() to be broken as of Python 3.10, and this method is not currently exposed by PurePosixPath.
What I found worked was to use posixpath.normpath(). Also found PurePosixPath.joinpath() to be broken. I.E. It will not join ".." with "myfile.txt" as expected. It will return just "myfile.txt". But posixpath.join() works perfectly; will return "../myfile.txt".
Note this is in path strings, but easily back to pathlib.Path(my_posix_path) et al for an OOP container.
And easily transpose to Windows platform paths too by just constructing this way, as the module takes care of the platform independence for you.
Might be the solution for others with Python file path woes..
Hey. I've got a project in Python, whose directory layout is the following:
root
|-bin
|-conf
|-[project]
Python files in [project] need to be able to read configuration data from the 'conf' directory, but I cannot guarantee the location of root, plus it may be used on both Linux, Mac and Windows machines so I am trying to relatively address the conf directory from the root directory.
At the minute it's working with a dirty hack (from root/bin, particular python filename is 8 chars long):
path = os.path.abspath(__file__)[:-8]
os.chdir(path)
os.chdir("..")
[projectclass].config('config/scans.json') #for example
But this is particularly horrid and is giving me nightmares. Is there a better way to accomplish what I'm trying to achieve that doesn't feel so dirty? I feel like I'm missing something very obvious. Thanks in advance.
Instead of:
path = os.path.abspath(__file__)[:-8]
use:
path = os.path.dirname(os.path.abspath(__file__))
See the docs here.