This is probably a silly question.
I am experimenting with python doctest, and I try to run this example
ending with
if __name__ == "__main__":
import doctest
doctest.testfile("example.txt")
I have put "example.txt" in the same folder as the source file containing the example code, but I get the following error:
Traceback (most recent call last):
File "test_av_funktioner.py", line 61, in <module>
doctest.testfile("example.txt")
File "C:\Python26\lib\doctest.py", line 1947, in testfile
text, filename = _load_testfile(filename, package, module_relative)
File "C:\Python26\lib\doctest.py", line 219, in _load_testfile
return open(filename).read(), filename
IOError: [Errno 2] No such file or directory: 'example.txt'
Can I somehow tell/set where the doctest module is searching for the specified file?
Doctest searches relative to the calling module's directory by default (but you can override this).
Quoting the docs for doctest.testfile:
Optional argument module_relative specifies how the filename should be interpreted:
If module_relative is True (the default), then filename specifies an OS-independent module-relative path. By default, this path is relative to the calling module’s directory; but if the package argument is specified, then it is relative to that package. To ensure OS-independence, filename should use / characters to separate path segments, and may not be an absolute path (i.e., it may not begin with /).
If module_relative is False, then filename specifies an OS-specific path. The path may be absolute or relative; relative paths are resolved with respect to the current working directory.
Related
Python can't find where my .txt file is, how can I find the correct path to it or where do I need to put the file?
I tried the following but got an error:
with open("C:\Users\raynaud\Downloads\notlar.txt","r",encoding="utf-8") as file:
with open("dosya.txt","r",encoding= "utf-8") as file:
FileNotFoundError: [Errno 2] No such file or directory: 'dosya.txt'
If you are not using absolute path, you must place the file in the same directory as the script is. On absolute paths you should not use \ like "C:\Users\UserName\", because \ is the escape character in Python (and many more languages). You must use it like this: "C:\\Users\\UserName\\"
Have you checked your current directory? It might be pointing to somewhere unexpected. Try:
import os
os.getcwd()
I am trying to read excel file with pandas.
df=pd.read_excel('abcd (xyz-9) Interim Report 01-03-18.xlsx')
which gives me file not found error. If I remove brackets and rename file to 'abcd Interim Report 01-03-18.xlsx', then it works fine.
I tried renaming with shutil but it gives me the same error
shutil.copyfile('abcd (xyz-9) Interim Report 01-03-18.xlsx','test.xlsx')
I tried
1. pd.read_excel('abcd ^(xyz-9) Interim Report 01-03-18.xlsx')
2. pd.read_excel('abcd \\(xyz-9\\) Interim Report 01-03-18.xlsx')
EDIT:
The file seem to work on local drive but not on network drive even if I change the cwd to the file location.
On using glob and os.path.exists:
for i in range(0,1):
for filename in glob.glob(fpath+"\\"+ldir[i]+"\\"+"*Interim*.xlsx"):
print(filename)
print(os.path.exists(filename))
\\Africa-me.xxx.com\Africa-me\xxx\xxx\xxx\xxx\06 xxx\02 xxx, xxx and xxxx xxx\03 xxx\04 xxx\05 xx xx & xx\12 2018 xx\06 xx xxx\\\AAA-61\abcd (xyz-9) Interim Report 01-03-18.xlsx
False
\\Africa-me.xxx.com\Africa-me\xxx\xxx\xxx\xxx\06 xxx\02 xxx, xxx and xxxx xxx\03 xxx\04 xxx\05 xx xx & xx\12 2018 xx\06 xx xxx\\\AAA-61\abcd Interim Report 01-03-18.xlsx
True
On using glob and os.stat:
import ctypes
for i in range(0,1):
for filename in glob.glob(fpath+"\\"+ldir[i]+"\\"+"*Interim*.xlsx"):
print(filename)
try:
print(os.stat(filename))
except OSError as e:
ntstatus = ctypes.windll.ntdll.RtlGetLastNtStatus()
print('winerror:', e.winerror)
print('ntstatus:', hex(ntstatus & (2**32-1)))
\\Africa-me.xxx.com\Africa-me\xxx\xxx\xxx\xxx\06 xxx\02 xxx, xxx and xxxx xxx\03 xxx\04 xxx\05 xx xx & xx\12 2018 xx\06 xx xxx\\\AAA-61\abcd (xyz-9) Interim Report 01-03-18.xlsx
winerror: 3
ntstatus: 0x80000006
\\Africa-me.xxx.com\Africa-me\xxx\xxx\xxx\xxx\06 xxx\02 xxx, xxx and xxxx xxx\03 xxx\04 xxx\05 xx xx & xx\12 2018 xx\06 xx xxx\\\AAA-61\abcd Interim Report 01-03-18.xlsx
os.stat_result(st_mode=33206, st_ino=15624813576354602, st_dev=3657573641, st_nlink=1, st_uid=0, st_gid=0, st_size=726670, st_atime=1563172745, st_mtime=1523347973, st_ctime=1563170560)
The os.stat test shows that accessing the path with brackets fails with ERROR_PATH_NOT_FOUND (3), which is either from a missing path component or a path that's too long. We know it's not a problem with finding the final path component, since in this case we expect the error to be ERROR_FILE_NOT_FOUND (2). We know it's not a problem with reserved characters, since in this case we expect the error to be ERROR_INVALID_NAME (123). Also, in both of these cases, the Windows API would have to make an NT system call, but we see that the last NT status is STATUS_NO_MORE_FILES (0x80000006), which is from the os.listdir call in glob.glob. Thus the problem is likely that the path is too long.
For brevity or privacy, the question seems to shorten some path component names to "xxx". Probably if expanded out to the true names, we'd see that the path with "(xyz-9)" is at least 260 characters, which is one more than the maximum allowed length for a DOS path, MAX_PATH - 1 (259) characters.
We can access a long path by converting it to an extended path, which is a Unicode string that begins with the "\\?\" device-path prefix, or "\\?\UNC\" for a UNC path. First, since bytes paths in Windows are limited to MAX_PATH characters, we have to decode a bytes path to Unicode. Next we have to normalize and qualify the path via os.path.abspath because extended paths bypass normalization when accessed. Here's an extpath function to convert a DOS path to an extended path:
import os
try:
from os import fsdecode
except ImportError: # Probably Python 2.x
import sys
def fsdecode(filename):
if isinstance(filename, type(u'')):
return filename
elif isinstance(filename, bytes):
return filename.decode(sys.getfilesystemencoding(), 'strict')
raise TypeError('expected string, not {}'.format(
type(filename).__name__))
def extpath(path):
path = os.path.abspath(fsdecode(path))
if not path.startswith(u'\\\\?\\'):
if path.startswith(u'\\\\.\\'):
path = u'\\\\?\\' + path[4:]
elif path.startswith(u'\\\\'):
path = u'\\\\?\\UNC\\' + path[2:]
else:
path = u'\\\\?\\' + path
return path
Background
At the core of Windows NT (i.e. all Windows versions since XP) is the NTOS operating system, which uses the NT kernel. This is similar to the way 16-bit Windows was layered over DOS in the 1980s and early 1990s. But NTOS is more tightly coupled to Windows than DOS was, and it's a more capable OS (e.g. support for symmetric multiprocessing, preemptive multithreading, virtual memory, asynchronous I/O, secured objects, and multiple users with simultaneous logons and sessions).
In some ways Windows still maintains its MS-DOS roots. In particular, for disk device and filesystem paths, the Windows API uses DOS paths instead of NT object paths. This includes DOS drives "A:" through "Z:" and UNC paths such as "\\server\share\path". The Windows API normalizes DOS paths to replace forward slash with backslash; resolve relative paths (i.e. paths with no root directory or no drive) using the process working directory or per-drive working directory; resolve "." and ".." components; and trim trailing spaces and dots from the final component.
When accessing a DOS path, Windows converts it to a device path that begins with one of the WINAPI device prefixes, "\\.\" and "\\?\" (e.g. "C:\Windows" -> "\\?\C:\Windows"). For drive-letter paths and relative paths (but not UNC paths), it reserves a small set of DOS device names in the final path component (e.g. "C:\Temp\con" -> "\\.\con", and "nul" -> "\\.\nul"). For UNC paths, it uses the "UNC" device mountpoint (e.g. \\server\share\path -> "\\?\UNC\server\share\path"). Before a call passes to the NT domain, the WINAPI device prefix ("\\.\" or "\\?\") gets replaced by the NTAPI device prefix ("\??\").
These device-namespace prefixes are shorthand for the caller's local mountpoint directory in NT's object namespace (i.e. "\Sessions\0\DosDevices\<Caller's Logon Session ID>"). The local mountpoint directory implicitly shadows the global mountpoint directory (i.e. "\Global??"). For example, "C:\Temp" becomes "\??\C:\Temp", which evaluates as "\Global??\C:\Temp" if global "C:" isn't locally shadowed. The global mountpoint can be referenced explicitly via the "Global" object link (e.g. "\\?\Global\C:\Temp"), which should always be available.
Classically, DOS path normalization uses string buffers with space for no more than MAX_PATH (260) characters. In Windows 10, this legacy limit is lifted (in most cases) if long paths are enabled for the system and the application manifest declares that it's long-path aware. The
"python[w].exe" executable in Python 3.6+ has this manifest setting.
If long DOS paths aren't enabled, most file API functions still support long paths. We just have to use a WINAPI device path that begins with the "\\?\" prefix, which is called an extended path. This is similar to a regular device path that begins with the "\\.\" prefix, except an extended path does not get normalized when accessed. The downside is that we have to implement our own path normalization. In Python this is implemented by os.path.abspath. We still have to manually rewrite UNC paths, but that's a simple matter of replacing a leading "\\" with "\\?\UNC\".
Note that the working directory does not support device paths, extended or not. The system has undefined behavior if we set a device path as the working directory. So don't use them with Python's os.chdir or the cwd parameter of subprocess.Popen. This means we can't get around the limit on using long relative paths by setting an extended path as the working directory. In Windows 10, if long DOS paths are enabled for the process, the working directory does support long paths, but it still only supports regular DOS paths (UNC and drive-letter paths), not device paths.
I always try to use python's pathlib module whenever I deal with files. There seems to be an error with the initial a in the filename
import pandas as pd
import pathlib
path_to_file = pathlib.Path("g:\Python\abcd (xyz-9) Interim Report 01-03-18.xlsx")
df = pd.read_exces(path_to_file)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pandas' has no attribute 'read_exces'
df = pd.read_excel(path_to_file)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python37\lib\site-packages\pandas\util\_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Python37\lib\site-packages\pandas\util\_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Python37\lib\site-packages\pandas\io\excel.py", line 350, in read_excel
io = ExcelFile(io, engine=engine)
File "C:\Program Files\Python37\lib\site-packages\pandas\io\excel.py", line 653, in __init__
self._reader = self._engines[engine](self._io)
File "C:\Program Files\Python37\lib\site-packages\pandas\io\excel.py", line 424, in __init__
self.book = xlrd.open_workbook(filepath_or_buffer)
File "C:\Program Files\Python37\lib\site-packages\xlrd\__init__.py", line 111, in open_workbook
with open(filename, "rb") as f:
OSError: [Errno 22] Invalid argument: 'g:\\Python\x07bcd (xyz-9) Interim Report 01-03-18.xlsx'
It seems that because of the preceding backslash, the initial a was at some point interpreted as the control symbol \x07 U+0007 : ALERT [BEL].
that's why it is necessary to use the raw string when defining the path, as Dawid suggested
path_to_file = pathlib.Path(r"g:\Python\abcd (xyz-9) Interim Report 01-03-18.xlsx")
quick question about Python on windows.
I have a script that compiles a program (using an install rule), and then moves the build products to a remote destination over the network.
However, I keep getting WindowsError 5 Access Denied.
All files are created from the script context, I have ownership and full control on all of them. The copy to the remote destination succeeds, but the failure is during the deletion process.
If I try to delete or rename the file manually within windows, I get no errors. Just the shutil.move fails.
I'm thinking maybe the API is trying to delete the file when the network operation is not yet complete?
Any input is very appreciated.
try:
shutil.move(directory, destination)
except OSError:
print "Failed to move %s to %s." %(directory, destination)
raise
...
Traceback (most recent call last):
File "C:\WIP\BuildMachine\build_machine.py", line 176, in <module>
main()
File "C:\WIP\BuildMachine.hg\BuilderInstance.py", line 496, in deployVersion
shutil.move(directory, destination)
File "C:\Python27\lib\shutil.py", line 300, in move
rmtree(src)
File "C:\Python27\lib\shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "C:\Python27\lib\shutil.py", line 250, in rmtree
os.remove(fullname)
WindowsError: [Error 5] Access is denied: '3_54_7__B_1\\Application_Release_Note.doc'
the problem with shutil.move on windows is that it doesn't handle the case where:
the source & destination aren't on the same drive AND
some files in the source directory are write-protected.
If both conditions are met, shutil.move cannot perform a os.rename, it has to:
copy the files (which isn't an issue)
delete the source files (which is an issue because of a limitation of shutil)
To fix that, I made myself a copy of the shutil module (under a different name) and added that line (for you it'll be right before line 250):
os.chmod(fullname,0o777) # <-- add that line
os.remove(fullname) # some versions have "unlink" instead
The rmtree function has the same issue on Windows.
On Linux this doesn't happen because file delete permissions aren't handled at the file level but at the directory level. On windows, it doesn't work that way. Adding os.chmod does the trick (even if it's a hack), and os.remove succeeds (unless the file is open in Word or whatever)
Note that shutil authors encourage you to copy & improve the functions. Also a note from the documentation of shutil.move:
A lot more could be done here... A look at a mv.c shows a lot of
the issues this implementation glosses over.
If you don't want to modify shutil, you can run a recursive chmod on the source files to make sure that shutil.move will work, for instance like this:
for root, dirs, files in os.walk(path):
for f in dirs+files:
os.chmod(os.path.join(root, f), 0o777)
You could also use shutil.copytree then a modified version of shutil.rmtree (since you know that source & dest aren't on the same filesystem)
I'm having trouble with os.walk() in python 2.7.8 on Windows.
When I supply it with a 'normal' path such as "D:\Test\master" it works as expected. However when I supply it with a UNC path such as "\\?\D:\Test\master" it will report the root directory as expected but it will not drill down into the sub directories, nor will it raise an exception.
My research: I read on the help page that os.walk() accepts a function argument to handle errors. By default this argument is None so no error is reported.
I passed a simple function to print the error and received the following for every directory.
def WalkError(Error):
raise Exception(Error)
Stack trace:
Traceback (most recent call last):
File "Compare.py", line 988, in StartServer
for root, dirs, files in os.walk(ROOT_DIR,True,WalkError):
File "C:\Program Files (x86)\Python2.7.8\lib\os.py", line 296, in walk
for x in walk(new_path, topdown, onerror, followlinks):
File "C:\Program Files (x86)\Python2.7.8\lib\os.py", line 281, in walk
onerror(err)
File "Compare.py", line 62, in WalkError
raise Exception(Error)
Exception: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\D:\\Test\\master\\localization/*.*'
Answer from the original author (originally posted as an edit to the question):
Instant update: In the process of inspecting \lib\os.py, I discovered the error stems from os.listdir(). I searched for the above error message in relation to os.listdir() and found this solution which worked for me.
It looks like if you're going to use UNC style paths with os. modules they need to Unixised (have their \ converted to /). \\\\?\\D:\\Test\\master\\ becomes //?/D:/Test/master/ (note: you no longer need to escape the \ which is handy).
This runs counter to the UNC 'spec' so be aware if you're working with other modules which respect Microsoft's UNC implementation.
(Sorry for the self-solution, I was going to close the tab but felt there was knowledge here which couldn't be found elsewhere.)
I am encountering an odd behaviour from the file() builtin. I am using the unittest-xml-reporting Python package to generate results for my unit tests. Here are the lines that open a file for writing, a file which (obviously does not exist):
report_file = file('%s%sTEST-%s.xml' % \
(test_runner.output, os.sep, suite), 'w')
(code is taken from the package's Github page)
However, I am given the following error:
...
File "/home/[...]/django-cms/.tox/pytest/local/lib/python2.7/site-packages/xmlrunner/__init__.py", line 240, in generate_reports
(test_runner.output, os.sep, suite), 'w')
IOError: [Errno 2] No such file or directory: './TEST-cms.tests.page.NoAdminPageTests.xml'
I found this weird because, as the Python docs state, if the w mode is used, the file should be created if it doesn't exist. Why is this happening and how can I fix this?
from man 2 read
ENOENT O_CREAT is not set and the named file does not exist. Or, a
directory component in pathname does not exist or is a dangling
symbolic link.
take your pick :)
in human terms:
your current working directory, ./ is removed by the time this command is ran,
./TEST-cms.tests.page.NoAdminPageTests.xml exists but is a symlink pointing to nowhere
"w" in your open/file call is somehow messed up, e.g. if you redefined file builtin
file will create a file, but not a directory. You have to create it first, as seen here
It seems like the file which needed to be created was attempted to be created in a directory that has already been deleted (since the path was given as . and most probably the test directory has been deleted by that point).
I managed to fix this by supplying an absolute path to test_runner.output and the result files are successfully created now.