I have a problem with os.walk(path). Some folders seems that cannot Traversed. I have tried the os.path.exists(path) but i have an exception even thought the directory already exists. From documentation is written that On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
So i have tried os.stat(path) and i get an error The system cannot find the file specified.
Finally i have tried os.listdir(path) and i get a False message. so i have tried to do the following x=os.listdir("C:\\Windows\\System32") but the folder v1 wasn't inside the list x when i have searched with the "v1" in x
the code is the following
import os
path = "C:\\Windows\\System32\\v1"
os.stat(path)
The solutions that i came across are no use to solve my problem and i want to ask is there a possible way to get permissions to execute os.stat() to that specific folder and furthermore to do it via python?
Well, you're trying to access Sys32 which is probably locked under read-only permissions. Run this as well beforehand to try to remove the read-only flag :
subprocess.check_call(["attrib", "-r", path])
Be sure to run your program as an Administrator as you're on Windows. All chmod operations are available, but are severely restricted to setting the R flags to files created by yourself and that's it. In case that fails, I'll refer you to this post, you'll have to use modules.
Related
import os, sys
inputFilename = 'X.txt'
if not os.path.exists(inputFilename):
print('The file %s does not exist. Quitting...' % (inputFilename))
sys.exit()
This code is only meant to run when the txt file X is not found. But for some reason it keeps running these lines, even though the file does exist in the right place. Any ideas what I'm doing wrong?
I've tried moving everything into a separate folder and renaming file, nothing seems to be working.
Make sure that the IDE you're using recognises the folder, otherwise it might not find the file even if it exists in the folder.
Fx if you're using vs code, click on the option "open folder" and add the folder as a working platform. Check how to for whichever IDE you're using.
On another note,
if you're only using files it might be better practice to use isfile() instead of exists()
exists() is for files and directories, whereas isfile() is for just files. For just directories you can use isdir()
Where is the text file located?
Where are you running the python code from?
The answers to 1. and 2. should both be the same. If not, there's your issue.
I think you might be able to pass in an absolute path instead of just 'X.txt' to inputFileName. This way the directory layout won't matter.
There doesn't seem to be anything inherently wrong with your code. I tested out it locally and it works.
An intern of mine is trying to use deeptools' bamCoverage function, and it throws a '/my/dir/data.bam' file does not exist error, despite the file being there. Yes, the file does exist, we can manipulate it using bash commands just fine so there's no real reason for it to throw that error.
According to this thread, it could be an issue with pysam or python. Both are fully up to date. Do you know how I could investigate issues with pysam or python IO further?
All of this is happening on a server that our team uses. Could it be an issue with python's paths for his user session?
For reference, here is the code I'm running in my bash terminal. It's pretty basic:
my.name#server:/mnt/data1/my.name/PROJET_X/DATA/BIGWIG$ bamCoverage -p 8 -b /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam -o /mnt/data1/my.name/PROJET_X/DATA/BIGWIG/G0-G00.Inputs.RPGC.bw --normalizeUsing RPGC --effectiveGenomeSize 2913022398 -bs 10
The file '/mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam' does not exist
my.name#server:/mnt/data1/my.name/PROJET_X/DATA/BIGWIG$ ll /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam
-rwxr-xr-x 1 my.name bioinfo 4171366400 juin 22 10:14 /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam
I have used this line probably 100 times this past year, but now it won't find the file.
I've also tried installing deeptools in my home directory using conda, but that gives the same errors.
EDIT: Apparently, it was just a problem with that one file. bamCoverage will work on other data files. It would be nice if deeptools would tell you that instead of just "file not found"...
It would be helpful to see what code is being run here - please,
provide a minimal reproducible example, including the full output / error, if at all possible.
Right now, I can think of 4 possible solutions to common problems:
As mentioned in the comments, providing the full path might be one solution. You can get the full path by using the pwd bash command in the folder where the file is stored and then add the file name at the end of the string.
Another thing that sometimes helps with relative (local) paths is to start the path with a ".", so the path would be something like this: './my/dir/data.bam'.
Another problem I've encountered (mostly when running things on Windows environments) was that the backslashes had to be used instead: '\my\dir\data.bam'
For the sake of completeness here, I'll mention that especially with beginning programmers, the path is often erroneously provided without commas (to make it a string), but that does not seem to be the case here.
Hope this helps!
Background:
I'm running the ABSApp for sentiment analysis which requires Linux or Mac to run on. Our network filesystems require permission to access our datasets, and I'm trying to figure out how to give that permission via my code so that I can run my preprocessing scripts on our data and store it to the network since I'm not allowed to store it locally. I can access the files via my Linux dual bootup by connecting to the server directly, but this permission doesn't carry across the system when I run my code.
Tried:
I've tried to access the dir with os.walk(dir, topdown=True), and when I step through the debugger I see this message:
top = fspath(top)
dirs = []
nondirs = []
walk_dirs = []
# We may not have read permission for top, in which case we can't
# get a list of the files the directory contains. os.walk
# always suppressed the exception then, rather than blow up for a
# minor reason when (say) a thousand readable directories are still
# left to visit. That logic is copied here.
I don't see anything useful when I jump to the definition for fspath(path) either.
I read the documentation for os.access(), but I already know I don't have permission to the files. It does say this at the bottom, but it doesn't tell me a work-around:
Note
I/O operations may fail even when access() indicates that they would succeed,
particularly for operations on network filesystems
which may have permissions semantics beyond the usual POSIX permission-bit model.
TLDR:
So does anyone have any solutions for accessing and writing to a dir on a local network server that requires permissions? I can do python, java and c++, so I'm open to any solutions that exist! Thanks in advance!!
Solved: Needed to add noperm at the end to mount with write permissions
sudo mount.cifs <domain> /home/<mount location> -o username=<username>,vers=<set accordingly>,noperm
Disclaimer: i have searched genericly (Google) and here for some information on this topic, but most of the answers are either very old, or don't seem to quite make sense to me, so I appologize in advance if this seems simple or uninformed.
Problem: My app accepts command line input which may be either a path or a file and I need to determine several things about it.
Is it a path or file,
Is it relative or absolute
Is it readable and / or writable (need read and write test results)(ignoring the possibility of a race situation)
One caveat, while a
try:
file=open(filename,'w')
except OSError as e:
{miscellaneous error handling code here}
would obviously tell me if the parameter (filename in above example) existed/ was writable etc. I do not understand enough about exception codes to know how to interpret the result of exception. It also wouldn't provide the relative/absolute information.
Assuming that there is no one method to do this then I would need to know three things:
How to determine relative / absolute
Is it pointing to a file or a directory
can the EUID of the program read the location, and same for write.
I am seeking to learn from the information I gather here, and I am new to python and have bit off a significant project. I have mastered all of it except this part. Any help would be appreciated. (Pointers to good help sites welcome!)(except docs.python.org that ones bookmarked already ;-) )
Here are your answers. The documentations specifies that the following works for both windows and linux.
How to determine relative / absolute
os.path.isabs returns True if the path is absolute, False if not.
Is it pointing to a file or a directory
Similarly Use os.path.isdir to find out if the path is directory or not.
YOu can use os.path.isfile(path) to find out if the path is file or not.
can the EUID of the program read the location, and same for write.
You can use os.access(path, mode) to know if operations requiring permissions specified by mode are possible on file specified by path or not.
P.S. This will not work for files being accessed over the network. You can use os.stat This is the right way to get all the information. However it is a more costly call and hence, you should try and get all the info in one call . To interpret the results, you can use the stat module
Seems like os.path would take care of most of these needs with isabs(), isfile(), isdir(), etc. Off the top of my head I can't think of a function that will give a simple True/False for read/write access for a given file, but one way to tackle this might be to use the os module's stat function and have a look at the file's permissions and owner.
I have a Python test suite that creates and deletes many temporary files. Under Windows 7, the shutil.rmtree operations sometimes fail (<1% of the time). The failure is apparently random, not always on the same files, not always in the same way, but it's always on rmtree operations. It seems to be some kind of timing issue. It is also reminiscent of Windows 7's increased vigilance about permissions and administrator rights, but there are no permission issues here (since the code had just created the files), and there are no administrator rights in the mix.
It also looks like a timing issue between two threads or processes, but there is no concurrency here either.
Two examples of (partial) stack traces:
File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
shutil.rmtree(cleandir)
File "c:\python23\lib\shutil.py", line 142, in rmtree
raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 5] Access is denied removing xml_1
File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
shutil.rmtree(cleandir)
File "c:\python23\lib\shutil.py", line 142, in rmtree
raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 3] The system cannot find the path specified removing out
On Windows XP, it never failed. On Windows 7, it fails like this, across a few different Python versions (2.3-2.6, not sure about 3.1).
Anyone seen anything like this and have a solution? The code itself is on bitbucket for the truly industrious.
It's a long shot, but are you running anything that scans directories in the background? I'm thinking antivirus/backup (maybe Windows 7 has something like that built in? I don't know). I have experienced occasional glitches when deleting/moving files from the TSVNCache.exe process that TortoiseSVN starts -- seems it watches directories for changes, then presumably opens them for scanning the files.
We had similar problems with shutil.rmtree on Windows, particularly looking like your first stack trace. We solved it by using an exception handler with rmtree. See this answer for details.
Just a thought, but if the test behavior (creating and deleting lots of temp files) isn't typical of what the app actually does, maybe you could move those test file operations to (c)StringIO, and keep a suite of function tests that exercises your app's actual file creation/deletion behavior.
That way, you can make sure that your app behaves correctly, without introducing extra complexity not related to the app.
My guess is that you should check up on the code that creates the file, and make SURE they are closed explicitly before moving on to delete it. If nothing is obvious there in the code, download a copy of Process Monitor and watch what's happening on the file system there. This tool will give you the exact error code coming from Windows, and should shed some light on the situation.
That "The system cannot find the path specified:" error will appear intermittently if the path is too long for Windows (260 chars). Automated tasks often create folder hierarchies using relative references that produce fully qualified paths longer than 260 characters. Any script that attempts to delete those folders using fully qualified paths will fail.
I built a quick workaround that used relative path references, but don't have generic code solution to offer, just a warning that the excellent answers provided might not help you.
I met same problem with shutil.rmtree command and this issue maybe cause by special file name.(Ex:Other country language:леме / Ö)
Please use the following format to delete the directory which you want:
import shutil
shutil.rmtree(os.path.join("<folder_name>").decode('ascii'))
Enjoy it !