Im trying to get a local directory from argv and iterate through the folder and print the contents of each file within. However i am getting a [Errno] 13 saying permission denied. Ive tried researching the problem but have come up empty handed.
#!/usr/bin/python
import os
import sys
path = open(sys.argv[1],'r') #'inputs/' path to working input dir
file_list = os.listdir(path) #create list of filenames in path dir
for fn in file_list:
file = open(path+'/'+fn) #open each file in dir for manipulation
for line in file:
print(line)
os.listdir(), as its name implies, returns a list of all occupants of the given directory, including both files and directories (and, if you're on Unix/Linux, other stuff like symlinks and devices and whatnot). You are then blindly trying to open() each item in the list and print() its contents. Unfortunately, open() only works on file-like objects, and specifically does not work on directories, hence Errno 13, Permission Denied.
An alternative is to use os.scandir(), which works a little bit differently. Instead of returning a flat list that you can read immediately, os.scandir() returns a generator which essentially gives you objects as you ask for them, instead of giving them all to you at once. In fact, the following code adapted from the docs is a good starting place for what you need:
for entry in os.scandir(path):
if entry.is_file():
print(entry.name)
os.scandir() is returning DirEntry objects. Simply use os.path.join() to create a full pathname out of the path argument you pass to os.listdir() in your original code, and entry.name from the code above, and then, using the with context manager, open() the file and display its contents:
for entry in os.scandir(path):
if entry.is_file():
with open(os.path.join(path, entry), "r") as f:
for line in f:
print(line)
One of the advantages of using with is that you don't have to remember to close the file handle that is assigned when you use something like this:
f = open("myfile.txt, "r")
# do stuff with f
...
f.close()
Otherwise, you have a dangling file handle that could potentially cause problems, depending on how many there are and what you've done with them. It's just a good practice to close() what you open(). With with, you don't have to worry about it - the file handle is closed as soon as you exit the block.
Related
I need to be able to create a temporary file with a specified file name and write data to it, then zip said file with filename up along with other files:
fd, path = tempfile.mkstemp(".bin", "filename", "~/path/to/working/directory/")
try:
with os.fdopen(fd, "wb") as tmp:
tmp.write(data)
with ZipFile("zip.zip", "w") as zip:
zip.write("filename")
zip.writestr("file2", file2_str)
zip.writestr("file3", file3_str)
# ...
finally:
os.remove(path)
I think I must be misunderstanding how mkstemp works, I get the error at the first line of code here:
FileNotFoundError: [Errno 2] No such file or directory: '~/path/to/working/directory/filenameq5st7dey.bin'
It looks like a bunch of garbage gets added to the file name before the suffix is put on the file. I've tried this without a suffix and I still get garbage at the end of the file name.
Aside from the garbage in the file name, why do I get a file not found error instead of having a temporary file created in my directory with that name (plus garbage)?
You supplied this argument:
"~/path/to/working/directory/"
Perfectly natural, it makes sense why you would supply it. But it is wrong. If you ls . you likely will not find a ~ directory.
What you were hoping for was expansion to ${HOME}, as the Bourne shell does. In python we must call this function:
os.path.expanduser("~/path/to/working/directory/")
Print the result it returns and you'll see why it's essential.
Some folks prefer to have pathlib do the work for them:
from pathlib import Path
Path("~/path/to/working/directory/").expanduser()
Try:
import os, shutil
wd = os.path.abspath(os.path.curdir)
newfile = os.path.join(wd, 'testfile')
print str(newfile)
with open(newfile, 'w') as f: f.write('Hello bugs')
shutil.move(newfile, os.path.join(wd, 'testfile:.txt')) # note the :
Now check the directory - newfile is deleted and no other file is created - Process finished with exit code 0.
If however you issue:
shutil.move(newfile, os.path.join(wd, 'testfile:')) # note no extension
it blows with:
Traceback (most recent call last):
File "C:/Users/MrD/.PyCharm40/config/scratches/scratch_3", line 9, in <module>
shutil.move(newfile, os.path.join(wd, 'testfile:'))
File "C:\_\Python27\lib\shutil.py", line 302, in move
copy2(src, real_dst)
File "C:\_\Python27\lib\shutil.py", line 130, in copy2
copyfile(src, dst)
File "C:\_\Python27\lib\shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 22] invalid mode ('wb') or filename: 'C:\\Users\\MrD\\.PyCharm40\\config\\scratches\\testfile:'
as it should.
Is it a bug ?
Context: I was testing the behavior of my code when illegal filenames were given (: is illegal in windows filenames) when to my amazement my program deleted the original file (bad!) and created a zero size file with the attributes of the original (yes in my case the file was created, just empty) and filename the filename given up to the : - soo a filename like textfile:.jpg gave me a zero byte textfile. It took a lot of debugging - here is the little critter inside the Python27\lib\shutil.py copyfile() (the line that blows above and did not blow):
I don't know why in my case the file was created though while when running the script no.
This isn't a bug in Python's shutil or os modules, it's just a weirdness in Windows. Peter Wood's link in the comments discusses "Advanced Data Streams" -- a Windows filesystem mechanism that attaches a hidden file containing metadata to a regular, visible file. A key word there is attached; The hidden file is deleted if the file it is attached to is deleted.
It appears that a colon is used to separate the path of the regular file from the hidden file. For example, if in the command line you write:
> notepad foo
Then close notepad, and write
> notepad foo.txt:bar
Notepad will open the hidden file. Go ahead and write something in it, save, and close. Typing > dir and the command line will only show foo.txt, not foo.txt:bar.txt. But sure enough, if you write
> notepad foo.txt:bar.txt
the file you just edited will appear, and your changes will be intact.
So what is happening with your Python code? The documentation for shutil.move says:
src is copied (using shutil.copy2()) to dst and then removed.
So when you move testfile to testfile:.txt, Python first copies testfile to the hidden testfile:.txt. But then it removes testfile, and by doing so removes the hidden testfile:.txt. Therefore it appears to you that the original file has been deleted, and no new file has been created.
The following snippet of code might make this clearer (I've saved it as demo.py, and I'm running it in the same, other-wise empty directory):
import os, shutil
with open('test', 'w') as f:
f.write('Hello bugs')
shutil.copy2('test', 'test:foo.txt')
with open('test:foo.txt') as f:
print(f.read())
print 'test: exists? ', os.path.exists('test')
print 'test:foo.txt exists? ', os.path.exists('test:foo.txt')
print os.listdir('.')
print('removing...')
os.remove('test')
print 'test: exists? ', os.path.exists('test')
print 'test:foo.txt exists? ', os.path.exists('test:foo.txt')
print os.listdir('.')
This prints:
Hello bugs
test exists? True
test:foo.txt exists? True
['demo.py', 'test']
removing...
test: exists? False
test:foo.txt exists? False
['demo.py']
This shows that we can create a normal file, write to it, and copy that normal file to its hidden stream, open, and read it just fine, and the result is as expected. Then we see that os.path.exists shows that both test and it's hidden attachment test:foo.txt exist, even though os.listdir only shows test. Then we delete test and we see that test:foo.txt no longer exists as well.
Lastly, you can't create a hidden data stream without a name, therefore test: is an invalid path. Python correctly throws an exception in this case.
So the Python code is actually functioning as it should under Windows -- "Alternate Data Streams" are just such a little-known "feature" that this behavior is surprising.
I am trying to write a script that will iterate through a specified directory, and write to any .txt files with a new string.
Edited after reading Lev Levitsky's explanation
import os
x = raw_input("Enter the directory path here: ")
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
try:
with open(os.paths.join(dirpaths, filename 'a')) as f:
f.write("newline")
except:
print "Directory empty or unable to open file"
return x
rootdir(x)
The script executes, however I get my "Directory empty or unable to open file" exception.
Thanks in advance for any input.
If this is the whole script, then your function is never called, so no wonder nothing happens to the files. You need to actually call the function with the user-provided path:
rootdir(x)
Other issues I see with your code:
The function will erase the text file's content and replace it with "newline". That is because you open the file in write mode. Consider using append mode ('a') instead.
There is no os.dirpaths. You need os.path.join(dirpaths, filename). Also, 'w' is an argument to join, but it should be an argument to open. So in fact the files will be opened in read mode and with incorrect names, resulting in an error.
Finally, because of the return statement inside the loop body, the function will return after processing just one file, without touching the rest.
Very new to Python and programming in general so apologies if I am missing anything straightforward.
I am trying to iterate through a directory and open the included .txt files and modify them with new content.
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
f=open(filename, 'r')
lines=f.read()
f.close()
for line in lines:
f=open(filename, 'w')
newline='rewritten content here'
f.write(newline)
f.close()
return x
rootdir("/Users/russellculver/documents/testfolder")`
Is giving me: IOError: [Errno 2] No such file or directory: 'TestText1.rtf'
EDIT: I should clarify there IS a file named 'TestText1.rtf' in the folder specified in the function argument. It is the first one of three text files.
When I try moving where the file is closed / opened as seen below:
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
f=open(filename, 'r+')
lines=f.read()
for line in lines:
newline='rewritten content here'
f.write(newline)
f.close()
return x
rootdir("/Users/russellculver/documents/testfolder")
It gives me: ValueError: I/O operation on closed file
Thanks for any thoughts in advance.
#mescalinum Okay so I've made amendments to what I've got based on everyones assistance (thanks!), but it is still failing to enter the text "newline" in any of the .txt files in the specified folder.
import os
x = raw_input("Enter the directory here: ")
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
try:
with open(os.dirpaths.join(filename, 'w')) as f:
f.write("newline")
return x
except:
print "There are no files in the directory or the files cannot be opened!"
return x
From https://docs.python.org/2/library/os.html#os.walk:
os.walk(top, topdown=True, onerror=None, followlinks=False)
Generate the file names in a directory tree by walking the tree either top-down or bottom-up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).
dirpath is a string, the path to the directory. dirnames is a list of the names of the subdirectories in dirpath (excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).
Also, f.close() should be outside for line in lines, otherwise you call it multiple times, and the second time you call it, f is already closed, and it will give that I/O error.
You should avoid explicitly open()ing and close()ing files, like:
f=open(filename, 'w')
f.write(newline)
f.close()
and instead use context managers (i.e. the with statement):
with open(filename, 'w'):
f.write(newline)
which does exactly the same thing, but implicitly closes the file when the body of with is finished.
Here is the code that does as you asked:
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
try:
f=open(os.path.join(dirpath, name), 'w')
f.write('new content here')
f.close()
except Exception, e:
print "Could not open " + filename
rootdir("/Users/xrisk/Desktop")
However, I have a feeling you don’t quite understand what’s happening here (no offence). First have a look at the documentation of os.walk provided by #mescalinum . The third tuple element files will contain only the file name. You need to combine it with paths to get a full path to the file.
Also, you don’t need to read the file first to write to it. On the other hand, if you want to append to the file, you should use the mode 'a' when opening the file
In general, when reading/writing a file, you only close it after finishing all the read/writes. Otherwise you will get an exception.
Thanks #mescalinum
This is the code:
def edit(aFile):
s = ''
filename = getMediaPath() + aFile
inputfile = open(filename, 'r')
read = inputfile.readlines()
inputfile.close()
for lines in read:
lines = lines.lower()
lines = lines.replace("it's", "this is")
lines = lines.capitalize()
s = s + str(lines)
newfile = getMediaPath() + 'happyEdited.txt'
x = open(newfile, 'w')
x.write(s)
x.close()
The error I get is on the "inputfile = " line. It says:
"I/O operation failed.
I tried to read a file, and couldn't. Are you sure that file exists? If it does exist, did you specify the correct directory/folder?"**
I've tried entering aFile as a string with the media path. I've tried setting aFile equal to it's media path but nothing works. When I take the parameter out and replace aFile in the code with the name of the .txt file the code works.
Thank y'all!
A few suggestions:
You could include a checking routine for debugging, e.g.,
import os
print os.path.exists(filename)
print os.path.isfile(filename)
And also, I would recommend to use
with open(filename,'r') as inputfile:
# do your stuff
instead of
inputfile = open(filename, 'r')
# do your stuff
inputfile.close()
Because with makes sure that the file stream will be definitely closed if a problem occurs in the # do your stuff section, otherwise you have to use excepts to ensure it, which is just a little bit more effort. with is just a more convenient way.
And I think what you need to get your case to work could be:
newfile = getMediaPath() + '/happyEdited.txt'
I am just adding kwatford`s comment as answer in here. What you need to change is
filename = os.path.join(getMediaPath(),aFile)
newfile = os.path.join(getMediaPath() , 'happyEdited.txt')
The main problem here is probably that you are using simple strings that represent relative file paths. If you were to provide a full traceback, then I could give you a better answer.
Now, this will give you problems a lot of the times, and so it is best practice to always use absolute paths.
Now, what is an absolute path, you say? Well, its the path that goes all the way from your drive to your actual file destination. For example: C:/Foo/Bar/Cheese/happy.py. A relative file path is a path relative to your current directory. For example you were in your command line and you were # C:/Foo/Bar/Cheese/happy.py, and if there was another file in the same directory, say more.py, then you could reference it as ./more.py, but this can lead to several problems as you are facing right now.
So, what is the solution? Like I said, use absolute paths, now how do you do this? Well you use a module called os.
So, something like this:
import os
file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "name_of_file_in_current_dir")).replace(os.pardir, "/")
Now let me tell you what this means, os.path.abspath gives you an absolute path. os.path.join allows you to join paths in a flexible ways, you can join folders. os.path.dirname gives you the absolute path a specified file, in this case __file__. __file__ is a special variable. Finally, internally an OS can use different ways to separate directories, some use //, some use \, some use \\. Now, this is what works best /, since it works on all systems. We use os.pardir because it will work on all systems, Windows, Linux and Unix, which makes your code portable! :D
Also, a good recommendation would be using the with statement. Like so:
with open(file_path) as file:
This is the same as putting a try/catch block around it, but in once simple line. It also opens and closes the file stream for you.