I am currently working on some code that will look through multiple directories from an .ini file then print and copy them to a new directory. I ran into a problem where the for loop that prints the files only executes once when it is supposed to execute 5 times. How can i fix it so the for loop works every time it is called?
Code:
def copyFiles(path):
rootPath = path
print(rootPath)
pattern = "*.wav"
search = ""
#searches the directories for the specified file type then prints the name
for root, dirs, files in os.walk(rootPath):
for filename in fnmatch.filter(files, pattern):
print(filename)
def main():
#opens the file containing all the directories
in_file = open('wheretolook.ini', "r")
#create the new directory all the files will be moved to
createDirectory()
#takes the path names one at a time and then passes them to copyFiles
for pathName in in_file:
copyFiles(pathName)
Output i get from running my code
The output should have the 0 through 4 files under every diretory.
Thank you for the help!
The pathName you get when iterating over the file has a newline character at the end for each line but the last. This is why you get the blank lines in your output after each path is printed.
You need to call strip on your paths to remove the newlines:
for pathName in in_file:
copyFiles(pathname.strip())
You could be more restrictive and use rstrip('\n'), but I suspect getting rid of all leading and trailing whitespace is better anyway.
Related
In Python, its the vagueness that I struggle with.
Lets start with what I know. I know that I want to search a specific directory for a file. And I want to search that file for a specific line that contains a specific string, and return only that line.
Which brings me to what I don't know. I have a vague description of the specific filename:
some_file_{variable}_{somedatestamp}_{sometimestamp}.log
So I know the file will start with some_file followed by a known variable and ending in .log. I don't know the date stamp or the time stamp. And to top it off, the file might not still exist at the time of searching, so I need to cover for that eventuality.
To better describe the problem, I have the line of the BASH script that accomplishes this:
ls -1tr /dir/some_file_${VARIABLE}_*.log | tail -2 | xargs -I % grep "SEARCH STRING" %
So basically, I want to recreate that line of code from the BASH script in Python, and throw a message in the case that the search returns no files.
Some variant of this will work. This will work for an arbitrary folder structure and will search all subfolders.... results will hold path to directory, filename, and txt, line number (1-based).
Some key pieces:
os.walk is beautiful for searching directory trees. The top= can be relative or absolute
use a context manager with ... as ... to open files as it closes them automatically
python iterates over text-based files in line-by line format
Code
from os import walk, path
magic_text = 'lucky charms'
results = []
for dirpath, _, filenames in walk(top='.'):
for f in filenames:
# check the name. can use string methods or indexing...
if f.startswith('some_name') and f[-4:] == '.log':
# read the lines and look for magic text
with open(path.join(dirpath, f), 'r') as src:
# if you iterate over a file, it returns line-by-line
for idx, line in enumerate(src):
# strings support the "in" operator...
if magic_text in line:
results.append((dirpath, f, idx+1, line))
for item in results:
print(f'on path {item[0]} in file {item[1]} on line {item[2]} found: {item[3]}')
In a trivial folder tree, I placed the magic words in one file and got this result:
on path ./subfolder/subsub in file some_name_331a.log on line 2 found: lucky charms are delicious
See if this works for you:
from glob import glob
import os
log_dir = 'C:\\Apps\\MyApp\\logs\\'
log_variable = input("Enter variable:")
filename = "some_file"+log_variable
# Option 1
searched_files1 = glob(log_dir+filename+'*.log')
print(f'Total files found = {len(searched_files1)}')
print(searched_files1)
# Option 2
searched_files2 = []
for object in os.listdir(log_dir):
if (os.path.isfile(os.path.join(log_dir,object)) and object.startswith(filename) and object.endswith('.log')):
searched_files2.append(object)
print(f'Total files found = {len(searched_files2)}')
print(searched_files2)
I want to add a character at the end of each line in all the files in a folder, so I've written some code in order to iterate through each file and add the desired change, however the output files have different filenames than the originals, below is the code that I've put together
import os
output = '/home/test/Playground/Python/filemodification/output/'
def modification():
with open(files, 'r') as istr:
with open(str(output) + str(files), 'w') as ostr:
for line in istr:
line = line.rstrip('\n') + 'S'
print(line, file=ostr)
directory = '/home/test/Playground/Python/filemodification/input'
for files in os.scandir(directory):
#print(files.path)
print(files)
#print(output)
#print(type(files))
modification()
Once I run the code I get the following filename
<DirEntry 'input.txt'>
and this is the original filename
input.txt
I know the issue is probably related with this
with open(str(output) + str(files), 'w') as ostr:
but I haven't found a way to perform this task differently
If someone could point me in the right direction or provide a code example that can acommplish this task it would be greatly appreciated
Thanks
os.scandir returns os.DirEntry objects. You can get their filename by accessing their .name attribute, or their full path through .path.
E.g.:
for entry in os.scandir(directory):
print(entry.path)
I am trying to find all log files in my C:\ and then in these log file find a string. If the string is found the output should be the abs path of the log file where the string is found. below is what I have done till now.
import os
rootdir=('C:\\')
for folder,dirs,file in os.walk(rootdir):
for files in file:
if files.endswith('.log'):
fullpath=open(os.path.join(folder,files),'r')
for line in fullpath.read():
if "saurabh" in line:
print(os.path.join(folder,files))
Your code is broken at:
for line in fullpath.read():
The statement fullpath.read() will return the entire file as one string, and when you iterate over it, you will be iterating a character at a time. You will never find the string 'saurabh' in a single character.
A file is its own iterator for lines, so just replace this statement with:
for line in fullpath:
Also, for cleanliness, you might want to close the file when you're done, either explicitly or by using a with statement.
Finally, you may want to break when you find a file, rather than printing the same file out multiple times (if there are multiple occurrences of your string):
import os
rootdir=('C:\\')
for folder, dirs, files in os.walk(rootdir):
for file in files:
if file.endswith('.log'):
fullpath = os.path.join(folder, file)
with open(fullpath, 'r') as f:
for line in f:
if "saurabh" in line:
print(fullpath)
break
I am trying to write a script that will iterate through a specified directory, and write to any .txt files with a new string.
Edited after reading Lev Levitsky's explanation
import os
x = raw_input("Enter the directory path here: ")
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
try:
with open(os.paths.join(dirpaths, filename 'a')) as f:
f.write("newline")
except:
print "Directory empty or unable to open file"
return x
rootdir(x)
The script executes, however I get my "Directory empty or unable to open file" exception.
Thanks in advance for any input.
If this is the whole script, then your function is never called, so no wonder nothing happens to the files. You need to actually call the function with the user-provided path:
rootdir(x)
Other issues I see with your code:
The function will erase the text file's content and replace it with "newline". That is because you open the file in write mode. Consider using append mode ('a') instead.
There is no os.dirpaths. You need os.path.join(dirpaths, filename). Also, 'w' is an argument to join, but it should be an argument to open. So in fact the files will be opened in read mode and with incorrect names, resulting in an error.
Finally, because of the return statement inside the loop body, the function will return after processing just one file, without touching the rest.
Very new to Python and programming in general so apologies if I am missing anything straightforward.
I am trying to iterate through a directory and open the included .txt files and modify them with new content.
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
f=open(filename, 'r')
lines=f.read()
f.close()
for line in lines:
f=open(filename, 'w')
newline='rewritten content here'
f.write(newline)
f.close()
return x
rootdir("/Users/russellculver/documents/testfolder")`
Is giving me: IOError: [Errno 2] No such file or directory: 'TestText1.rtf'
EDIT: I should clarify there IS a file named 'TestText1.rtf' in the folder specified in the function argument. It is the first one of three text files.
When I try moving where the file is closed / opened as seen below:
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
f=open(filename, 'r+')
lines=f.read()
for line in lines:
newline='rewritten content here'
f.write(newline)
f.close()
return x
rootdir("/Users/russellculver/documents/testfolder")
It gives me: ValueError: I/O operation on closed file
Thanks for any thoughts in advance.
#mescalinum Okay so I've made amendments to what I've got based on everyones assistance (thanks!), but it is still failing to enter the text "newline" in any of the .txt files in the specified folder.
import os
x = raw_input("Enter the directory here: ")
def rootdir(x):
for dirpaths, dirnames, files in os.walk(x):
for filename in files:
try:
with open(os.dirpaths.join(filename, 'w')) as f:
f.write("newline")
return x
except:
print "There are no files in the directory or the files cannot be opened!"
return x
From https://docs.python.org/2/library/os.html#os.walk:
os.walk(top, topdown=True, onerror=None, followlinks=False)
Generate the file names in a directory tree by walking the tree either top-down or bottom-up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).
dirpath is a string, the path to the directory. dirnames is a list of the names of the subdirectories in dirpath (excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).
Also, f.close() should be outside for line in lines, otherwise you call it multiple times, and the second time you call it, f is already closed, and it will give that I/O error.
You should avoid explicitly open()ing and close()ing files, like:
f=open(filename, 'w')
f.write(newline)
f.close()
and instead use context managers (i.e. the with statement):
with open(filename, 'w'):
f.write(newline)
which does exactly the same thing, but implicitly closes the file when the body of with is finished.
Here is the code that does as you asked:
import os
def rootdir(x):
for paths, dirs, files in os.walk(x):
for filename in files:
try:
f=open(os.path.join(dirpath, name), 'w')
f.write('new content here')
f.close()
except Exception, e:
print "Could not open " + filename
rootdir("/Users/xrisk/Desktop")
However, I have a feeling you don’t quite understand what’s happening here (no offence). First have a look at the documentation of os.walk provided by #mescalinum . The third tuple element files will contain only the file name. You need to combine it with paths to get a full path to the file.
Also, you don’t need to read the file first to write to it. On the other hand, if you want to append to the file, you should use the mode 'a' when opening the file
In general, when reading/writing a file, you only close it after finishing all the read/writes. Otherwise you will get an exception.
Thanks #mescalinum