I can't save this output, maybe someone have the solution. I'm listing a directory and some singles files. But when I save the output just catch the directory files, and Not the singles files.
My code:
import os
tosave = open('/tmp/list','ab')
thesource = ["/etc/ssh","/var/log/syslog","/etc/hosts"]
for f in thesource:
print f
for top, dirs, files in os.walk(f):
for nm in files:
print os.path.join(top, nm)
try:
tosave.write(top+nm+'\n')
finally:
tosave.close
I saw in the console all files and directory, but in the saved list, just ssh files. Why didn't save syslog and hosts too?
Thank you !!
In case you missed the () at tosave.close while pasting: (otherwise check harsh's answer)
The finally is wrong here. The code in finally will be executed after the try block, so after the first execution of tosave.write(top+nm+'\n') the file will be closed because of tosave.close().
Possibly you intended to use except:
# snip
try:
tosave.write(top+nm+'\n')
except:
tosave.close()
Edit: To answer your comment, you want the last line to be the same as the print statement:
tosave.write(os.path.join(top, nm) + '\n')
You could try adding tosave.flush() at the end. It does cause problems sometimes. Sometimes, a flush call is required to empty the contents of the buffer into the file.
Check if this works for you
import os
tosave = open('/tmp/list','ab')
thesource = ["/etc/ssh","/var/log/syslog","/etc/hosts"]
for f in thesource:
if os.path.isdir(f):
for top, dirs, files in os.walk(f):
for nm in files:
try:
tosave.write(top+nm+'\n')
if os.path.isfile(f):
tosave.write(f+'\n')
to.close()
With all your help I found a solution and it's working. I share it.
tosave = open('/tmp/list','ab')
thesource = ["/etc/ssh","/var/log/syslog","/etc/hosts"]
for f in thesource:
if os.path.isfile(f):
print f
tosave.write(f+'\n')
else:
for top, dirs, files in os.walk(f):
for nm in files:
print os.path.join(top, nm)
tosave.write(top+nm+'\n')
Thank you all for your help !!!
Maybe this is because you're opening the file in append mode (the 'a') and then looking at its beginning? Look at its end - you might see your new files listed there.
In append mode, each time the script runs it appends its output to the file. Usually what one wants is just the write mode ('w' instead of 'a'), which overwrites the file each time.
Put the close() at the end of the script.
Otherwise it will close the file after the first step in the loop, making the file unwritable.
Related
I want to iterate through the filenames in a particular folder. I then wish to choose the first filename that satisfies a criteria (file name ends with '.txt')
Should I use a For loop and break it when I see the first file ending with .txt?
Or should I use a While loop?
The While loop does not seem to work. It keeps on continuing. It keeps on printing the filename as per below code.
Following is the code I am using:
import os
import pdb
asciipath = 'C:\\Users\\rmore\\Desktop\\Datalab'
x = 0
file = os.listdir(asciipath)
while file[x].endswith('.txt'):
print(file[x])
x = x+1
It is possible to do this with a while loop, but it will overly complicate the code.
I would use a for loop here. I would also rename file to files just to make what is happening a little more clear.
Edit:
As pointed out, an else clause for the loop would make for a
files = os.listdir(asciipath)
for file in files:
if file.endswith('.txt'):
print(file)
break
else:
print('No txt file found')
The break is key for stopping the loop after you find the first file that ends with .txt
Also note that the else statement is on the for loop and NOT inside of the loop. The else will only be triggered if the break statement is NOT triggered.
A pythonic way would be to use next on a generator:
next((f for f in file if f.endswith('.txt')), 'file not found')
Or you can loop over the files and return as soon as the condition is matched:
def find_first_txt_file(files)
for file in files:
if file.endswith('.txt'):
return file
return 'file not found'
Make it easier for yourself and use pathlib and glob.
from pathlib import Path
p = Path(asciipath)
print(next(p.glob('*.txt')))
I want to open a file to write to.
with open('test.txt','a') as textfile:
...
It works like this.
Now I want this file to be opened/created from a directory called args.runkeyword.
with open(os.path.join(args.runkeyword, 'test.txt'),'a') as textfile:
t says it can't find test/test.txt (supposing runkeyword is test).
I also tried by appending with os.getcwd() but it still can't find or create the file.
Any ideas?
os.getcwd() is irrelevant on your work actually. Use os.listdir() to see every folder in a directory. If anything named by test before it may be problem.
A recursive function like this may usefull for you;
import os
def tara(directory):
start = os.getcwd()
files = []
os.chdir(directory)
for oge in os.listdir(os.curdir):
if not os.path.isdir(oge):
files.append(oge)
else:
files.extend(tara(oge))
os.chdir(start)
return files
file = open('test.txt', 'a+')
You should have 'a+' not 'a', the + allows you to append.
so I'm a rookie at programming and I'm trying to make a program in python that basically opens a text file with a bunch of columns and writes the data to 3 different text files based on a string in the row. As my program stands right now, I have it change the directory to a specific output folder using os.chdir so it can open my text file but what I want is it to do something like this:
Imagine a folder set up like this :
Source Folder contains N number of folders. Each of those folders contains N number of output folders. Each output folder contains 1 Results.txt.
The idea is to have the program start at the source folder, look into Folder 1, look for output 1, open the .txt file then do it's thing. Once it's done, it should go back to folder 1 and open output 2 and do it's thing again. Then it should go back to Folder 1 and if it can't find any more output folders, it needs to go to Folder A and then enter Folder 2 and repeat the process until there are no more folders. Honestly not sure where to really start with this, the best I could do is make a small program that prints all my .txt files but I'm not sure how to open them at all. Hope my question makes sense and thanks for the help.
If all you need is to process each file in a directory recursively:
import os
def process_dir(dir):
for subdir, dirs, files in os.walk(dir):
for file in files:
file_path = os.path.join(subdir, file)
print file_path
# process file here
This will process each file in the root dir recursively. If you're looking for conditional iteration you might need to make the loop a little smarter.
Read the base folder path and stored into variable and move to sub folder and process the text file using chdir and base path change the directory and read the sub folder once again.
dirlist = os.listdir(os.getcwd())
dirlist = filter(lambda x: os.path.isdir(x), filelist)
for dirname in dirlist:
print os.path.join(os.getcwd(),dirname,'Results.txt')
first, i think you could format your question for better reading.
Concerning your question, here's a naïve implementation example :
import os
where = "J:/tmp/"
what = "Results.txt"
def processpath(where, name):
for elem in os.listdir(where):
elempath = os.path.join(where,elem)
if (elem == name):
# Do something with your file
f = open(elempath, "w") # example
f.write("modified2") # example
elif(os.path.isdir(elempath)):
processpath(elempath, name)
processpath(where, what)
I would do this without chdir. The most straight forward solution to me is to use os.listdir and filter the results. Then os.path.join to construct complete relative paths instead of chdir. I suspect this would be less prone to bugs such as winding up in an unexpected current working directory where all your relative paths are then wrong.
nfolders = [d for d in os.listdir(".") if re.match("^Folder [0-9]+$", d)]
for f1 in nfolders:
noutputs = [d for d in os.listdir(f1) if re.match("^Output [0-9]+$", d)]
for f2 in noutputs:
resultsFilename = os.path.join(f1, f2, "results.txt")
#do whatever with resultsFilename
I have a thread that I would like to loop through all of the .txt files in a certain directory (C:\files\) All I need is help reading anything from that directory that is a .txt file. I cant seem to figure it out.. Here is my current code that looks for specific files:
def file_Read(self):
if self.is_connected:
threading.Timer(5, self.file_Read).start();
print '~~~~~~~~~~~~Thread test~~~~~~~~~~~~~~~'
try:
with open('C:\\files\\test.txt', 'r') as content_file:
content = content_file.read()
Num,Message = content.strip().split(';')
print Num
print Message
print Num
self.send_message(Num + , Message)
content_file.close()
os.remove("test.txt")
#except
except Exception as e:
print 'no file ', e
time.sleep(10)
does anyone have a simple fix for this? I have found a lot of threads using methods like:
directory = os.path.join("c:\\files\\","path")
threading.Timer(5, self.file_Read).start();
print '~~~~~~~~~~~~Thread test~~~~~~~~~~~~~~~'
try:
for root,dirs,files in os.walk(directory):
for file in files:
if file.endswith(".txt"):
content_file = open(file, 'r')
but this doesn't seem to be working.
Any help would be appreciated. Thanks in advance...
I would do something like this, by using glob:
import glob
import os
txtpattern = os.path.join("c:\\files\\", "*.txt")
files = glob.glob(txtpattern)
for f in file:
print "Filename : %s" % f
# Do what you want with the file
This method works only if you want to read .txt in your directory and not in its potential subdirectories.
Take a look at the manual entries for os.walk - if you need to recurse sub-directories or glob.glob if you are only interested in a single directory.
The main problem is that the first thing you do in the function that you want to start in the threads is that you create a new thread with that function.
Since every thread will start a new thread, you should get an increasing number of threads starting new threads, which also seems to be what happens.
If you want to do some work on all the files, and you want to do that in parallel on a multi-core machine (which is what I'm guessing) take a look at the multiprocessing module, and the Queue class. But get the file handling code working first before you try to parallelize it.
Im rather new to python but I have been attemping to learn the basics.
Anyways I have several files that once i have extracted from their zip files (painfully slow process btw) produce several hundred subdirectories with 2-3 files in each. Now what I want to do is extract all those files ending with 'dem.tif' and place them in a seperate file (move not copy).
I may have attempted to jump into the deep end here but the code i've written runs without error so it must not be finding the files (that do exist!) as it gives me the else statement. Here is the code i've created
import os
src = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Extracted' # input
dst = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Analyses' # desired location
def move():
for (dirpath, dirs, files) in os.walk(src):
if files.endswith('dem.tif'):
shutil.move(os.path.join(src,files),dst)
print ('Moving ', + files, + ' to ', + dst)
else:
print 'No Such File Exists'
First, welcome to the community, and python! You might want to change your user name, especially if you frequent here. :)
I suggest the following (stolen from Mr. Beazley):
# genfind.py
#
# A function that generates files that match a given filename pattern
import os
import shutil
import fnmatch
def gen_find(filepat,top):
for path, dirlist, filelist in os.walk(top):
for name in fnmatch.filter(filelist,filepat):
yield os.path.join(path,name)
# Example use
if __name__ == '__main__':
src = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Extracted' # input
dst = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Analyses' # desired location
filesToMove = gen_find("*dem.tif",src)
for name in filesToMove:
shutil.move(name, dst)
I think you've mixed up the way you should be using os.walk().
for dirpath, dirs, files in os.walk(src):
print dirpath
print dirs
print files
for filename in files:
if filename.endswith('dem.tif'):
shutil.move(...)
else:
...
Update: the questioner has clarified below that he / she is actually calling the move function, which was the first point in my answer.
There are a few other things to consider:
You've got the order of elements returned in each tuple from os.walk wrong, I'm afraid - check the documentation for that function.
Assuming you've fixed that, also bear in mind that you need to iterate over files, and you need to os.join each of those to root, rather than src
The above would be obvious, hopefully, if you print out the values returned by os.walk and comment out the rest of the code in that loop.
With code that does potentially destructive operations like moving files, I would always first try some code that just prints out the parameters to shutil.move until you're sure that it's right.
Any particular reason you need to do it in Python? Would a simple shell command not be simpler? If you're on a Unix-like system, or have access to Cygwin on Windows:
find src_dir -name "*dem.tif" -exec mv {} dst_dir