Develop Reference

Need to read every file of directory for particular word around 172 directories,we have done for single directory

Here is the below code we have developed for single directory of files
from os import listdir
with open("/user/results.txt", "w") as f:
for filename in listdir("/user/stream"):
with open('/user/stream/' + filename) as currentFile:
text = currentFile.read()
if 'checksum' in text:
f.write('current word in ' + filename[:-4] + '\n')
else:
f.write('NOT ' + filename[:-4] + '\n')
I want loop for all directories
Thanks in advance

If you're using UNIX you can use grep:
grep "checksum" -R /user/stream
The -R flag allows for a recursive search inside the directory, following the symbolic links if there are any.

My suggestion is to use glob.
The glob module allows you to work with files. In the Unix universe, a directory is / should be a file so it should be able to help you with your task.
More over, you don't have to install anything, glob comes with python.
Note: For the following code, you will need python3.5 or greater
This should help you out.
import os
import glob
for path in glob.glob('/ai2/data/prod/admin/inf/**', recursive=True):
# At some point, `path` will be `/ai2/data/prod/admin/inf/inf_<$APP>_pvt/error`
if not os.path.isdir(path):
# Check the `id` of the file
# Do things with the file
# If there are files inside `/ai2/data/prod/admin/inf/inf_<$APP>_pvt/error` you will be able to access them here
What glob.glob does is, it Return a possibly-empty list of path names that match pathname. In this case, it will match every file (including directories) in /user/stream/. If these files are not directories, you can do whatever you want with them.
I hope this will help you!
Clarification
Regarding your 3 point comment attempting to clarify the question, especially this part we need to put appi dynamically in that path then we need to read all files inside that directory
No, you do not need to do this. Please read my answer carefully and please read glob documentation.
In this case, it will match every file (including directories) in /user/stream/
If you replace /user/stream/ with /ai2/data/prod/admin/inf/, you will have access to every file in /ai2/data/prod/admin/inf/. Assuming your app ids are 1, 2, 3, this means, you will have access to the following files.
/ai2/data/prod/admin/inf/inf_1_pvt/error
/ai2/data/prod/admin/inf/inf_2_pvt/error
/ai2/data/prod/admin/inf/inf_3_pvt/error
You do not have to specify the id, because you will be iterating over all files. If you do need the id, you can just extract it from the path.
If everything looks like this, /ai2/data/prod/admin/inf/inf_<$APP>_pvt/error, you can get the id by removing /ai2/data/prod/admin/inf/ and taking everything until you encounter _.

Renaming files (while counting the number)

I'm currently trying to write a python script to rename a bunch of files. The file is named like this: [Name][Number]-[Number]. To give a specific example: milk-00-00. The next file is milk-00-01, then 02, 03 until X. After that milk-01-00 starts with the same pattern.
What I need to do is to switch 'milk' into a number and replace the '-XX-XX' by '-01', '02', ...
I hope you guys get the idea. The current state of my code is pretty poor, it was hard enough to get it this far though. It looks like this and with this I'm at least able to replace something. I'll also manage to get rid of the 'milk' with the help of google. However, if there is an easier way, I'd really appreciate a push in the right direction!
import os
import sys
path = 'C:/Users/milk/Desktop/asd'
i=00
for filename in os.listdir(path):
if filename.endswith('.tiff'):
newname = filename.replace('00', 'i')
os.rename(filename,newname)
i=i+1

You can use the format function
temp = (' ').join(filename.split('.')[:-1])
os.rename(filename, '10{}-{}.tiff'.format(temp.split('-')[-2],temp.split('-')[-1]))
Since filename has the .tiff extension this program first creates a version of filename without the extension - temp - and then creates new names from that.

os.rename(filename, '1000-%02d.tiff' % i)
i += 1

Python, Opening files in loop (dicom)

I am currently reading in 200 dicom images manually using the code:
ds1 = dicom.read_file('1.dcm')
so far, this has worked but I am trying to make my code shorter and easier to use by creating a loop to read in the files using this code:
for filename in os.listdir(dirName):
dicom_file = os.path.join("/",dirName,filename)
exists = os.path.isfile(dicom_file)
print filename
ds = dicom.read_file(dicom_file)
This code is not currently working and I am receiving the error:
"raise InvalidDicomError("File is missing 'DICM' marker. "
dicom.errors.InvalidDicomError: File is missing 'DICM' marker. Use
force=True to force reading
Could anyone advice me on where I am going wrong please?

I think the line:
dicom_file = os.path.join("/",dirName,filename)
might be an issue? It will join all three to form a path rooted at '/'. For example:
os.path.join("/","directory","file")
will give you "/directory/file" (an absolute path), while:
os.path.join("directory","file")
will give you "directory/file" (a relative path)
If you know that all the files you want are "*.dcm"
you can try the glob module:
import glob
files_with_dcm = glob.glob("*.dcm")
This will also work with full paths:
import glob
files_with_dcm = glob.glob("/full/path/to/files/*.dcm")
But also, os.listdir(dirName) will include everything in the directory including other directories, dot files, and whatnot
Your exists = os.path.isfile(dicom_file) line will filter out all the non files if you use an "if exists:" before reading.
I would recommend the glob approach, if you know the pattern, otherwise:
if exists:
try:
ds = dicom.read_file(dicom_file)
except InvalidDicomError as exc:
print "something wrong with", dicom_file
If you do a try/except, the if exists: is a bit redundant, but doesn't hurt...

Try adding:
dicom_file = os.path.join("/",dirName,filename)
if not dicom_file.endswith('.dcm'):
continue

Editing Paths in Python

I'm trying to create a program that duplicates itself to another location and creates a batch file on the desktop. I can make it duplicate itself and I can create the batch file but I need some help with the paths.
I can find the path that my program is currently in. Both the direct path and the path to the directory. My problem lies in the fact that I want to place the file in (let's just say for simplicity) 'C:\Users\Me\Documents'. How would I edit the path? I want to be able to place this on a generic windows computer so I can't hard code the path in because each user will be different. This goes the same for placing the batch file and setting it for the right directory to run the python script in documents.
I have tried both
import os
print os.path.dirname(os.path.abspath(__file__))
and
import os
print os.path.abspath(__file__)
but am clueless as to how to edit the path. When I try googling for it and searching this wonderful site, all I get is stuff about configuring the Python path on windows and other stuff that I can't quite understand at my current level of Python.
Now I turn to you, can you help? Any input would be appreciated, if you could explain how it worked that would be even better!
<>
Due to some questions about my code (and a specific one to post it) Here it is
from sys import argv # Imports module
import os
script, create = argv # Gets script name and desired amount of copies
data = open(script) # Creates a variable to store the script
indata = copy.read() # Creates the data to be copied from the script
batData = """
echo off
%s
""" % # This is not finished, creating that batch file
createT = int(create) + 1
for i in range(1, createT): # Runs a set amount of times
copyName = "%s.py" % str(i) # Creates the name for the file
copy = open(copyName, 'w+') # Opens/creates the file for editing
copy.write(indata) # Writies the indata to the file opened
copy.close # Closes that file
batName = "%s.bat" % str(i)
bat = open(batName, 'w+')
It is not finished but hopefully you get the gist. The argv at the beginning is so I can change the amount of copies made, that will be deleted later as I evolve the code but for now I like it there.
I have currently tried the following to find the path:
import os
print os.path.abspath(__file__)
print os.path.dirname(os.path.abspath(__file__))
print os.path.dirname(__file__)
test = os.path.dirname(__file__)
a, b, c, d, e, f, g, h, i = test.split("\\")
print c
What I want to happen (or think I want to happen) is for the path to be found, then split into pieces (either each directory or break off everything after the username). Then I want to append the document folder tag to the end. For the batch instead of the document tag it will be for the desktop.
among a few others that people have posted. I hope this helps!

Your code snippet returns a string. Just take that string and edit to make the path you want. I'm on a mac so I can't test with an actual windows directory but I'll try to make it look Windows-ish. For instance, lets say this code:
directory_path = os.path.dirname(os.path.abspath(__file__))
print(directory_path)
gives you:
C:\Users\username\AppData
You can use the split function to break the path into pieces (docs).
stuff = path_string.split('\')
print(stuff)
Code output:
['C:', 'Users', 'username', 'AppData']
You can use the pieces create the path you want and then use it to write the file. So, if you want the username folder just loop until you find it. Some example code is below (just an example to get you started - read up on Python if you need help understanding the code).
username = ""
for i in range(0, len(stuff)):
if stuff[i] == "Users":
username = stuff[i + 1]
Not sure if that answers your question but hope it helps.

Am I correct that you are trying to figure out how to make a file path to some location on the user's directory without knowing who the user is going to be that is executing the program?
You may be able to take advantage of environment variables for this. For instance, I can get a file path to the C://Users/username/ directory of whoever is executing the code with:
my_path = os.path.join("C:\\", "Users", os.getenv("USERNAME"))
Where os.getenv("USERNAME") returns the value of the USERNAME environment variable (should be the name of the user that is currently logged on).
You can list all of the available environment variables and their values in Python with:
for key, val in os.environ.items():
print("{} \t {}\n".format(key, val)) # or however you want to prettify it
You may get lucky and find an environment variable that gives you most of the path to begin with.
In general, os.path.join(), os.path.relpath(), and os.path.abspath() combined with some environment variables might be able to help you. Check out the os documentation and os.path documentation.

Many times when modifying a path, I am looking to add/remove folders. Here is my simple method for adding a path, e.g. if I want to move the path of an object into a folder added_path='/train/.
Since my paths are usually uniform, I check the last split characters in the first file location. Usually, my experience is that windows have \\ at the end while Mac and Linux have `/', which makes this work across operating systems. (note: if the paths are not uniform, you obviously place the if-else in the for-loop.)
if '\\' in data[0].img_file:
split_char = '\\'
else:
split_char = '/'
for img in data:
img_path = img.img_file.split(split_char)
label_path = img.label_file.split(split_char)
img.img_file = '/'.join(img_path[:-1]) + added_path + img_path[-1]
img.label_file = '/'.join(label_path[:-1]) + added_path + label_path[-1]
So, this for loop uses all the folders up until the file name, where we insert the extra folder, and then, at last, add the file name.
Example input path: 'data/combined/18.png'
Example output path: 'data/combined/train/18.png'

Moving specific files in subdirectories into a directory - python

Im rather new to python but I have been attemping to learn the basics.
Anyways I have several files that once i have extracted from their zip files (painfully slow process btw) produce several hundred subdirectories with 2-3 files in each. Now what I want to do is extract all those files ending with 'dem.tif' and place them in a seperate file (move not copy).
I may have attempted to jump into the deep end here but the code i've written runs without error so it must not be finding the files (that do exist!) as it gives me the else statement. Here is the code i've created
import os
src = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Extracted' # input
dst = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Analyses' # desired location
def move():
for (dirpath, dirs, files) in os.walk(src):
if files.endswith('dem.tif'):
shutil.move(os.path.join(src,files),dst)
print ('Moving ', + files, + ' to ', + dst)
else:
print 'No Such File Exists'

First, welcome to the community, and python! You might want to change your user name, especially if you frequent here. :)
I suggest the following (stolen from Mr. Beazley):
# genfind.py
#
# A function that generates files that match a given filename pattern
import os
import shutil
import fnmatch
def gen_find(filepat,top):
for path, dirlist, filelist in os.walk(top):
for name in fnmatch.filter(filelist,filepat):
yield os.path.join(path,name)
# Example use
if __name__ == '__main__':
src = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Extracted' # input
dst = 'O:\DATA\ASTER GDEM\Original\North America\UTM Zone 14\USA\Analyses' # desired location
filesToMove = gen_find("*dem.tif",src)
for name in filesToMove:
shutil.move(name, dst)

I think you've mixed up the way you should be using os.walk().
for dirpath, dirs, files in os.walk(src):
print dirpath
print dirs
print files
for filename in files:
if filename.endswith('dem.tif'):
shutil.move(...)
else:
...

Update: the questioner has clarified below that he / she is actually calling the move function, which was the first point in my answer.
There are a few other things to consider:
You've got the order of elements returned in each tuple from os.walk wrong, I'm afraid - check the documentation for that function.
Assuming you've fixed that, also bear in mind that you need to iterate over files, and you need to os.join each of those to root, rather than src
The above would be obvious, hopefully, if you print out the values returned by os.walk and comment out the rest of the code in that loop.
With code that does potentially destructive operations like moving files, I would always first try some code that just prints out the parameters to shutil.move until you're sure that it's right.

Any particular reason you need to do it in Python? Would a simple shell command not be simpler? If you're on a Unix-like system, or have access to Cygwin on Windows:
find src_dir -name "*dem.tif" -exec mv {} dst_dir