Develop Reference

Verify the format of a filename in Python

Every week I get two files with following pattern.
EMEA_{sample}_Tracker_{year}_KW{week}
E.g.
EMEA_G_Tracker_2019_KW52.xlsx
EMEA_BC_Tracker_2019_KW52.xlsx
Next files would look like these
EMEA_G_Tracker_2020_KW1.xlsx
EMEA_BC_Tracker_2020_KW1.xlsx
Placeholders:
sample = G or BC
year = current year [YYYY]
week = calendar week [0 - ~52]
The only changes are made in the placeholders, everything else will stay the same.
How can I extract these values from the filename and check if the filename has this format?
Right now I only read all files using os.walk():
path_files = "Files/"
files = []
for (_, _, filenames) in walk(path_files):
files.extend(filenames)
break

If filename is the name of the file you've got:
import re
result = re.match(r'EMEA_(.*?)_Tracker_(\d+)_KW(\d+)', filename)
sample, year, week = result.groups()

Here is an example of how to collect all files matching your pattern into a list using regex and list comprehension. Then you can use the list as you wish in later code.
import os
import re
# Compile the regular expression pattern.
re_emea = re.compile('^EMEA_(G|BC)_Tracker_20\d{2}_KW\d{1,2}.xlsx$')
# Set path to be searched.
path = '/home/username/Desktop/so/emea_files'
# Collect all filenames matching the pattern into a list.
files = [f for f in os.listdir(path) if re_emea.match(f)]
# View the results.
print(files)
All files in the directory:
['EMEA_G_Tracker_2020_KW2.xlsx',
'other_file_3.txt',
'EMEA_G_Tracker_2020_KW1.xlsx',
'other_file_2.txt',
'other_file_5.txt',
'other_file_4.txt',
'EMEA_BC_Tracker_2019_KW52.xlsx',
'other_file_1.txt',
'EMEA_G_Tracker_2019_KW52.xlsx',
'EMEA_BC_Tracker_2020_KW2.xlsx',
'EMEA_BC_Tracker_2020_KW1.xlsx']
The results from pattern matching:
['EMEA_G_Tracker_2020_KW2.xlsx',
'EMEA_G_Tracker_2020_KW1.xlsx',
'EMEA_BC_Tracker_2019_KW52.xlsx',
'EMEA_G_Tracker_2019_KW52.xlsx',
'EMEA_BC_Tracker_2020_KW2.xlsx',
'EMEA_BC_Tracker_2020_KW1.xlsx']
Hope this helps! If not, just give me a shout.

Extracting numbers from a filename string in python

I have a number of html files in a directory. I am trying to store the filenames in a list so that I can use it later to compare with another list.
Eg: Prod224_0055_00007464_20170930.html is one of the filenames. From the filename, I want to extract '00007464' and store this value in a list and repeat the same for all the other files in the directory. How do I go about doing this? I am new to Python and any help would be greatly appreciated!
Please let me know if you need more information to answer the question.

Split the filename on underscores and select the third element (index 2).
>>> 'Prod224_0055_00007464_20170930.html'.split('_')[2]
'00007464'
In context that might look like this:
nums = [f.split('_')[2] for f in os.listdir(dir) if f.endswith('.html')]

you may try this (assuming you are in the folder with the files:
import os
num_list = []
r, d, files = os.walk( '.' ).next()
for f in files :
parts = f.split('_') # now `parts` contains ['Prod224', '0055', '00007464', '20170930.html']
print parts[2] # this outputs '00007464'
num_list.append( parts[2] )

Assuming you have a certain pattern for your files, you can use a regex:
>>> import re
>>> s = 'Prod224_0055_00007464_20170930.html'
>>> desired_number = re.findall("\d+", s)[2]
>>> desired_number
'00007464'
Using a regex will help you getting not only that specific number you want, but also other numbers in the file name.
This will work if the name of your files follow the pattern "[some text][number]_[number]_[desired_number]_[a date].html". After getting the number, I think it will be very simple to use the append method to add that number to any list you want.

How to identify files that have increasing numbers and a similar form of filename?

I have a directory of files, some of them image files. Some of those image files are a sequence of images. They could be named image-000001.png, image-000002.png and so on, or perhaps 001_sequence.png, 002_sequence.png etc.
How can we identify images that would, to a human, appear by their names to be fairly obviously in a sequence? This would mean identifying only those image filenames that have increasing numbers and all have a similar form of filename.
The similar part of the filename would not be pre-defined.

You can use a regular expression to get files adhering to a certain pattern, e.g. .*\d+.*\.(jpg|png) for anything, then a number, then more anything, and an image extension.
files = ["image-000001.png", "image-000002.png", "001_sequence.png",
"002_sequence.png", "not an image 1.doc", "not an image 2.doc",
"other stuff.txt", "singular image.jpg"]
import re
image_files = [f for f in files if re.match(r".*\d+.*\.(jpg|png)", f)]
Now, group those image files by replacing the number with some generic string, e.g. XXX:
patterns = collections.defaultdict(list)
for f in image_files:
p = re.sub("\d+", "XXX", f)
patterns[p].append(f)
As a result, patterns is
{'image-XXX.png': ['image-000001.png', 'image-000002.png'],
'XXX_sequence.png': ['001_sequence.png', '002_sequence.png']}
Similarly, it should not be too hard to check whether all those numbers are consecutive, but maybe that's not really necessary after all. Note, however, that this will have problems discriminating numbered series such as "series1_001.jpg", and "series2_001.jpg".

What I would suggest is to use regex trough files and group matching pattern with list of associated numbers from the file-name.
Once this is done, just loop trough the dictionnaries keys and ensure that count of elements is the same that the range of matched numbers.
import re
from collections import defaultdict
from os import listdir
files = listdir("/the/path/")
found_patterns = defaultdict(list)
p = re.compile("(.*?)(\d+)(.*)\.png")
for f in files:
if p.match(f):
s = p.search(f)
pattern = s.group(1) + "___" + s.group(3)
num = int(s.group(2))
found_patterns[pattern].append(num)
for pattern, found in found_patterns.items():
mini, maxi = min(found), max(found)
if len(found) == maxi - mini + 1:
print("Pattern correct: %s" % pattern)
Of course, this will not work if there are some missing value but you can use some acceptance error.

File name matching - middle of the string

I have a directory with files that follow the format: LnLnnnnLnnn.txt
where L = letters and n = numbers. E.g: p2c0789c001.txt
I would like to separate these files based on whether the second number (i.e. 0789) is odd or even.
I've only managed to get this to work if the second number ranges between 0001-0009 using the code:
odd_files = []
for root, dirs, filenames in os.walk('.'):
for filename in fnmatch.filter(filenames, 'p2c000[13579]*.txt'):
odd_files.append(os.path.join(root, filename))
This will return the files: ['./p2c0001c054.txt', './p2c0003c055.txt', './p2c0005c056.txt', './p2c0007c057.txt', './p2c0009c058.txt']
Any suggestion how could I get this to work for any given four digit number?

The easiest solution would be to expand your wildcard to match a wider array of things.
to that end I would probably do something like:
for filename in fnmatch.filter(filenames, '??????[13579]*.txt'):
This will match any characters before your values, it will match any of the odd values in your wildcard class and then it will accept anything to match afterwards.
This is a bit gross because as it is aaaaaaaa3alkjfdhalkjfshglkjzsdhfgs.txt would match and that is super gross. If you know that the data in the directories you are walking is well controlled that might be ok. A better solution might be to specify things a bit more. This could be done with the following expression:
'[a-z][0-0][a-z][0-9][0-9][0-9][13579][a-z][0-9][0-9][0-9].txt'
The fnmatch.filter method using Unix style wildcards. That means you can use the following:
? - match any single character
* - matches anything from nothing to everything
[] - this matches a class of things, use a - for a range and ! for exclusion

Would this do it?
import re
regex = re.compile("[a-z][0-9][a-z]([0-9]{4})[a-z][0-9]{3}.txt")
filter(lambda x: int(regex.match(x).groups()[0]) % 2 == 1, fnmatch)

If it's getting a little hairy, you could always turn that into a generator and code the tests by hand:
def odd_files_generator():
for root, dirs, filenames in os.walk('.'):
for filename in filenames:
if filename[6] in '13579':
yield filename
odd_files = list(odd_files_generator)
If your test is growing exceedingly hard to express tersely, replace the if filename ... line with your explicit test code.

There's no particular magic to constructing this kind of filter. It just
requires carefully constructing the appropriate regular expression and testing
against it. When using complex patterns with a lot of repetitive components,
errors can easily creep in. I like to define helper functions that make the
specification more human-readable and easier to modify later if need be.
import re
import os
# helper functions for legible re construction
LETTER = lambda n='': '({0}{1})'.format('[A-Za-z]', n)
NUM = lambda n='': '({0}{1})'.format('\d', n)
FILENAME = LETTER() + NUM() + LETTER() + NUM('{4}') + LETTER() + NUM('{3}') + '\.txt'
FILENAME_RE = re.compile(FILENAME)
is_odd = lambda n: int(n) % 2 > 0
def odd_nnnn(f):
"""
Determine if the given filename `f` matches our desired LnLnnnnLnnn.txt pattern
with the second group of numbers (nnnn) odd.
"""
m = FILENAME_RE.search(f)
return m is not None and is_odd(m.group(4))
if __name__ == '__main__':
print "Search pattern:", FILENAME
files = ['./p2c0001c054.txt', './p2c0001c055.txt', './p2c0003c055.txt', './p2c0005c056.txt', './p2c0022c056.txt', './p2c0004c056.txt', './p2c0007c057.txt', './p2c0009c058.txt', './p2c8888c056.txt', ]
files = [ os.path.normpath(f) for f in files ]
root = '/users/test/whatever'
odd_paths = [ os.path.join(root, f) for f in files if odd_nnnn(f) ]
print odd_paths
The only real downside to this is that it's a little more verbose, especially compared to a hyper-compact answer like Brad Beattie's.
[Update] It later occurred to me that a more compact way to define the regular expression might be:
FILENAME = "LnL(nnnn)Lnnn\.txt"
FILENAME_PAT = FILENAME.replace('L', r'[A-Za-z]').replace('n', r'\d')
FILENAME_RE = re.compile(FILENAME_PAT)
This more closely follows the original 'LnLnnnLnnn.txt' description. The match expression would have to change from m.group(4) to m.group(1), because just one group is captured this way.

Rename a group of files in python

I'm trying to rename some files in a directory using Python. I've looked around the forums here, and because I'm a newbie, I can't adapt what I need from what is out there.
Say in a directory I have a group of files called
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602125225754_7_S110472_I238620.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602125236347_8_S110472_I238620.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602125236894_5_S110472_I238621.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602125248691_6_S110472_I238621.jpg
and I want to remove "125225754", "125236347", "125236894" and "125248691" here so my resulting filename will be
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602_7_S110472_I238620.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602_8_S110472_I238620.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602_5_S110472_I238621.jpg
FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602_6_S110472_I238621.jpg
I'm trying to use the os.path.split but it's not working properly.
I have also considered using string manipulations, but have not been successful with that either.
Any help would be greatly appreciated. Thanks.

os.path.split splits a path (/home/mattdmo/work/projects/python/2014/website/index.html) into its component directories and file name.
As #wim suggested, if the file names are all exactly the same length, you can use string slicing to split out whatever occurs between two indexes, then join them back together. So, in your example,
filename = "FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602125248691_6_S110472_I238621.jpg"
newname = filename[:57] + filename[66:]
print(newname)
# FILENAME_002_S_0295_MR_3_Plane_Localizer__br_raw_20110602_6_S110472_I238621.jpg
This takes the first 58 characters of the string (remember in Python string indexes are 0-based) and joins it to all characters after the 67 one.
Now that you can do this, just put all the filenames into a list and iterate over it to get your new filenames:
import os
filelist = os.listdir('.') # get files in current directory
for filename in filelist:
if ".jpg" in filename: # only process pictures
newname = filename[:57] + filename[66:]
print(filename + " will be renamed as " + newname)
os.rename(filename, newname)

Can we assume that the files are all the same name up to the date _20110602[difference here]?
If that's the case then it's actually fairly easy to do.
First you need the index of that difference. Starting from character 0 which is 'F' in this case, count right until you hit that first difference. You can programatically do this by this:
s1 = 'String1'
s2 = 'String2'
i = 0
while(i < len(s1) && i < len(s2)):
if(s1[i] == s2[i]) i++
else break
And i is now set to the first difference of s1 and s2 (or if there is none, their length).
From here you know that you want to strip everything from this index to the following _.
j = i
while(j < len(s1)):
if(s1[j] != '_') j++
else break
# j is the index of the _ character after i
p1 = s1[:i] # Everything up to i
p2 = s1[j:] # Everything after j
s1 = p1.concat(p2)
# Do the same for s2, or even better, do this in a loop.
The only caveat here is that they have to be the same name up to this point for this to work. If they are the same length then this is still fairly easy, but you have to figure out yourself what the indices are rather than using the string difference method.

If you always have exact string: '20110602' in the file names stored in 'my_directory' folder:
import re #for regular expression
from os import rename
from glob import glob
for filename in glob('my_directory/*.jpg'):
match = re.search('20110602', filename)
if match:
newname = re.sub(r'20110602[0-9]+_','20110602_', filename)
rename(filename, newname)
A more general code to match any YYYYMMDD (or YYYYDDMM):
import re #for regular expression
from os import rename
from glob import glob
for filename in glob('my_directory/*.jpg'):
match = re.search(r'\d{4}\d{2}\d{2}\d+_', filename)
if match:
newname = re.sub(r'(\d{4}\d{2}\d{2})(\d+)(_)', '\\1'+'\\3', filename)
rename(filename, newname)
'\\1': This is match.group(1) that refers to the first set of parentheses
'\\3': This is match.group(3) that refers to the third set of parentheses
\d or [0-9]: are the same. They match any digit
{number}: the number of times the previous token (in this case a digit) are repeated
+ : 1 or more of previous expression (in this case a digit)