How to iterate through multiple folders within a folder and export specific CSV filename that begins for example- "abc*.csv" and export them into a new folder directory.
Trying to search similar example in Stack overflow but most examples were reading multiple CSV files within a folder and combine them to one data frame. Thanks.
import os
import shutil
files = []
source_dir = ['./files/']
dest_dir = './export/'
# List all files from all directories within the path
while len(source_dir) > 0:
for (dirpath, dirnames, filenames) in os.walk(source_dir.pop()):
source_dir.extend(dirnames)
files.extend(map(lambda n: os.path.join(
*n), zip([dirpath] * len(filenames), filenames)))
# Loop thru files to copy/move the matching CSVs
for file in files:
if file.startswith('abc') and file.endswith('.csv'):
shutil.copy(file, dest_dir)
# shutil.move(file, dest_dir) # Use .move to move the file instead
If you have this file structure:
files
├── dir1
│ ├── abc789.csv
│ └── abc789.txt
├── dir2
│ ├── abc098.csv
│ └── abc098.txt
└── dir3
├── abc456.csv
└── subdir3
├── abc123.csv
└── abc123.txt
Only the matching files will be exported:
$ ls export/
abc098.csv abc123.csv abc456.csv abc789.csv
I'm working on a directory with 27 folders and various XML files inside each folder.
was able to:
parse one XML file and write to a CSV file
traverse one folder and read and parse all XML files in it
challenges:
having a problem trying to go through and parse all the XML files from the root folder to its subfolders
Send help and thank you. Code snippets below
# working in one folder only
import csv
import xml.etree.ElementTree as ET
import os
## directory
path = "/Users.../y"
filenames = []
## Count the number of xml files of each folder
files = os.listdir(path)
print("\n")
xml_data_to_csv = open('/Users.../xml_extract.csv', 'w')
list_head = []
csvwriter = csv.writer(xml_data_to_csv)
# Read XML files in a folder
for filename in os.listdir(path):
if not filename.endswith('.xml'):
continue
fullname = os.path.join(path,filename)
print("\n", fullname)
filenames.append(fullname)
# parse elements in each XML file
for filename in filenames:
tree = ET.parse(filename)
root = tree.getroot()
extract_xml=[]
## extract child elements per xml file
print("\n")
for x in root.iter('Info'):
for element in x:
print(element.tag,element.text)
extract_xml.append(element.text)
## Write list nodes to csv
csvwriter.writerow(extract_xml)
## Close CSV file
xml_data_to_csv.close()
You can get the list of all the XML files in a given path with
import os
path = "main/root"
filelist = []
for root, dirs, files in os.walk(path):
for file in files:
if not file.endswith('.xml'):
continue
filelist.append(os.path.join(root, file))
for file in filelist:
print(file)
# or in your case parse the XML 'file'
If for instance:
$ tree /main/root
/main/root
├── a
│ ├── a.xml
│ ├── b.xml
│ └── c.xml
├── b
│ ├── d.xml
│ ├── e.xml
│ └── x.txt
└── c
├── f.xml
└── g.xml
we get:
/main/root/c/g.xml
/main/root/c/f.xml
/main/root/b/e.xml
/main/root/b/d.xml
/main/root/a/c.xml
/main/root/a/b.xml
/main/root/a/a.xml
If you want to sort directories and files:
for root, dirs, files in os.walk(path):
dirs.sort()
for file in sorted(files):
if not file.endswith('.xml'):
continue
filelist.append(os.path.join(root, file))
You can use os.walk:
import os
for dir_name, dirs, files in os.walk('<root_dir>'):
# parse files
You can use the pathlib module to "glob" the XML files. It will search all subdirectories for the pattern you supply and return Path objects that already include the path to the file. Cleaning up your script a bit, you would have
import csv
import xml.etree.ElementTree as ET
from pathlib import Path
## directory
path = Path("/Users.../y")
with open('/Users.../xml_extract.csv', 'w') as xml_data_to_csv:
csvwriter = csv.writer(xml_data_to_csv)
# Read XML files in a folder
for filepath in path.glob("**/*.xml"):
tree = ET.parse(filename)
root = tree.getroot()
extract_xml=[]
## extract child elements per xml file
print("\n")
for x in root.iter('Info'):
for element in x:
print(element.tag,element.text)
extract_xml.append(element.text)
## Write list nodes to csv
csvwriter.writerow(extract_xml)
Is there a way of importing all the files within folder1? Each csv file is contained within a subfolder. Below is the file structure.
C:/downloads/folder1 > tree /F
C:.
│ tree
│
├───2020-06
│ test1.csv
│
├───2020-07
│ test2.csv
│
├───2020-08
│ test3.csv
│
├───2020-09
│ test4.csv
I'm aware of glob, below, to take all files within a folder. However can this be used for subfolders?
import glob
import pandas as pd
# Get a list of all the csv files
csv_files = glob.____('*.csv')
# List comprehension that loads of all the files
dfs = [pd.read_csv(____) for ____ in ____]
# List comprehension that looks at the shape of all DataFrames
print(____)
Use the recursive keyword argument of the glob.glob() method:
glob.glob('**\\*.csv', recursive=True)
You can use os.walk to find all sub_folder and get the required files
here's a code sample
import os
import pandas as pd
path = '<Insert Path>'
file_extension = '.csv'
csv_file_list = []
for root, dirs, files in os.walk(path):
for name in files:
if name.endswith(file_extension):
file_path = os.path.join(root, name)
csv_file_list.append(file_path)
dfs = [pd.read_csv(f) for f in csv_file_list]
I found this on Kite's website, check it out
path = "./directory/src_folder"
text_files = glob.glob(path + "/**/*.txt", recursive = True)
print(text_files)
OUTPUT
['./directory/src_folder/src_file.txt', './directory/src_folder/subdirectory/subdirectory_file.txt']
I am relatively new to Python, I am trying to make a script which will only zip subfolders that I define in a list, with their content of course. I tried modifying various codes from Stack Overflow and whatever I found on internet but either I zip all subfolders, or I zip subfolders that I want but without content.
List could be full path to subfolder or can I define the path to the root folder and then specify subfolders?
This is the idea: 3 subfolders 1,2,3 and I want to zip only subfolders 1 and 3. I added the last code that I was modifying but I just can't return the list in a function.
Folder
|- SubFolder1
| |- file1.txt
| |- file2.txt
|- SubFolder2
| |- file1.txt
| |- file2.txt
|- SubFolder3
| |- file1.txt
| |- file2.txt
The code:
import os
import zipfile
list=["SubFolder1", "SubFolder3"]
def zipdir(path, ziph):
# ziph is zipfile handle
for root, dirs, files in os.walk(path):
for file in files:
ziph.write(os.path.join(root, file))
if __name__ == '__main__':
zipf = zipfile.ZipFile('Python.zip', 'w', zipfile.ZIP_DEFLATED)
zipdir(**list**, zipf)
zipf.close()
I modified your code and think this way it should work as expected if I understand your request correctly. Please check out this thread too, I think it's pretty much what you are looking for ;-)
1) os.walk goes through each of your subfolders, too. So just wait until each subfolder becomes the "root" (here is an example).
2) Check whether the folder is contained in the list. If it is, zip all files within this folder
3) Recreate the subfolder structure relative to your original root-folder
import os
import zipfile
rootpath = r'C:\path\to\your\rootfolder'
list=["SubFolder1", "SubFolder3"]
def zipdir(path, ziph, dirlist):
# ziph is zipfile handle
for root, dirs, files in os.walk(path):
if os.path.basename(root) in dirlist:
for file in files:
file_path = os.path.join(root, file)
relative_path = os.path.relpath(file_path, path)
ziph.write(file_path, relative_path)
if __name__ == '__main__':
zipf = zipfile.ZipFile('Python.zip', 'w', zipfile.ZIP_DEFLATED)
zipdir(rootpath, zipf, list)
zipf.close()
I have a Python script that zips a file (new.txt):
tofile = "/root/files/result/"+file
targetzipfile = new.zip # This is how I want my zip to look like
zf = zipfile.ZipFile(targetzipfile, mode='w')
try:
#adding to archive
zf.write(tofile)
finally:
zf.close()
When I do this I get the zip file. But when I try to unzip the file I get the text file inside of a series of directories corresponding to the path of the file i.e I see a folder called root in the result directory and more directories within it, i.e. I have
/root/files/result/new.zip
and when I unzip new.zip I have a directory structure that looks like
/root/files/result/root/files/result/new.txt
Is there a way I can zip such that when I unzip I only get new.txt?
In other words I have /root/files/result/new.zip and when I unzip new.zip, it should look like
/root/files/results/new.txt
The zipfile.write() method takes an optional arcname argument that specifies what the name of the file should be inside the zipfile
I think you need to do a modification for the destination, otherwise it will duplicate the directory. Use :arcname to avoid it. try like this:
import os
import zipfile
def zip(src, dst):
zf = zipfile.ZipFile("%s.zip" % (dst), "w", zipfile.ZIP_DEFLATED)
abs_src = os.path.abspath(src)
for dirname, subdirs, files in os.walk(src):
for filename in files:
absname = os.path.abspath(os.path.join(dirname, filename))
arcname = absname[len(abs_src) + 1:]
print 'zipping %s as %s' % (os.path.join(dirname, filename),
arcname)
zf.write(absname, arcname)
zf.close()
zip("src", "dst")
zf.write(tofile)
to change
zf.write(tofile, zipfile_dir)
for example
zf.write("/root/files/result/root/files/result/new.txt", "/root/files/results/new.txt")
To illustrate most clearly,
directory structure:
/Users
└── /user
. ├── /pixmaps
. │ ├── pixmap_00.raw
. │ ├── pixmap_01.raw
│ ├── /jpeg
│ │ ├── pixmap_00.jpg
│ │ └── pixmap_01.jpg
│ └── /png
│ ├── pixmap_00.png
│ └── pixmap_01.png
├── /docs
├── /programs
├── /misc
.
.
.
Directory of interest: /Users/user/pixmaps
First attemp
import os
import zipfile
TARGET_DIRECTORY = "/Users/user/pixmaps"
ZIPFILE_NAME = "CompressedDir.zip"
def zip_dir(directory, zipname):
"""
Compress a directory (ZIP file).
"""
if os.path.exists(directory):
outZipFile = zipfile.ZipFile(zipname, 'w', zipfile.ZIP_DEFLATED)
for dirpath, dirnames, filenames in os.walk(directory):
for filename in filenames:
filepath = os.path.join(dirpath, filename)
outZipFile.write(filepath)
outZipFile.close()
if __name__ == '__main__':
zip_dir(TARGET_DIRECTORY, ZIPFILE_NAME)
ZIP file structure:
CompressedDir.zip
.
└── /Users
└── /user
└── /pixmaps
├── pixmap_00.raw
├── pixmap_01.raw
├── /jpeg
│ ├── pixmap_00.jpg
│ └── pixmap_01.jpg
└── /png
├── pixmap_00.png
└── pixmap_01.png
Avoiding the full directory path
def zip_dir(directory, zipname):
"""
Compress a directory (ZIP file).
"""
if os.path.exists(directory):
outZipFile = zipfile.ZipFile(zipname, 'w', zipfile.ZIP_DEFLATED)
# The root directory within the ZIP file.
rootdir = os.path.basename(directory)
for dirpath, dirnames, filenames in os.walk(directory):
for filename in filenames:
# Write the file named filename to the archive,
# giving it the archive name 'arcname'.
filepath = os.path.join(dirpath, filename)
parentpath = os.path.relpath(filepath, directory)
arcname = os.path.join(rootdir, parentpath)
outZipFile.write(filepath, arcname)
outZipFile.close()
if __name__ == '__main__':
zip_dir(TARGET_DIRECTORY, ZIPFILE_NAME)
ZIP file structure:
CompressedDir.zip
.
└── /pixmaps
├── pixmap_00.raw
├── pixmap_01.raw
├── /jpeg
│ ├── pixmap_00.jpg
│ └── pixmap_01.jpg
└── /png
├── pixmap_00.png
└── pixmap_01.png
The arcname parameter in the write method specifies what will be the name of the file inside the zipfile:
import os
import zipfile
# 1. Create a zip file which we will write files to
zip_file = "/home/username/test.zip"
zipf = zipfile.ZipFile(zip_file, 'w', zipfile.ZIP_DEFLATED)
# 2. Write files found in "/home/username/files/" to the test.zip
files_to_zip = "/home/username/files/"
for file_to_zip in os.listdir(files_to_zip):
file_to_zip_full_path = os.path.join(files_to_zip, file_to_zip)
# arcname argument specifies what will be the name of the file inside the zipfile
zipf.write(filename=file_to_zip_full_path, arcname=file_to_zip)
zipf.close()
You can isolate just the file name of your sources files using:
name_file_only= name_full_path.split(os.sep)[-1]
For example, if name_full_path is /root/files/results/myfile.txt, then name_file_only will be myfile.txt. To zip myfile.txt to the root of the archive zf, you can then use:
zf.write(name_full_path, name_file_only)
Check out the documentation for Zipfile.write.
ZipFile.write(filename[, arcname[, compress_type]]) Write the file
named filename to the archive, giving it the archive name arcname (by
default, this will be the same as filename, but without a drive letter
and with leading path separators removed)
https://docs.python.org/2/library/zipfile.html#zipfile.ZipFile.write
Try the following:
import zipfile
import os
filename = 'foo.txt'
# Using os.path.join is better than using '/' it is OS agnostic
path = os.path.join(os.path.sep, 'tmp', 'bar', 'baz', filename)
zip_filename = os.path.splitext(filename)[0] + '.zip'
zip_path = os.path.join(os.path.dirname(path), zip_filename)
# If you need exception handling wrap this in a try/except block
with zipfile.ZipFile(zip_path, 'w') as zf:
zf.write(path, zip_filename)
The bottom line is that if you do not supply an archive name then the filename is used as the archive name and it will contain the full path to the file.
It is much simpler than expected, I configured the module using the parameter "arcname" as "file_to_be_zipped.txt", so the folders do not appear in my final zipped file:
mmpk_zip_file = zipfile.ZipFile("c:\\Destination_folder_name\newzippedfilename.zip", mode='w', compression=zipfile.ZIP_DEFLATED)
mmpk_zip_file.write("c:\\Source_folder_name\file_to_be_zipped.txt", "file_to_be_zipped.txt")
mmpk_zip_file.close()
We can use this
import os
# single File
os.system(f"cd {destinationFolder} && zip fname.zip fname")
# directory
os.system(f"cd {destinationFolder} && zip -r folder.zip folder")
For me, This is working.
Specify the arcname input of the write method as following:
tofile = "/root/files/result/"+file
NewRoot = "files/result/"
zf.write(tofile, arcname=tofile.split(NewRoot)[1])
More info:
ZipFile.write(filename, arcname=None, compress_type=None,
compresslevel=None)
https://docs.python.org/3/library/zipfile.html
I face the same problem and i solve it with writestr. You can use it like this:
zipObject.writestr(<filename> , <file data, bytes or string>)
If you want an elegant way to do it with pathlib you can use it this way:
from pathlib import Path
import zipfile
def zip_dir(path_to_zip: Path):
zip_file = Path(path_to_zip).with_suffix('.zip')
z = zipfile.ZipFile(zip_file, 'w', zipfile.ZIP_DEFLATED)
for f in list(path_to_zip.rglob('*.*')):
z.write(f, arcname=f.relative_to(path_to_zip))
To get rid of the absolute path, I came up with this:
def create_zip(root_path, file_name, ignored=[], storage_path=None):
"""Create a ZIP
This function creates a ZIP file of the provided root path.
Args:
root_path (str): Root path to start from when picking files and directories.
file_name (str): File name to save the created ZIP file as.
ignored (list): A list of files and/or directories that you want to ignore. This
selection is applied in root directory only.
storage_path: If provided, ZIP file will be placed in this location. If None, the
ZIP will be created in root_path
"""
if storage_path is not None:
zip_root = os.path.join(storage_path, file_name)
else:
zip_root = os.path.join(root_path, file_name)
zipf = zipfile.ZipFile(zip_root, 'w', zipfile.ZIP_DEFLATED)
def iter_subtree(path, layer=0):
# iter the directory
path = Path(path)
for p in path.iterdir():
if layer == 0 and p.name in ignored:
continue
zipf.write(p, str(p).replace(root_path, '').lstrip('/'))
if p.is_dir():
iter_subtree(p, layer=layer+1)
iter_subtree(root_path)
zipf.close()
Maybe it isn't the most elegant solution, but this works. If we just use p.name when providing the file name to write() method, then it doesn't create the proper directory structure.
Moreover, if it's needed to ignore the selected directories or files from the root path, this ignores those selections too.
This is an example I used. I have one excel file, Treport where I am using python + pandas in my dowork function to create pivot tables, etc. for each of the companies in CompanyNames. I create a zip file of the csv and a non-zip file so I can check as well.
The writer specifies the path where I want my .xlsx to go and for my zip files, I specify that in the zip.write(). I just specify the name of the xlsx file that was recently created, and that is what gets zipped up, not the whole directory. Beforehand I was just specifying 'writer' and would zip up the whole directory. This allows me to zip up just the recently created excel file.
Treport = 'TestReportData.csv'
CompanyNames = ['Company1','Company2','Company3']
for CompName in CompanyNames:
strcomp = str(CompName)
#Writer Creates pathway to output report to. Each company gets unique file.
writer = pd.ExcelWriter(f"C:\\Users\\MyUser\\Documents\\{strcomp}addReview.xlsx", engine='xlsxwriter')
DoWorkFunction(CompName, Treport, writer)
writer.save()
with ZipFile(f"C:\\Users\\MyUser\\Documents\\{strcomp}addR.zip", 'w') as zip:
zip.write(writer, f"{strcomp}addReview.xlsx")