I have a folder path, for example /docs/word, and I would like to get the ID of the "word" folder (the last folder), in order to upload a file there. How do I get the ID?
So I figured it out. What you have to do is get the id of the root drive_service.about().get().execute()["rootFolderId"] , and then get the files in root, go to the next folder in the path, etc. btw, the function i wrote to list the folders inside a path and save them to a dictionary (using self.addPath())
def listFolders(self, path):
fId = self.getPathId(path) #get the id of the parent folder
files = self.drive_service.children().list(folderId=fId).execute() #Request children
files = files["items"] #All of the items in the folder
folders = []
for i in range(len(files)):
sId = files[i]["id"]
sFile = self.drive_service.files().get(fileId=sId).execute()
if sFile["labels"]["trashed"] == False and sFile["mimeType"] == "application/vnd.google-apps.folder":
self.addPath(path+sFile["title"]+"/", sFile["id"])
folders.append(sFile["title"])
return folders
Related
I would like some help to loop through some directories and subdirectories and extracting data. I have a directory with three levels, with the third level containing several .csv.gz files. The structure is like this
I need to access level 2 (where subfolders are) of each folder and check the existence of a specific folder (in my example, this will be subfolder 3; I left the other folders empty for this example, but in real cases they will have data). If checking returns True, then I want to change the name of files within the target subfolder3 and transfer all files to another folder.
Bellow is my code. It is quite cumbersome and there is probably better ways of doing it. I tried using os.walk() and this is the closest I got to a solution but it won't move the files.
import os
import shutil
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
if not files.startswith("."):
# this is to change the name of the file
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
subject_id = target
#make a new name
origin = subject_id + "/"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_name = os.path.join(destination_dir, origin)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_name)
Any suggestions on how to make this work and / or more eficient?
simply enough, you can use the built-in os module, with os.walk(path) returns you root directories and files found
import os
for root, _, files in os.walk(path):
#your code here
for your problem, do this
import os
for root, dirs, files in os.walk(parent_directory);
for file in files:
#exctract the data from the "file"
check this for more information os.walk()
and if you want to get the name of the file, you can use os.path.basename(path)
you can even check for just the gzipped csv files you're looking for using built-in fnmatch module
import fnmathch, os
def find_csv_files(path):
result = []
for root, _, files in os.walk(path):
for name in files:
if fnmatch.fnmatch(name, "*.csv.gz"): # find csv.gz using regex paterns
result.append(os.path.join(root, name))
return list(set(results)) #to get the unique paths if for some reason duplicated
Ok, guys, I was finally able to find a solution. Here it is. Not the cleanest one, but it works in my case. Thanks for the help.
def organizer(parent_dir, target_dir, destination_dir):
for root, dirs, files in os.walk(parent_dir):
if root.endswith(target_dir):
target = root
for files in os.listdir(target):
#this one because I have several .DS store files in the folder which I don't want to extract
if not files.startswith("."):
fullname = files.split(".")
just_name = fullname[0]
csv_extension = fullname[1]
gz_extension = fullname[2]
origin = target + "/" + files
full_folder_name = origin.split("/")
#make a new name
new_name = full_folder_name[5] + "_"+ just_name + "." + csv_extension + "." + gz_extension
#make a path based on this new name
new_path = os.path.join(destination_dir, new_name)
#move file from origin folder to destination folder and rename the file
shutil.move(origin, new_path)
The guess the problem was that was passing a variable that was a renamed file (in my example, I wrongly called this variable origin) as the origin path to shutil.move(). Since this path does not exist, then the files weren't moved.
I'm working on uploading some countries' admin data to my Dropbox app. Here below is my code that does that:
# importing the required libraries
import dropbox, sys, os
import requests
# Get your app key and secret from the Dropbox developer website
app_key = 'qie********'
app_secret = 'qom**********'
dbx = dropbox.Dropbox('YYPRp-*******************_JzclLe-***************-3Js')
# verify if the account is connected
dbx.users_get_current_account()
#find all the folders present
for entry in dbx.files_list_folder('').entries:
print(entry.name)
# creating a path to where all the data to be uploaded is
root_dir = "H:/WORK/Upwork/Project 7 - Python School Data Analysis/Planning and Costing Model/Updated Script/Extracted"
print ("Attempting to upload...")
z = 1
for dir, dirs, files in os.walk(root_dir):
# the first dir is the root dir itself so we skip it
if z == 1:
z = z + 1
continue
# uploading contents of the file path
elif z > 15:
# split the path to get the country, which is the very last item after split (-1)
split_dir = dir.split('\\')
folder_name = split_dir[-1] # country name
# creating a new folder in my Dropbox for each country
country_name = dbx.files_create_folder('/Data/'+ folder_name)
dropbox_folder = country_name.path_display #obtaining the name of the folder
folder_split = dropbox_folder.split('/') # splitting the path to get root folder and created folder
folder_created = folder_split[-1] #created/country folder
dest_path = os.path.join('/Data/', folder_created) #joining the two to make a full path
print(dest_path)
# looping through the files in each of the country's folder
for file in files:
try:
# getting the path for each of the file in the folder
file_path = os.path.join(dir, file)
print(f'Uploading to {folder_name} in Dropbox')
f = open(file_path, 'rb')
connect = '/' # will be used to separate the destination path and the file
# this is where the file will be saved
d_path = os.path.join(dest_path, connect, file)
dbx.files_upload(f.read(), d_path, mode=dropbox.files.WriteMode.overwrite)
print(dest_path)
print(file_path)
print(dir)
print('\n')
except Exception as err:
print("Error!", file, err)
z = z + 1
The code runs successfully with no errors. Here is how it looks at the console:
It successfully creates the folders for each of the countries. Note that in my countries' folders, it has several files (max of 15). When I visit my dropbox app, the folders are there but nothing is inside the folders. There are completely not files, I receive the message notification that says:
This Folder is Empty
See below the images:
With folders created
One of the countries, with no files:
I have given it over an hour but nothing changes. Also note that I configured all permissions necessary for writing files and folders. Could there be something I may have done wrong? I will appreciate any assistance. Thanks!
After some help from Greg, I was able to find where the issue was. The files_upload class functions expects a working path as part of its parameters. The path provided could not be found in the App and so I added the following to make it work: d_path = dest_path+d_path
Here is the full working code:
# importing the required libraries
import dropbox, sys, os
import requests
# Get your app key and secret from the Dropbox developer website
app_key = 'qie********'
app_secret = 'qom**********'
dbx = dropbox.Dropbox('YYPRp-*******************_JzclLe-***************-3Js')
# verify if the account is connected
dbx.users_get_current_account()
#find all the folders present
for entry in dbx.files_list_folder('').entries:
print(entry.name)
# creating a path to where all the data to be uploaded is
root_dir = "H:/WORK/Upwork/Project 7 - Python School Data Analysis/Planning and Costing Model/Updated Script/Extracted"
print ("Attempting to upload...")
z = 1
for dir, dirs, files in os.walk(root_dir):
# the first dir is the root dir itself so we skip it
if z == 1:
z = z + 1
continue
# uploading contents of the file path
elif z > 15:
# split the path to get the country, which is the very last item after split (-1)
split_dir = dir.split('\\')
folder_name = split_dir[-1] # country name
# creating a new folder in my Dropbox for each country
country_name = dbx.files_create_folder('/Data/'+ folder_name)
dropbox_folder = country_name.path_display #obtaining the name of the folder
folder_split = dropbox_folder.split('/') # splitting the path to get root folder and created folder
folder_created = folder_split[-1] #created/country folder
dest_path = os.path.join('/Data/', folder_created) #joining the two to make a full path
print(dest_path)
# looping through the files in each of the country's folder
for file in files:
try:
# getting the path for each of the file in the folder
file_path = os.path.join(dir, file)
print(f'Uploading to {folder_name} in Dropbox')
f = open(file_path, 'rb')
connect = '/' # will be used to separate the destination path and the file
# this is where the file will be saved
d_path = os.path.join(dest_path, connect, file)
d_path = dest_path+d_path
dbx.files_upload(f.read(), d_path, mode=dropbox.files.WriteMode.overwrite)
print(dest_path)
print(file_path)
print(dir)
print('\n')
except Exception as err:
print("Error!", file, err)
z = z + 1
I am doing a school assignment where I have to take input from a user and save it to a text file.
My file structure will be something like:
- Customer register
- Customer ID
- .txt files 1-5
It can be saved in the python folder and I can make the folders like this:
os.makedirs("Customer register/Customer ID")
My question is, how do I set the path the text files are to be stored in, in the directory when I don't know the directory? So that no matter where the program is run it is saved in the "Customer ID" folder I create (but on the computer the program is run on)?
Also, how do I make this work on both windows and mac?
I also want to program to be able to be executed several times, and check if the folder is there and save to the "Customer ID" folder if it already exists. Is there a way to do that?
EDIT:
This is the code I am trying to use:
try:
dirs = os.makedirs("Folder")
path = os.getcwd()
os.chdir(path + "/Folder")
print (os.getcwd())
except:
if os.path.exists:
path = os.getcwd()
unique_filename = str(uuid.uuid4())
customerpath = os.getcwd()
os.chdir(customerpath + "/Folder/" + unique_filename)
I am able to create a folder and change the directory (everything in "try" works as I want).
When this folder is created I want to create a second folder with a random generated folder name (used for saving customer files). I can't get this to work in the same way.
Error:
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\48736\PycharmProjects\tina/Folder/979b9026-b2f6-4526-a17a-3b53384f60c4'
EDIT 2:
try:
os.makedirs("Folder")
path = os.getcwd()
os.chdir(path + "/Folder")
print (os.getcwd())
except:
if os.path.exists:
path = os.getcwd()
os.chdir(os.path.join(path, 'Folder'))
print(os.getcwd())
def userId(folderid):
try:
if not os.path.exists(folderid):
os.makedirs(folderid)
except:
if os.path.exists(folderid):
os.chdir(path + "/Folder/" + folderid)
userId(str(uuid.uuid4()))
print(os.getcwd())
So I can now create a folder, change directory to the folder I have created and create a new folder with a unique filename within that folder.
But I can't change the directory again to the folder with the unique filename.
Any suggestions?
I have tried:
os.chdir(path + "/Folder/" + folderid)
os.chdir(path, 'Folder', folderid)
os.chdir(os.path.join(path, 'Folder', folderid))
But is still just stays in: C:\Users\47896\PycharmProjects\tina\Folder
You can use relative paths in your create directory command, i.e.
os.makedirs("./Customer register/Customer ID")
to create folder in project root (=where the primary caller is located) or
os.makedirs("../Customer register/Customer ID") in parent directory.
You can, of course, traverse the files tree as you need.
For specific options mentioned in your question, please, see makedirs documentation at Python 3 docs
here is solution
import os
import shutil
import uuid
path_on_system = os.getcwd() # directory where you want to save data
path = r'Folder' # your working directory
dir_path = os.path.join(path_on_system, path)
if not os.path.exists(dir_path):
os.makedirs(dir_path)
file_name = str(uuid.uuid4()) # file which you have created
if os.path.exists(file_name) and os.path.exists(dir_path):
shutil.move(file_name,os.path.join(dir_path,file_name))
else:
print(" {} does not exist".format(file_name))
I have files I would like to move to different folders based on their file name.
examples of the file names are:
296_A_H_1_1_20070405.pdf
296_A_H_10_1_20070405.pdf
296_A_H_2_1_20070405.pdf
296_A_H_20_1_20070405.pdf
The corresponding folder names are:
296_A_H_1
296_A_H_2
296_A_H_10
296_A_H_20
I want to move the files to the correct folder based on the file name. For example, 296_A_H_1_1_20070405.pdf should go in the 296_A_H_1 folder. Here is the code I have so far:
import shutil
import os
#filepath to files
source = 'C:\\Users\\Desktop\\test folder'
#filepath to destination folders
dest1 = 'C:\\Users\\Desktop\\move file\\296_A_H_1'
dest2 = 'C:\\Users\\Desktop\\move file\\296_A_H_2'
dest3 = 'C:\\Users\\Desktop\\move file\\296_A_H_10'
dest4 = 'C:\\Users\\Desktop\\move file\\296_A_H_20'
files = [os.path.join(source, f) for f in os.listdir(source)]
#move files to destination folders based on file path and name
for f in files:
if (f.startswith("C:\\Users\\Desktop\\test folder\\296_A_H_1_")):
shutil.copy(f,dest1)
elif (f.startswith("C:\\Users\\Desktop\\test folder\\296_A_H_2_")):
shutil.copy(f,dest2)
elif (f.startswith("C:\\Users\\Desktop\\test folder\\296_A_H_10")):
shutil.copy(f, dest3)
elif (f.startswith("C:\\Users\\Desktop\\test folder\\296_A_H_20")):
shutil.copy(f, dest4)
This code works but I need to move 400 files to different folders and would have to write hundreds of elif statements. How can I do this by matching the file name to the destination folder and using shutil to copy the file to that folder? I am just starting to learn Python so any help here would be greatly appreciated!
How about this? You basically map destinations to the files that should go to those destinations using string membership testing (i.e. if d in x). This of course assumes that all of the files to be moved start in the source folder and all of the destinations are in the same folder ('C:\\Users\\Desktop\\move file').
source = 'C:\\Users\\Desktop\\test folder'
dests = os.listdir('C:\\Users\\Desktop\\move file')
# Map each destination to the files that should go to that destination
dest_file_mapping = {d: {os.path.join(source, x
for x in os.listdir(source) if d in x} for d in dests}
for dest, files in dest_file_mapping.items():
for f in files:
shutil.copy(f, dest)
Using Python 3.4
I am generating files that are will look like 'Report_XXXXXX.xlsx' with X being unique customer numbers. I have a server with folders that are named 'XXXXXX.CustomerName'. I am trying to loop through each file/report and upload it to the matching folder based on the customer no. I have something that is working in my small test environment but I don't think it is working how I want. It uploads the files, but I am trying to catch anytime it doesn't find a match. Currently it fails my IF statment for every file. I think I am looping too many times or over the wrong items.
import os
import ftplib
creds = [line.rstrip('\n') for line in open('C:\\folder\\credentials.txt')]
ftp = ftplib.FTP_TLS("server.com")
ftp.login(creds[0], creds[1])
ftp.prot_p()
src_dir = 'C:\\Reports\\'
src_files = os.listdir('C:\\Reports\\')
for folder_name in ftp.nlst():
for file_name in src_files:
if folder_name[0:6] == file_name[7:-5]:
ftp.cwd('/'+folder_name)
open_file = open(src_dir+file_name, 'rb')
ftp.storbinary('STOR '+file_name, open_file)
open_file.close()
else:
print('Folder ' + folder_name + ' Not Found')
ftp.quit()
So for example the source directory has 3 files: 'Report_100002.xlsx, Report_100003.xlsx, Report_100007.xlsx' And the server has matching folders and a few extra folders. The files upload, and the output looks like so:
Folder 100000.CustomerName Not Found
Folder 100000.CustomerName Not Found
Folder 100000.CustomerName Not Found
Folder 100002.CustomerName Not Found
Folder 100002.CustomerName Not Found
Folder 100003.CustomerName Not Found
Folder 100003.CustomerName Not Found
Folder 100007.CustomerName Not Found
Folder 100007.CustomerName Not Found
I am trying to get to a state where I can properly log each item and whether it was a success, what folder it landed in, etc...
In your inner for loop you compare all 3 file names in src_dir with folder_name, but maximally only one satisfies the condition in your if statement. So the other 2 or 3 files that don't match cause the output you are seeing, for every folder on the ftp server. You could use a flag to keep track of whether a match was found and print your output based on that flag.
Another thing is that you should start iterating over src_files and then find matching folder names by iterating over ftp.nlist() (you are interested in source files that don't have a matching folder, not the other way around). So something like this (assuming a source file is allowed to end up in multiple folders):
....
folder_names = ftp.nlst()
for file_name in src_files:
folder_found = False
for folder_name in folder_names:
if folder_name[0:6] == file_name[7:-5]:
folder_found = True
ftp.cwd('/'+folder_name)
open_file = open(src_dir+file_name, 'rb')
ftp.storbinary('STOR '+file_name, open_file)
open_file.close()
if not folder_found:
print('No destination folder found for ' + file_name)
ftp.quit()
(the folder_names = ftp.nlst() is there so you don't repeatedly list the directories on the server)