I am new to python and I want to know if we can upload files from our local system to box.com?
Or else can we take help from any mediator like Jenkins to upload this files?
You can use the below boxsdk library code.
def upload_file_to_box(client, folder_id, filename):
folder = client.folder(folder_id=folder_id)
items = folder.get_items()
for item in items:
if item.name == filename:
updated_file = client.file(item.id).update_contents(item.name)
print('File "{0}" has been updated'.format(updated_file.name))
return
uploaded_file = folder.upload(filename)
print('File "{0}" has been uploaded'.format(uploaded_file.name))
This will check for a specific file name and compare it with all files names in the folder and updates a new version if exists, otherwise uploads a new file.
Also you can search the filename inside a folder using search API by using the below code. But the search API has a time lag of 10 minutes or greater.
items = client.search().query(query='"{}"'.format(filename), limit=100, ancestor_folders=[folder])
I don't know if I understood your question correctly, but there is a package for python to connect to the box platform through an API http://opensource.box.com/box-python-sdk/tutorials/intro.html
Related
I have this script written in python which looks thrue folder 'CSVtoGD', list every CSV there and send those CSV's as independent sheets to my google drive. I am trying to write a line which will delete the old files when I run the program again. What am I missing here? I am trying to achieve that by using:
sh = gc.del_spreadsheet(filename.split(".")[0]+" TTF")
Unfortunately the script is doing the same thing after adding this line. It is uploading new files but not deleting old ones.
Whole script looks like that
import gspread
import os
gc = gspread.oauth(credentials_filename='/users/user/credentials.json')
os.chdir('/users/user/CSVtoGD')
files = os.listdir()
for filename in files:
if filename.split(".")[1] == "csv":
folder_id = '19vrbvaeDqWcxFGwPV82APWYTmB'
sh = gc.del_spreadsheet(filename.split(".")[0]+" TTF")
sh = gc.create(filename.split(".")[0]+" TTF", folder_id)
content = open(filename, 'r').read().encode('utf-8')
gc.import_csv(sh.id, content)
Everything is working fine, CSVs from folder are uploaded to google drive, my problem is with deleting the old CSV (with the same name as new ones)
When I saw the document of gspread, it seems that the argument of the method of del_spreadsheet is the file ID. Ref When I saw your script, you are using the filename as the argument. I thought that this might be the reason for your issue. When this is reflected in your script, it becomes as follows.
From:
sh = gc.del_spreadsheet(filename.split(".")[0]+" TTF")
To:
sh = gc.del_spreadsheet(gc.open(filename.split(".")[0] + " TTF").id)
Note:
When the Spreadsheet of the filename of filename.split(".")[0] + " TTF" is not existing, an error occurs. Please be careful about this.
Reference:
del_spreadsheet(file_id)
Added:
From your reply of When I try do delete other file using this method from My Drive it is working well., it was found that my proposed modification can be used for "My Drive". But, it seems that this cannot be used for the shared drive.
When I saw the script of gspread again, I noticed that the current request cannot search the files in the shared drive using the filename. And also, I confirmed that in the current gspread, the Spreadsheet ID cannot be retrieved using gspread. Because the files cannot be searched from all shared drives. By this, I would like to propose the following modified script.
Modified script:
import gspread
import os
from googleapiclient.discovery import build
gc = gspread.oauth(credentials_filename='/users/user/credentials.json')
service = build("drive", "v3", credentials=gc.auth)
def getSpreadsheetId(filename):
q = f"name='{filename}' and mimeType='application/vnd.google-apps.spreadsheet' and trashed=false" # or q = "name='" + filename + "' and mimeType='application/vnd.google-apps.spreadsheet' and trashed=false"
res = service.files().list(q=q, fields="files(id)", corpora="allDrives", includeItemsFromAllDrives=True, supportsAllDrives=True).execute()
items = res.get("files", [])
if not items:
print("No files found.")
exit()
return items[0]["id"]
os.chdir('/users/user/CSVtoGD')
files = os.listdir()
for filename in files:
fname = filename.split(".")
if fname[1] == "csv":
folder_id = '19vrbvaeDqWcxFGwPV82APWYTmB'
oldSpreadsheetId = getSpreadsheetId(fname[0] + " TTF")
sh = gc.del_spreadsheet(oldSpreadsheetId)
sh = gc.create(fname[0] + " TTF", folder_id)
content = open(filename, "r").read().encode("utf-8")
gc.import_csv(sh.id, content)
In this modification, in order to retrieve the Spreadsheet ID from the filename in the shared drive, googleapis for python is used. Ref
But, in this case, it supposes that you have the permission for writing to the shared drive. Please be careful about this.
I'm trying to pull a list of dictionaries from Dropbox API for some images and to be represented as follows;
[{name: 'XYZ', url: 'THIS IS A URL'}]
I've use the following which generates a list of file names and path's just fine.
path = ''
files_list = []
def dropbox_list_files(path):
dbx = dropbox_connect()
try:
files = dbx.files_list_folder(path).entries
for file in files:
if isinstance(file, dropbox.files.FileMetadata):
metadata = {
'name': file.name,
'path_lower': file.path_lower,
}
files_list.append(metadata)
But I'm stuck on now creating the sharable links, aswell as then appending these to each of the relevant files in a list.
In the Dropbox API documentation they have the 'sharing_create_shared_link_with_settings' function which looks like it can do this, but requires a file path, which I am not sure if this is just a single path, or how I could A) pass in an iterable of paths for all the files in sequence, and then B) how would I append these together for the purpose explained above?
Any help is much appreciated.
Once you have the files_list populated, loop through and pass the file path to the sharing_create_shared_link method. To do that, you would add something like this to your script to print a list of links..
for file in files_list:
try:
link = dbx.sharing_create_shared_link(file['path_lower'])
print(link.url)
except Exception as e:
print(e)
Assuming you are using the Dropbox SDK, you will want to ensure you have appropriate scopes set (e.g. sharing.write) otherwise will you encounter a permission error.
service = self.auth()
items = self.listFilesInFolder(downLoadFolderKey)
for item in items:
file_id = (item.get('id'))
file_name = (item.get('name'))
request = service.files().get_media(fileId=file_id)
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()
print ("Download %d%%." % int(status.progress() * 100) + file_name)
filepath = fileDownPath + file_name
with io.open(filepath, 'wb') as f:
fh.seek(0)
f.write(fh.read())
I am using Google Drive API v3.
I am trying to download a full directory. But the problem is the directory itself contains folders and when I try to run this bit of code. This error happens.
<HttpError 403 when requesting https://www.googleapis.com/drive/v3/files/1ssF0XD8pi6oh6DXB1prIJPWKMz9dggm2?alt=media returned "Only files with binary content can be downloaded. Use Export with Google Docs files.">
The error I figure is due to it trying to download the folders, within the directory. But how do I download the full directory?
P.S The directory changes so I cannot hard code file IDs and then download the files.
I believe your situation and goal as follows.
By items = self.listFilesInFolder(downLoadFolderKey), you have already been able to retrieve all file and folder list including the subfolders under the specific folder.
items include the mimeType for each files and folders.
In your issue, when the folder is used in the loop, the error occurs.
You want to remove this error.
For this, how about this answer?
Modification point:
When the mimeType is included in items of items = self.listFilesInFolder(downLoadFolderKey), the folder can be checked by the mimeType. The mimeType of folder is application/vnd.google-apps.folder.
From your script, I think that when the Google Docs file (Spreadsheet, Document, Slides and so on) is downloaded with the method of "Files: get", the same error occurs.
In order to download the Google Docs files, it is required to use the method of "Files: export".
When above point is reflected to your script, how about the following modification?
Modified script:
From:
request = service.files().get_media(fileId=file_id)
To:
file_mimeType = (item.get('mimeType'))
if file_mimeType == 'application/vnd.google-apps.folder':
continue
request = service.files().export_media(fileId=file_id, mimeType='application/pdf') if 'application/vnd.google-apps' in file_mimeType else service.files().get_media(fileId=file_id)
In this modification, at first, please confirm whether the file mimeType to items of items = self.listFilesInFolder(downLoadFolderKey) is included, again. By this, the folder can be skipped and also, Google Docs files and the files except for Google Docs can be downloaded using the value of mimeType.
In this modification, as a sample modification, Google Docs files are downloaded as the PDF file. If you want to change the output mimeType, please modify mimeType='application/pdf'.
References:
G Suite and Drive MIME Types
Files: get
Files: export
So, before anyone tells me about the flat structure of S3, I already know, but the fact is you can create 'folders' in S3. My objective with this Python code is to create a new folder named using the date of running and appending the user's input to this (which is the createS3Folder function) - I then want to sync a folder in a local directory to this folder.
The problem is that my upload_files function creates a new folder in S3 that exactly emulates the folder structure of my local set up.
Can anyone suggest how I would just sync the folder into the newly created one without changing names?
import sys
import boto3
import datetime
import os
teamName = raw_input("Please enter the name of your project: ")
bucketFolderName = ""
def createS3Folder():
date = datetime.date.today().strftime("%Y") + "." +
datetime.date.today().strftime("%B") + "." +
datetime.date.today().strftime("%d")
date1 = datetime.date.today()
date = str(date1) + "/" #In order to generate a file, you must
put "/" at the end of key
bucketFolderName = date + teamName + "/"
client = boto3.client('s3')
client.put_object(Bucket='MY_BUCKET',Key=bucketFolderName)
upload_files('/Users/local/directory/to/sync')
def upload_files(path):
session = boto3.Session()
s3 = session.resource('s3')
bucket = s3.Bucket('MY_BUCKET')
for subdir, dirs, files in os.walk(path):
for file in files:
full_path = os.path.join(subdir, file)
with open(full_path, 'rb') as data:
bucket.put_object(Key=bucketFolderName, Body=data)
def main():
createS3Folder()
if __name__ == "__main__":
main()
Your upload_files() function is uploading to:
bucket.put_object(Key=bucketFolderName, Body=data)
This means that the filename ("Key") on S3 will be the name of the 'folder'. It should be:
bucket.put_object(Key=bucketFolderName + '/' + file, Body=data)
The Key is the full path of the destination object, including the filename (not just a 'directory').
In fact, there is no need to create the 'folder' beforehand -- just upload to the desired Key.
If you are feeling lazy, use the AWS Command-Line Interface (CLI) aws s3 sync command to do it for you!
"the fact is you can create 'folders' in S3"
No, you can't.
You can create an empty object that looks like a folder in the console, but it is still not a folder, it still has no meaning, it is still unnecessary, and if you delete it via the API, all the files you thought were "in" the folder will still be in the bucket. (If you delete it from the console, all the contents are deleted from the bucket, because the console explicitly deletes every object starting with that key prefix.)
The folder you are creating is not a container and cannot have anything inside it, because S3 does not have folders that are containers.
If I want to store a file cat.png and make it look like it's in the hat/ folder, you simply set the object key to hat/cat.png. This has exactly the same effect as observed in the console, whether or not the hat/ folder was explicitly created or not.
To so what you want, you simply build the desired object key for each object with string manipulation, including your common prefix ("folder name") and / delimiters. Any folder structure the / delimiters imply will be displayed in the console as a result.
I have a script that is intended to be run by multiple users on multiple computers, and they don't all have their Dropbox folders in their respective home directories. I'd hate to have to hard code paths in the script. I'd much rather figure out the path programatically.
Any suggestions welcome.
EDIT:
I am not using the Dropbox API in the script, the script simply reads files in a specific Dropbox folder shared between the users. The only thing I need is the path to the Dropbox folder, as I of course already know the relative path within the Dropbox file structure.
EDIT:
If it matters, I am using Windows 7.
I found the answer here. Setting s equal to the 2nd line in ~\AppData\Roaming\Dropbox\host.db and then decoding it with base64 gives the path.
def _get_appdata_path():
import ctypes
from ctypes import wintypes, windll
CSIDL_APPDATA = 26
_SHGetFolderPath = windll.shell32.SHGetFolderPathW
_SHGetFolderPath.argtypes = [wintypes.HWND,
ctypes.c_int,
wintypes.HANDLE,
wintypes.DWORD,
wintypes.LPCWSTR]
path_buf = wintypes.create_unicode_buffer(wintypes.MAX_PATH)
result = _SHGetFolderPath(0, CSIDL_APPDATA, 0, 0, path_buf)
return path_buf.value
def dropbox_home():
from platform import system
import base64
import os.path
_system = system()
if _system in ('Windows', 'cli'):
host_db_path = os.path.join(_get_appdata_path(),
'Dropbox',
'host.db')
elif _system in ('Linux', 'Darwin'):
host_db_path = os.path.expanduser('~'
'/.dropbox'
'/host.db')
else:
raise RuntimeError('Unknown system={}'
.format(_system))
if not os.path.exists(host_db_path):
raise RuntimeError("Config path={} doesn't exists"
.format(host_db_path))
with open(host_db_path, 'r') as f:
data = f.read().split()
return base64.b64decode(data[1])
There is an answer to this on Dropbox Help Center - How can I programmatically find the Dropbox folder paths?
Short version:
Use ~/.dropbox/info.json or %APPDATA%\Dropbox\info.json
Long version:
Access the valid %APPDATA% or %LOCALAPPDATA% location this way:
import os
from pathlib import Path
import json
try:
json_path = (Path(os.getenv('LOCALAPPDATA'))/'Dropbox'/'info.json').resolve()
except FileNotFoundError:
json_path = (Path(os.getenv('APPDATA'))/'Dropbox'/'info.json').resolve()
with open(str(json_path)) as f:
j = json.load(f)
personal_dbox_path = Path(j['personal']['path'])
business_dbox_path = Path(j['business']['path'])
You could search the file system using os.walk. The Dropbox folder is probably within the home directory of the user, so to save some time you could limit your search to that. Example:
import os
dropbox_folder = None
for dirname, dirnames, filenames in os.walk(os.path.expanduser('~')):
for subdirname in dirnames:
if(subdirname == 'Dropbox'):
dropbox_folder = os.path.join(dirname, subdirname)
break
if dropbox_folder:
break
# dropbox_folder now contains the full path to the Dropbox folder, or
# None if the folder wasn't found
Alternatively you could prompt the user for the Dropbox folder location, or make it configurable via a config file.
This adaptation based on J.F. Sebastian's suggestion works for me on Ubuntu:
os.path.expanduser('~/Dropbox')
And to actually set the working directory to be there:
os.chdir(os.path.expanduser('~/Dropbox'))
Note: answer is valid for Dropbox v2.8 and higher
Windows
jq -r ".personal.path" < %APPDATA%\Dropbox\info.json
This needs jq - JSON parser utility to be installed. If you are happy user of Chocolatey package manager, just run choco install jq before.
Linux
jq -r ".personal.path" < ~/.dropbox/info.json
Just similarly to Windows install jq using package manager of your distro.
Note: requires Dropbox >= 2.8
Dropbox now stores the paths in json format in a file called info.json. It is located in one of the two following locations:
%APPDATA%\Dropbox\info.json
%LOCALAPPDATA%\Dropbox\info.json
I can access the %APPDATA% environment variable in Python by os.environ['APPDATA'], however I check both that and os.environ['LOCALAPPDATA']. Then I convert the JSON into a dictionary and read the 'path' value under the appropriate Dropbox (business or personal).
Calling get_dropbox_location() from the code below will return the filepath of the business Dropbox, while get_dropbox_location('personal') will return the file path of the personal Dropbox.
import os
import json
def get_dropbox_location(account_type='business'):
"""
Returns a string of the filepath of the Dropbox for this user
:param account_type: str, 'business' or 'personal'
"""
info_path = _get_dropbox_info_path()
info_dict = _get_dictionary_from_path_to_json(info_path)
return _get_dropbox_path_from_dictionary(info_dict, account_type)
def _get_dropbox_info_path():
"""
Returns filepath of Dropbox file info.json
"""
path = _create_dropox_info_path('APPDATA')
if path:
return path
return _create_dropox_info_path('LOCALAPPDATA')
def _create_dropox_info_path(appdata_str):
r"""
Looks up the environment variable given by appdata_str and combines with \Dropbox\info.json
Then checks if the info.json exists at that path, and if so returns the filepath, otherwise
returns False
"""
path = os.path.join(os.environ[appdata_str], r'Dropbox\info.json')
if os.path.exists(path):
return path
return False
def _get_dictionary_from_path_to_json(info_path):
"""
Loads a json file and returns as a dictionary
"""
with open(info_path, 'r') as f:
text = f.read()
return json.loads(text)
def _get_dropbox_path_from_dictionary(info_dict, account_type):
"""
Returns the 'path' value under the account_type dictionary within the main dictionary
"""
return info_dict[account_type]['path']
This is a pure Python solution, unlike the other solution using info.json.
One option is you could go searching for the .dropbox.cache directory which (at least on Mac and Linux) is a hidden folder in the Dropbox directory.
I am fairly certain that Dropbox stores its preferences in an encrypted .dbx container, so extracting it using the same method that Dropbox uses is not trivial.
This should work on Win7. The use of getEnvironmentVariable("APPDATA") instead of os.getenv('APPDATA') supports Unicode filepaths -- see question titled Problems with umlauts in python appdata environvent variable.
import base64
import ctypes
import os
def getEnvironmentVariable(name):
""" read windows native unicode environment variables """
# (could just use os.environ dict in Python 3)
name = unicode(name) # make sure string argument is unicode
n = ctypes.windll.kernel32.GetEnvironmentVariableW(name, None, 0)
if not n:
return None
else:
buf = ctypes.create_unicode_buffer(u'\0'*n)
ctypes.windll.kernel32.GetEnvironmentVariableW(name, buf, n)
return buf.value
def getDropboxRoot():
# find the path for Dropbox's root watch folder from its sqlite host.db database.
# Dropbox stores its databases under the currently logged in user's %APPDATA% path.
# If you have installed multiple instances of dropbox under the same login this only finds the 1st one.
# Dropbox stores its databases under the currently logged in user's %APPDATA% path.
# usually "C:\Documents and Settings\<login_account>\Application Data"
sConfigFile = os.path.join(getEnvironmentVariable("APPDATA"),
'Dropbox', 'host.db')
# return null string if can't find or work database file.
if not os.path.exists(sConfigFile):
return None
# Dropbox Watch Folder Location is base64 encoded as the last line of the host.db file.
with open(sConfigFile) as dbxfile:
for sLine in dbxfile:
pass
# decode last line, path to dropbox watch folder with no trailing slash.
return base64.b64decode(sLine)
if __name__ == '__main__':
print getDropboxRoot()