I'm writing a script to upload files to Dropbox using python dropbox module (V2). The files will be uploaded in one go or via a session depending on the size. The relevant code is as follows:
with open(file, 'rb') as f:
try:
file_size = os.path.getsize(file)
chunk_size = 4*1024*1024
if file_size < chunk_size:
dbx.files_upload(f.read(), file_to, mode=dropbox.files.WriteMode.overwrite)
else:
session_start_result = dbx.files_upload_session_start(f.read(chunk_size))
cursor = dropbox.files.UploadSessionCursor(session_id=session_start_result.session_id, offset=f.tell())
commit = dropbox.files.CommitInfo(path=file_to)
while f.tell() < file_size:
if (file_size - f.tell()) <= chunk_size:
dbx.files_upload_session_finish(f.read(chunk_size), cursor, commit)
else:
dbx.files_upload_session_append_v2(f.read(chunk_size), cursor)
cursor.offset = f.tell()
However, this will raise an error if a the session option is used to upload a large file with the same name as one already in the Dropbox folder. For a small file upload, you can set the WriteMode, but I couldn't find any documentation on how to do this when using a session/cursor.
Any help or a nudge in the right direction would be much appreciated.
When using upload sessions, you can set the WriteMode on the dropbox.files.CommitInfo object, via the mode parameter. That should be a dropbox.files.WriteMode, just like in the small file scenario.
You then pass that CommitInfo to files_upload_session_finish like you're already doing.
Related
For a project, I'm trying to get an uploaded image file, stored in a bucket. I'm trying to have Python save a copy temporarily, just to perform a few tasks on this file (read, decode and give the decoded file back as JSON). After this is done, the temp file needs to be deleted.
I'm using Python 3.8, if that helps at all.
If you want some snippets of what I tried, I'm happy to provide :)
#edit
So far, I tried just downloading the file from the bucket, which works. But I can't seem to figure out how to temporarily save it to just decode (I got an API that will decode the image and get data from that file). This is the code for downloading
def download_file_from_bucket(blob_name, file_path, bucket_name):
try:
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(blob_name)
with open(file_path, 'wb') as f:
storage_client.download_blob_to_file(blob, f)
except Exception as e:
print(e)
return False
bucket_name = 'white-cards-with-qr'
download_file_from_bucket('My first Blob Image', os.path.join(os.getcwd(), 'file2.jpg'), bucket_name)
for object store in cloud environment, you can sign your object to give access for ones who don't have account for that object, you may read this for google cloud
You can use the tempfile library. This is a really basic snippet. You can also name the file or read it after writing it.
import tempfile
temp = tempfile.TemporaryFile()
try:
temp.write(blob)
finally:
temp.close()
I am trying to upload a zip folder to dropbox. The zip is a backup, with a custom name according to the current date time. The folder is correctly zipped, named and stored. Sadly, there is always an issue uploading to dropbox. I have tested a readme.txt using the same code, which works. I don’t understand where this is going wrong. Thanks for any help.
import dropbox
import os
import datetime
dt = ('{:%Y%m%d_%H%M}'.format(datetime.datetime.now()))
name = dt + "_pa.zip"
os.system("zip -r " + name + " /home/obliss")
class TransferData:
def __init__(self, access_token):
self.access_token = access_token
def upload_file(self, file_from, file_to):
dbx = dropbox.Dropbox(self.access_token)
with open(file_from, 'rb') as f:
dbx.files_upload(f.read(), file_to, mode=dropbox.files.WriteMode.overwrite)
access_token = "[hidden]"
file_from = "/home/olbliss/"+name
file_to = "/Work/Python Anywhere Backups/"+name
transferData = TransferData(access_token)
try:
transferData.upload_file(file_from, file_to)
except:
os.remove(name)
print('uploaded failed, '+name+' removed from /home/olbliss/')
try:
os.remove(name)
except:
pass
Failure message:
The 413 status code indicates that the payload was too large. The files_upload method only officially supports files up to 150 MB in size. You'll need to use upload_sessions for larger files.
Here's a basic example that uses the Dropbox Python SDK to upload a file to the Dropbox API from the local file as specified by file_path to the remote path as specified by dest_path. It also chooses whether or not to use an upload session based on the size of the file:
f = open(file_path)
file_size = os.path.getsize(file_path)
CHUNK_SIZE = 8 * 1024 * 1024
if file_size <= CHUNK_SIZE:
print dbx.files_upload(f.read(), dest_path)
else:
upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
offset=f.tell())
commit = dropbox.files.CommitInfo(path=dest_path)
while f.tell() <= file_size:
if ((file_size - f.tell()) <= CHUNK_SIZE):
print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
cursor,
commit)
break
else:
dbx.files_upload_session_append_v2(f.read(CHUNK_SIZE),
cursor)
cursor.offset = f.tell()
f.close()
Note: this should only serve as an example. It hasn't been extensively tested and doesn't implement error handling.
I am trying to upload a large zipped file to Dropbox (about 2-3GB) using Python and the Dropbox API v2. I am using the "chunked method" found here: dropbox API v2 upload large files using python. Here is the code again for convenience:
f = open(file_path)
file_size = os.path.getsize(file_path)
CHUNK_SIZE = 4 * 1024 * 1024
if file_size <= CHUNK_SIZE:
print dbx.files_upload(f, dest_path)
else:
upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
offset=f.tell())
commit = dropbox.files.CommitInfo(path=dest_path)
while f.tell() < file_size:
if ((file_size - f.tell()) <= CHUNK_SIZE):
print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
cursor,
commit)
else:
dbx.files_upload_session_append(f.read(CHUNK_SIZE),
cursor.session_id,
cursor.offset)
cursor.offset = f.tell()
However, when I run it I am getting a
dropbox.exceptions.APIError
as well as
UploadSessoionLookupError
and a
UploadSessionOffsetError
I think the error might be occurring at this line specifically:
dbx.files_upload_session_append(f.read(CHUNK_SIZE),
cursor.session_id,
cursor.offset)
I've tried swapping that for
dbx.files_upload_session_append_v2(
f.read(self.CHUNK_SIZE), cursor)
but that didn't work either. Any suggestions?
On windows, be sure to open using binary mode
f = open(file_path, 'rb')
f.read(chunk_size) and f.tell() were off.
from python docs
On Windows, tell() can return illegal values (after an fgets()) when reading files with Unix-style line-endings. Use binary mode (‘rb’) to circumvent this problem.
I am trying to perform a task to transfer files between two different FTP locations. And the simple goal is that I would want to specific file type from FTP Location A to FTP Location B for only last few hours using Python script.
I am using ftplib to perform the task and have put together below code.
So far the file transfer is working fine for single file defined in the from_sock variable, but I am hitting road block when I am wanting to loop through all files which were created within last 2 hours and copy them. So the script I have written is basically copying individual file but I want to I wan't to move all files with particular extension example *.jpg which were created within last 2 hours. I tired to use MDTM to find the file modification time but I am not able to implement in right way.
Any help on this is much appreciated. Below is the current code:
import ftplib
srcFTP = ftplib.FTP("test.com", "username", "pass")
srcFTP.cwd("/somefolder")
desFTP = ftplib.FTP("test2.com", "username", "pass")
desFTP.cwd("/")
from_Sock = srcFTP.transfercmd("RETR Test1.text")
to_Sock = desFTP.transfercmd("STOR test1.text")
state = 0
while 1:
block = from_Sock.recv(1024)
if len(block) == 0:
break
state += len(block)
while len(block) > 0:
sentlen = to_Sock.send(block)
block = block[sentlen:]
print state, "Total Bytes Transferred"
from_Sock.close()
to_Sock.close()
srcFTP.quit()
desFTP.quit()
Thanks,
DD
Here a short code that takes the path and uploads every file with an extension of .jpg via ftp. Its not exactly what you want but I stumbled on your answer and this might help you on your way.
import os
from ftplib import FTP
def ftpPush(filepathSource, filename, filepathDestination):
ftp = FTP(IP, username, password)
ftp.cwd(filepathDestination)
ftp.storlines("STOR "+filename, open(filepathSource+filename, 'r'))
ftp.quit()
path = '/some/path/'
for fileName in os.listdir(path):
if fileName.endswith(".jpg"):
ftpPush(filepathSource=path, filename=fileName, filepathDestination='/some/destination/')
The creation time of a file can be checked on an ftp server using this example.
fileName = "nameOfFile.txt"
modifiedTime = ftp.sendcmd('MDTM ' + fileName)
# successful response: '213 20120222090254'
ftp.quit()
Now you just need to check when the file that have been modified, download it if it is below you wished for threshold and then upload them to the other computer.
I'm using web.py for a small project and I have files I want the user to be able to access in /files directory on the server. I can't seem to find how to return a file on a GET request so I can't work how to do this.
Exactly want to do essentially is:
urls = ('/files/+', 'files')
class files:
def GET(self)
#RETURN SOME FILE
Is there a simple way to return a file from a GET request?
Playing around I came up with this webpy GET method:
def GET(self):
request = web.input( path=None )
getPath = request.path
if os.path.exists( getPath ):
getFile = file( getPath, 'rb' )
web.header('Content-type','application/octet-stream')
web.header('Content-transfer-encoding','base64')
return base64.standard_b64encode( getFile.read( ) )
else:
raise web.notfound( )
Other respondants are correct when they advise you consider carefully the security implications. In my case we will include code like this to an administrative web service that will be (should be!) available only within our internal LAN.
You can read the contents of a file and stream them down to the user, but I don't believe that a file handle is serializable.
It would seem to be a potential security hole to allow users to access and modify files on the server or to copy files down to their own machine. I think you should reassess what you're trying to accomplish.
This is how I do it by using generator and not reading the whole file into memory:
web.header("Content-Disposition", "attachment; filename=%s" % doc.filename)
web.header("Content-Type", doc.filetype)
web.header("Transfer-Encoding","chunked")
f = open(os.path.join(config.upload_dir, doc.path, doc.filename), "rb")
while 1:
buf = f.read(1024 * 8)
if not buf:
break
yield buf