I have dilemma.. I'm uploading files both in scribd store and blobstore using tipfy as framework.
I have webform with action is not created by blobstore.create_upload_url (i'm just using url_for('myhandler')). I did it because if i'm using blobstore handler the POST response parsed and I cannot use normal python-scribd api to upload file into scribd store.
Now I have working scribd saver:
class UploadScribdHandler(RequestHandler, BlobstoreUploadMixin):
def post(self):
uploaded_file = self.request.files.get('upload_file')
fname = uploaded_file.filename.strip()
try:
self.post_to_scribd(uploaded_file, fname)
except Exception, e:
# ... get the exception message and do something with it
msg = e.message
# ...
# reset the stream to zero (beginning) so the file can be read again
uploaded_file.seek(0)
#removed try-except to see debug info in browser window
# Create the file
file_name = files.blobstore.create(_blobinfo_uploaded_filename=fname)
# Open the file and write to it
with files.open(file_name, 'a') as f:
f.write(uploaded_file.read())
# Finalize the file. Do this before attempting to read it.
files.finalize(file_name)
# Get the file's blob key
blob_key = files.blobstore.get_blob_key(file_name)
return Response('done')
def post_to_scribd(self, uploaded_file, fname):
errmsg =''
uploaded_file = self.request.files.get('upload_file')
fname = uploaded_file.filename.strip()
fext = fname[fname.rfind('.')+1:].lower()
if (fext not in ALLOWED_EXTENSION):
raise Exception('This file type does not allowed to be uploaded\n')
if SCRIBD_ENABLED:
doc_title = self.request.form.get('title')
doc_description = self.request.form.get('description')
doc_tags = self.request.form.get('tags')
try:
document = scribd.api_user.upload(uploaded_file, fname, access='private')
#while document.get_conversion_status() != 'DONE':
# time.sleep(2)
if not doc_title:
document.title = fname[:fname.rfind('.')]
else:
document.title = doc_title
if not doc_description:
document.description = 'This document was uploaded at ' + str(datetime.datetime.now()) +'\n'
else:
document.description = doc_description
document.tags = doc_tags
document.save()
except scribd.ResponseError, err:
raise Exception('Scribd failed: error code:%d, error message: %s\n' % (err.errno, err.strerror))
except scribd.NotReadyError, err:
raise Exception('Scribd failed: error code:%d, error message: %s\n' % (err.errno, err.strerror))
except:
raise Exception('something wrong exception')
As you can see it also saves file into blobstore.. But If i'm uploading big file (i.e. 5Mb) I'm receiving
RequestTooLargeError: The request to API call file.Append() was too large.
Request: docs.upload(access='private', doc_type='pdf', file=('PK\x03\x04\n\x00\x00\x00\x00\x00"\x01\x10=\x00\x00(...)', 'test.pdf'))
How can I fix it?
Thanks!
You need to make multiple, smaller calls to the file API, for instance like this:
with files.open(file_name, 'a') as f:
data = uploaded_file.read(65536)
while data:
f.write(data)
data = uploaded_file.read(65536)
Note that the payload size limit on regular requests to App Engine apps is 10MB; if you want to upload larger files, you'll need to use the regular blobstore upload mechanism.
finally i found solution.
Nick Johneson's answer occurred attribute error because uploaded_file is treated as string.
string didn't have read() method.
Cause string doesn't have method read(), i spliced file string and write it just like he wrote.
class UploadRankingHandler(webapp.RequestHandler):
def post(self):
fish_image_file = self.request.get('file')
file_name = files.blobstore.create(mime_type='image/png', _blobinfo_uploaded_filename="testfilename.png")
file_str_list = splitCount(fish_image_file,65520)
with files.open(file_name, 'a') as f:
for line in file_str_list:
f.write(line)
you can check about splitCount(). here
http://www.bdhwan.com/entry/gaewritebigfile
Related
I'm using two JSON files, one for storing and loading device variables and another one for mqtt infos. I'm using a load_config function to load the correct file and then load it as JSON. When the file exists, it works without any problem, but when the file is not existing, it throws a file not found error, obviously. but My function contains an exception block to handle this by creating the file, but it isn't called. Here's my code for the function:
def load_config(config_path):
with open(config_path) as f: #Config
try:
return json.load(f)
except OSError:
print("file not there, creating it")
open(config_path, "w")
except json.JSONDecodeError:
return {}
f.close()
I call that function like this:
DEVICE_PATH = 'config.json'
MQTT_PATH = 'mqtt.json'
conf = load_config(DEVICE_PATH) #load device config
mqtt_conf = load_config(MQTT_PATH) #load mqtt config
mqtt_broker_ip = mqtt_conf['ip'] #setup mqtt
mqtt_broker_port = mqtt_conf['port']
mqtt_user = mqtt_conf['username']
mqtt_pass = mqtt_conf['password']
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(mqtt_user, password=mqtt_pass)
client.connect(mqtt_broker_ip, mqtt_broker_port, keepalive = 60, bind_address="" )
what am I doing wrong? When I open the file directly with the load_config via with open(config_path, "a") as f: everything in it gets deleted, with x it just throws an exception if the file exists and with w, it gets also overwritten.
What you are trying to accomplish is already an open() built-in functionality.
Just skip the whole file existence check and load the JSON in w+ mode:
with open("file.json", "w+") as f:
try:
data = json.load(f)
except JSONDecodeError:
data = {}
w+ opens any file in read and write mode and creates the filename if it doesn't exist.
Keep in mind that dumping any data with this mode will entirely overwrite any existing file's content.
Just as a side note, you might need to explore some basic knowledge about file processing, to avoid being stuck with a similar issue very soon.
Since I had a logic error, the exception IOError would never been raised. I opened the file and tried to load into json. Now I just check beforehands, if the file not exists, and create it.
def load_config(config_path):
if not os.path.isfile(config_path):
open(config_path, "w+")
with open(config_path) as f: #Config
try:
return json.load(f)
except json.JSONDecodeError:
return {}
I am attempting to make a function that records a few strings to a logfile on a server. For some reason, this function takes forever to run through..about 20 seconds before it will return the Exception. I think it's the try: statement with the file open command.
Any ideas how I can do this correctly?
def writeUserRecord():
""" Given a path, logs user name, fetch version, and time"""
global fetchVersion
global fetchHome
filename = 'users.log'
logFile = os.path.normpath(os.path.join(fetchHome, filename))
timeStamp = str(datetime.datetime.now()).split('.')[0]
userID = getpass.getuser()
try:
file = open(logFile, 'a')
file.write('{} {} {}'.format(userID, timeStamp, fetchVersion))
file.close()
except IOError:
print('Error Accessing Log File')
pass
I have converted on Excel(with 2 tabs) to HTML. Now I want to display this generated HTML on a webpage with Django code. But I am not able to get the entire data in my webpage.
Following is my Django code.
#api_view(['GET'])
def download_y9cfile1(request, file_name):
filePath = CommonUtils.get_absolute_file_path('app','static','generated','HTML', file_name)
logger.debug("File name is %s" % filePath )
try:
relevantFile = open(filePath,'rb')
response = HttpResponse((relevantFile), content_type='text/html')
response['Content-Disposition'] = 'inline; filename="%s"' %file_name
except IOError:
logger.exception("File doesn't exist")
return HttpResponse("File doesn't exist", status=500)
return response
I think the issue is that Django is not able to read the supporting CSS file for HTML.
Try passing the file contents to HttpResponse instead of the file handle:
response = HttpResponse(relevantFile.read(), content_type='text/html')
This is my code for uploading image that I have defined in a class based view,
def _handle_uploaded_file(self, request):
folder = settings.MEDIA_ROOT
uploaded_filename = request.FILES['img_fl'].name
BASE_PATH ='/home/admin1/Desktop/virtualenv-12.1.1/mapfied'
# create the folder if it doesn't exist.
try:
os.mkdir(os.path.join(BASE_PATH, folder))
except Exception as e:
pass
# save the uploaded file inside that folder.
full_filename = os.path.join(BASE_PATH, folder, uploaded_filename)
fd = open(full_filename, 'wb')
file_content = ContentFile( request.FILES['img_fl'].read() )
try:
for chunk in file_content.chunks():
fout.write(chunk)
fout.close()
html = "<html><body>SAVED</body></html>"
print(html)
except Exception as e:
print(e)
The image file is getting saved to correct location with name, but it is corrupted.I am unable to find the exact reason for this , Am I doing something wrong here?
This is what I had from a previous project for writing upload files to disk:
def view_handling_function(request):
for key, value in request.FILES.iteritems():
full_path = ...
save_uploadfile_to_disk(full_path, value)
def save_uploadfile_to_disk(full_path, file):
with open(full_path, 'w+') as destination:
for chunk in file.chunks():
destination.write(chunk)
I think since you're looking to write a binary upload you need to open the file with writable binary mode which is actually wb+.
You could also tidy up a bit by using the 'with' keyword; see Django example here.
Sidenote: if you're persisting the file as a FileField (or a derived class) you could just provide the 'upload_to' function that returns the full path and file name for where you'd like to store the file. That'll let the framework take care of the file io for you.
I am currently setting up a website where I get a file uploaded from the user , do some processing on it and provide a link for the user to download the processed file from. I presently want to provide a path to the file on my local system, I am new to web2py, and am having trouble doing this.
Could someone please help me do this?
Regards
see this link for some hint: webpy: how to stream files , and may be add some code like this:
BUF_SIZE = 262144
class download:
def GET(self):
file_name = # get from url
file_path = os.path.join('/path to your file', file_name)
f = None
try:
f = open(file_path, "rb")
webpy.header('Content-Type','application/octet-stream')
webpy.header('Content-disposition', 'attachment; filename=%s' % file_name)
while True:
c = f.read(BUF_SIZE)
if c:
yield c
else:
break
except Exception, e:
# throw 403 or 500 or just leave it
pass
finally:
if f:
f.close()