def fetch(self, query, secret):
if secret != self.secret: raise AccessDenied
result = self.query(query)
f = open(join(self.dirname, query), 'w')
f.write(result)
f.close()
return 0
I am trying to get peers fetch files from one host to another using this method(peer-to-peer program).
This method only takes text as it is opening the file and writing the contents to f.
How can I copy .pdf , .mpeg, jpeg files copied/downloaded to the peers directory!
As long as your query method supports binary, try 'wb' instead of 'w'.
To write binary data you should open the file using the file mode 'wb' (write binary). i.e.:
f = open(join(self.dirname, query), 'wb')
Related
I have a zip file that I receive when the user uploads a file. The zip essentially contains a json file which I want to read and process without having to create the zip file first, then unzipping it and then reading the content of the inner file.
Currently I only the longer process which is something like below
import json
import zipfile
#csrf_exempt
def get_zip(request):
try:
if request.method == "POST":
try:
client_file = request.FILES['file']
file_path = "/some/path/"
# first dump the zip file to a directory
with open(file_path + '%s' % client_file.name, 'wb+') as dest:
for chunk in client_file.chunks():
dest.write(chunk)
# unzip the zip file to the same directory
with zipfile.ZipFile(file_path + client_file.name, 'r') as zip_ref:
zip_ref.extractall(file_path)
# at this point we get a json file from the zip say `test.json`
# read the json file content
with open(file_path + "test.json", "r") as fo:
json_content = json.load(fo)
doSomething(json_content)
return HttpResponse(0)
except Exception as e:
return HttpResponse(1)
As you can see, this involves 3 steps to finally get the content from the zip file into memory. What I want is get the content of the zip file and load directly into memory.
I did find some similar questions in stack overflow like this one https://stackoverflow.com/a/2463819 . But I am not sure at what point do I invoke this operation mentioned in the post
How can I achieve this?
Note: I am using django in backend.
There will always be one json file in the zip.
From what I understand, what #jason is trying to say here is to first open a zipFile just like you have done here with zipfile.ZipFile(file_path + client_file.name, 'r') as zip_ref:.
class zipfile.ZipFile(file[, mode[, compression[, allowZip64]]])
Open a ZIP file, where file can be either a path to a file (a string) or a file-like object.
And then use BytesIO read in the bytes of a file-like object. But from above you are reading in r mode and not rb mode. So change it as follows.
with open(filename, 'rb') as file_data:
bytes_content = file_data.read()
file_like_object = io.BytesIO(bytes_content)
zipfile_ob = zipfile.ZipFile(file_like_object)
Now zipfile_ob can be accessed from memory.
The first argument to zipfile.ZipFile() can be a file object rather than a pathname. I think the Django UploadedFile object supports this use, so you can read directly from that rather than having to copy into a file.
You can also open the file directly from the zip archive rather than extracting that into a file.
import json
import zipfile
#csrf_exempt
def get_zip(request):
try:
if request.method == "POST":
try:
client_file = request.FILES['file']
# unzip the zip file to the same directory
with zipfile.ZipFile(client_file, 'r') as zip_ref:
first = zip_ref.infolist()[0]
with zip_ref.open(first, "r") as fo:
json_content = json.load(fo)
doSomething(json_content)
return HttpResponse(0)
except Exception as e:
return HttpResponse(1)
I am working on a project where I have to upload a file from file storage (via web form) to MongoDB. In order to achieve this, I need to open the file in "rb" mode, then encode the file and finally upload to MongoDb. I am stuck when opening the file "rb" mode.
if form.validate():
for inFile in request.files.getlist("file"):
connection = pymongo.MongoClient()
db = connection.test
uploads = db.uploads
with open(inFile, "rb") as fin:
f = fin.read()
encoded = Binary(f,0)
try:
uploads.insert({"binFile": encoded})
check = True
except Exception as e:
self.errorList.append("Document upload is unsuccessful"+e)
check = False
The above code is throwing TypeError: coercing to Unicode: need string or buffer, FileStorage found in the open step, i.e. this line:
with open(inFile, "rb") as fin:
Is there a way I can change my code to make it work?
Thanks in advance
The FileStorage object is already file-like so you can use as a file. You don't need to use open on it, just call inFile.read().
If this doesn't work for you for some reason, you can save the file to disk first using inFile.save() and open it from there.
Reference: http://werkzeug.pocoo.org/docs/0.11/datastructures/#werkzeug.datastructures.FileStorage
I am tying to write a binary data to a zip file.
The below works but if I try to add a .zip as a file extension to "check" in the variable x nothing is written to the file. I am stuck manually adding .zip
urla = "some url"
tok = "some token"
pp = {"token": tok}
t = requests.get(urla, params=pp)
b = t.content
x = r"C:\temp" + "\check"
z = 'C:\temp\checks.zip'
with open(x, "wb") as work:
work.write(b)
In order to have the correct extension appended to the file I attempted to use the module ZipFile
with ZipFile(x, "wb") as work:
work.write(b)
but get a RuntimeError:
RuntimeError: ZipFile() requires mode "r", "w", or "a"
If I remove the b flag an empty zipfile is created and I get a TypeError:
TypeError: must be encoded string without NULL bytes, not str
I also tried but it creates a corrupted zipfile.
os.rename(x, z )
How do you write binary data to a zip file.
I converted a zip file into binary data and was able to regenerate the zip file in the following way:
bin_data=b"\x0\x12" #Whatever binary data you have store in a variable
binary_file_path = 'file.zip' #Name for new zip file you want to regenerate
with open(binary_file_path, 'wb') as f:
f.write(bin_data)
Use the writestr method.
import zipfile
z = zipfile.ZipFile(path, 'w')
z.writestr(filename, bytes)
z.close()
zipfile.ZipFile.writestr
You don't write the data directly to the zip file. You write it to a file, then you write the filepath to the zip file.
binary_file_path = '/path/to/binary/file.ext'
with open(binary_file_path, 'wb') as f:
f.write('BINARYDATA')
zip_file_path = '/path/to/zip/file.zip'
with ZipFile(zip_file_path, 'w') as zip_file:
zip_file.write(binary_file_path)
The computer is toying with me, I know it!
I am creating a zip folder in Python. The individual files are generated in memory and then the whole thing is zipped and saved to a file. I am allowed to add 9 files to the zip. I am allowed to add 11 files to the zip. But 10, no, not 10 files. The zip file IS saved to my computer, but I'm not allowed to open it; Windows says that the compressed zipped folder is invalid.
I use the code below, which I got from another stackoverflow question. It appends 10 files and saves the zipped folder. When I click on the folder, I cannot extract it. BUT, remove one of the appends() and it's fine. Or, add another append and it works!
What am I missing here? How can I make this work every time?
imz = InMemoryZip()
imz.append("1a.txt", "a").append("2a.txt", "a").append("3a.txt", "a").append("4a.txt", "a").append("5a.txt", "a").append("6a.txt", "a").append("7a.txt", "a").append("8a.txt", "a").append("9a.txt", "a").append("10a.txt", "a")
imz.writetofile("C:/path/test.zip")
import zipfile
import StringIO
class InMemoryZip(object):
def __init__(self):
# Create the in-memory file-like object
self.in_memory_zip = StringIO.StringIO()
def append(self, filename_in_zip, file_contents):
'''Appends a file with name filename_in_zip and contents of
file_contents to the in-memory zip.'''
# Get a handle to the in-memory zip in append mode
zf = zipfile.ZipFile(self.in_memory_zip, "a", zipfile.ZIP_DEFLATED, False)
# Write the file to the in-memory zip
zf.writestr(filename_in_zip, file_contents)
# Mark the files as having been created on Windows so that
# Unix permissions are not inferred as 0000
for zfile in zf.filelist:
zfile.create_system = 0
return self
def read(self):
'''Returns a string with the contents of the in-memory zip.'''
self.in_memory_zip.seek(0)
return self.in_memory_zip.read()
def writetofile(self, filename):
'''Writes the in-memory zip to a file.'''
f = file(filename, "w")
f.write(self.read())
f.close()
You should use the 'wb' mode when creating the file you are saving to the file system. This will ensure that the file is written in binary.
Otherwise, any time a newline (\n) character happens to be encountered in the zip file python will replace it to match the windows line ending (\r\n). The reason 10 files is a problem is that 10 happens to be the code for \n.
So your write function should look like this:
def writetofile(self, filename):
'''Writes the in-memory zip to a file.'''
f = file(filename, 'wb')
f.write(self.read())
f.close()
This should fix your problem and work for the files in your example. Although, in your case you might find it easier to write the zip file directly to the file system like this code which includes some of the comments from above:
import StringIO
import zipfile
class ZipCreator:
buffer = None
def __init__(self, fileName=None):
if fileName:
self.zipFile = zipfile.ZipFile(fileName, 'w', zipfile.ZIP_DEFLATED, False)
return
self.buffer = StringIO.StringIO()
self.zipFile = zipfile.ZipFile(self.buffer, 'w', zipfile.ZIP_DEFLATED, False)
def addToZipFromFileSystem(self, filePath, filenameInZip):
self.zipFile.write(filePath, filenameInZip)
def addToZipFromMemory(self, filenameInZip, fileContents):
self.zipFile.writestr(filenameInZip, fileContents)
for zipFile in self.zipFile.filelist:
zipFile.create_system = 0
def write(self, fileName):
if not self.buffer: # If the buffer was not initialized the file is written by the ZipFile
self.zipFile.close()
return
f = file(fileName, 'wb')
f.write(self.buffer.getvalue())
f.close()
# Use File Handle
zipCreator = ZipCreator('C:/path/test.zip')
# Use Memory Buffer
# zipCreator = ZipCreator()
for i in range(1, 10):
zipCreator.addToZipFromMemory('test/%sa.txt' % i, 'a')
zipCreator.write('C:/path/test.zip')
Ideally, you would probably use separate classes for an in-memory zip and a zip that is tied to the file system from the beginning. I have also seem some issues with the in-memory zip when folders are added which are difficult to recreate and which I am still trying to track down.
Using BottlePy, I use the following code to upload a file and write it to disk :
upload = request.files.get('upload')
raw = upload.file.read()
filename = upload.filename
with open(filename, 'w') as f:
f.write(raw)
return "You uploaded %s (%d bytes)." % (filename, len(raw))
It returns the proper amount of bytes every single time.
The upload works fine for file like .txt, .php, .css ...
But it results in a corrupted file for other files like .jpg, .png, .pdf, .xls ...
I tried to change the open() function
with open(filename, 'wb') as f:
It returns the following error:
TypeError('must be bytes or buffer, not str',)
I guess its an issue related to binary files ?
Is there something to install on top of Python to run upload for any file type ?
Update
Just to be sure, as pointed out by #thkang I tried to code this using the dev version of bottlepy and the built-in method .save()
upload = request.files.get('upload')
upload.save(upload.filename)
It returns the exact same Exception error
TypeError('must be bytes or buffer, not str',)
Update 2
Here the final code which "works" (and dont pop the error TypeError('must be bytes or buffer, not str',)
upload = request.files.get('upload')
raw = upload.file.read().encode()
filename = upload.filename
with open(filename, 'wb') as f:
f.write(raw)
Unfortunately, the result is the same : every .txt file works fine, but other files like .jpg, .pdf ... are corrupted
I've also noticed that those file (the corrupted one) have a larger size than the orginal (before upload)
This binary thing must be the issue with Python 3x
Note :
I use python 3.1.3
I use BottlePy 0.11.6 (raw bottle.py file, no 2to3 on it or anything)
Try this:
upload = request.files.get('upload')
with open(upload.file, "rb") as f1:
raw = f1.read()
filename = upload.filename
with open(filename, 'wb') as f:
f.write(raw)
return "You uploaded %s (%d bytes)." % (filename, len(raw))
Update
Try value:
# Get a cgi.FieldStorage object
upload = request.files.get('upload')
# Get the data
raw = upload.value;
# Write to file
filename = upload.filename
with open(filename, 'wb') as f:
f.write(raw)
return "You uploaded %s (%d bytes)." % (filename, len(raw))
Update 2
See this thread, it seems to do same as what you are trying...
# Test if the file was uploaded
if fileitem.filename:
# strip leading path from file name to avoid directory traversal attacks
fn = os.path.basename(fileitem.filename)
open('files/' + fn, 'wb').write(fileitem.file.read())
message = 'The file "' + fn + '" was uploaded successfully'
else:
message = 'No file was uploaded'
In Python 3x all strings are now unicode, so you need to convert the read() function used in this file upload code.
The read() function returns a unicode string aswell, which you can convert into proper bytes via encode() function
Use the code contained in my first question, and replace the line
raw = upload.file.read()
with
raw = upload.file.read().encode('ISO-8859-1')
That's all ;)
Further reading : http://python3porting.com/problems.html