Develop Reference

Python - Decoding an image from json

Working with an api and the requests library.
Background info: creating a program to interop with an sms messaging platform. All responses from the api are provided in json. When working with the endpoint to obtain sms attachments, the response is in json and is basically a large block of encoded text.
I'm able to send the proper request to obtain the response I want, but I don't know what to do with the response.
How can I decode the json and work with the image file?
The complete response is too long to post here, but I imagine anything important is included in the beginning... I'm just unsure what to do with it.
b'{"id":1067442,"friendlyName":"attachment0","md5":"4dCsb2PEljAqYY1JBl3FXA==","contentType":"image/jpeg","size":1017241,"createdDate":1582649116340,"updatedDate":1582649116340,"thumbnailBase64":"/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCACWAJYDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDu1AwB29B/9al4BPr7f5/Gk7Dn86Xt/niuY1HZ5HApcZ6ikXpxxTvccUhABwD1/rUkZ2nHUUwc07GRkUrgTBfTp7U9QB0pEyyg96cxSNGkdwqqCxZjwB6+3elcLmfq19aaRCt/dSMqAGMqD97jI49eP1rz2bUdc8YGR7eaLS9EVtpup22ofYZxvP6VieMfFP8AbGrMu9RaRnbBEcksvqR2z/8AWrkpdRS7vIobi7mlSFdsUKnIX2FdEY6ak8x6zp/hDwhHEJbrUotQY9ZJbsBc+wUgfzrQi0XwUp+SHSH+ro38zXlkdzEBuYiOIDqzYAqvcahaO4W3thcTngNInC/nT5Q5j2iLQfD8ozb6fprD1jhQ4/IVTvfBGkz5aGFYj6xqB+nSvJrU3EUgeGcxSA53wHaQfwrttL+J/wBhtI7fWrZ5pFYoZ4mA3Y9Qcc/jScRqRHqHgu7sQ09oInRPmYL8mQPUdDVeLXWh037HJtLbt0cj9AOhUiux/wCEx0XUtFu5rS4/eLEcROMNk8DHY8ntXO+B9PXUb2+nlQNBHGIfmGQWY5P8h+dS1pqProUrfVBuCGOEjHy+W+P0NU9KeeDVL2a6sLiO2kHlrKUyD................

First parse the json string:
import json
response = b'{"id":1067442,"friendlyName":"attachment0","md5":"4dCsb2PEljAqYY1JBl3FXA==","contentType":"image/jpeg","size":1017241,"createdDate":1582649116340,"updatedDate":1582649116340,"thumbnailBase64":"/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCACWAJYDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDu1AwB29B/9al4BPr7f5/Gk7Dn86Xt/niuY1HZ5HApcZ6ikXpxxTvccUhABwD1/rUkZ2nHUUwc07GRkUrgTBfTp7U9QB0pEyyg96cxSNGkdwqqCxZjwB6+3elcLmfq19aaRCt/dSMqAGMqD97jI49eP1rz2bUdc8YGR7eaLS9EVtpup22ofYZxvP6VieMfFP8AbGrMu9RaRnbBEcksvqR2z/8AWrkpdRS7vIobi7mlSFdsUKnIX2FdEY6ak8x6zp/hDwhHEJbrUotQY9ZJbsBc+wUgfzrQi0XwUp+SHSH+ro38zXlkdzEBuYiOIDqzYAqvcahaO4W3thcTngNInC/nT5Q5j2iLQfD8ozb6fprD1jhQ4/IVTvfBGkz5aGFYj6xqB+nSvJrU3EUgeGcxSA53wHaQfwrttL+J/wBhtI7fWrZ5pFYoZ4mA3Y9Qcc/jScRqRHqHgu7sQ09oInRPmYL8mQPUdDVeLXWh037HJtLbt0cj9AOhUiux/wCEx0XUtFu5rS4/eLEcROMNk8DHY8ntXO+B9PXUb2+nlQNBHGIfmGQWY5P8h+dS1pqProUrfVBuCGOEjHy+W+P0NU9KeeDVL2a6sLiO2kHlrKUyD................"}'
response_parsed = json.loads(response)
response_parsed['contentType'] being image/jpeg hints to us that the format of the image is jpeg.
You didn't provide us with the actual image data, just the beginning of the thumbnail of the image which is stored in thumbnailBase64.
The thumbnail is encoded in base64. We can decode the base64 string into bytes using base64.b64decode:
from base64 import b64decode
thumbnail_bytes = b64decode(response_parsed['thumbnailBase64'])
Now that we have the thumbnail bytes we can save it to a file and view it like any regular image file:
with open(r'thumbnail.jpg', 'wb') as x:
x.write(thumbnail_bytes)
The actual image is probably in a JSON field called "image" or "imageData" or something similar. You did not include it in your question so there's no way to be sure.
This results in:
Obviously it's corrupt because you only included the beginning of the thumbnail.

Something like that could do the trick
import json, base64
resp = b'{"id":1067442,"friendlyName":"attachment0","md5":"4dCsb2PEljAqYY1JBl3FXA==","contentType":"image/jpeg","size":1017241,"createdDate":1582649116340,"updatedDate":1582649116340,"thumbnailBase64":"/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgG...'
resp_dict = json.loads(resp)
with open("img.jpeg", "wb") as fp:
content = base64.b64decode(resp_dict['thumbnailBase64'])
fp.write(content)
In addition, keep in mind that base64 value can be used directly to print an image into a HTML page
<div>
<p>Taken from wikpedia</p>
<img src="data:image/jpeg;base64, iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot" />
</div>
In all case, you will need to check the content-type to correctly interpret the data you received. This is not always jpeg, but may be png, gif, tiff, etc.

How to build an image from an str containing the binary data?

I have a byte string, and trying to convert it into a file but I keep getting an error:
OS Error cannot identify image file <_io.BytesIO object at 0x03BBD330>
The code that I've implemented is below:
def test_view(request):
with open("test_json.json") as imageFile:
b=bytearray(json.load(imageFile)["IMAGE_DATA"], 'utf8')
print(b)
image = Image.open(io.BytesIO(b))
response = HttpResponse(content_type="image/jpeg")
image.save(response, "JPEG")
return response
The byte string is read from a json file, it looks like this, but it is much longer, I've just copied some characters.
"b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x01\\x00`\\x00`\\x00\\x00\\xff\\xdb\\x00C\\x00\\x05\\x04\\x04\\x04\\x04\\x03\\x05\\x04\\x04\\x04\\x06\\x05\\x05\\x06\\x08\\r\\x08\\x08\\x07\\x07\\x08\\x10\\x0b\\x0c\\t\\r\\x13\\x10\\x14\\x13\\x12\\x10\\x12\\x12\\x14\\x17\\x1d\\x19\\x14\\x16\\x1c\\x16\\x12\\x12\\x1a#\\x1a\\x1c\\x1e\\x1f!!!\\x14\\x19$\\'$ &\\x1d ! \\xff\\xdb\\x00C\\x01\\x05\\x06\\x06\\x08\\x07\\x08\\x0f\\x08\\x08\\x0f \\x15\\x12\\x15 \\xff\\xc0\\x00\\x11\\x08\\x01&\\x02q\\x03\\x01\"\\x00\\x02\\x11\\x01\\x03\\x11\\x01\\xff\\xc4\\x00\\x1f\\x00\\x00\\x01\\x05\\x01\\x01\\x01\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07\\x08\\t\\n\\x0b\\xff\\xc4\\x00\\xb5\\x10\\x00\\x02\\x01\\x03\\x03\\x02\\x04\\x03\\x05\\x05\\x04\\x04\\x00\\x00\\x01}\\x01\\x02\\x03\\x00\\x04\\x11\\x05\\x12!1A\\x06\\x13Qa\\x07\"q\\x142\\x81\\x91\\xa1\\x08#B\\xb1\\xc1\\x15R\\xd1\\xf0$3br\\x82\\t\\n\\x16\\x17\\x18\\x19\\x1a%&\\'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\\x83\\x84\\x85\\x86\\x87\\x88\\x89\\x8a\\x92\\x93\\x94\\x95\\x96\\x97\\x98\\x99\\x9a\\xa2\\xa3\\xa4\\xa5\\xa6\\xa7\\xa8\\xa9\\xaa\\xb2\\xb3\\xb4\\xb5\\xb6\\xb7\\xb8\\xb9\\xba\\xc2\\xc3\\xc4\\xc5\\xc6\\xc7\\xc8\\xc9\\xca\\xd2\\xd3\\xd4\\xd5\\xd6\\xd7\\xd8\\xd9\\xda\\xe1\\xe2\\xe3\\xe4\\xe5\\xe6\\xe7\\xe8\\xe9\\xea\\xf1\\xf2\\xf3\\xf4\\xf5\\xf6\\xf7\\xf8\\xf9\\xfa\\xff\\xc4\\x00\\x1f\\x01\\x00\\x03\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07\\x08\\t\\n\\x0b\\xff\\xc4\\x00\\xb5\\x11\\x00\\x02\\x01\\x02\\x04\\x04\\x03\\x04\\x07\\x05\\x04\\x04\\x00\\x01\\x02w\\x00\\x01\\x02\\x03\\x11\\x04\\x05!1\\x06\\x12AQ\\x07aq\\x13\"2\\x81\\x08\\x14B\\x91\\xa1\\xb1\\xc1\\t#3R\\xf0\\x15br\\xd1\\n\\x16$4\\xe1%\\xf1\\x17\\x18\\x19\\x1a&\\'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\\x82\\x83\\x84\\x85\\x86\\x87\\x88\\x89\\x8a\\x92\\x93\\x94\\x95\\x96\\x97\\x98\\x99'"

I am not entirely sure what exactly is the best solution to this issue because the problem you're running into is that json.load(imageFile)["IMAGE_DATA"] is going to return "b'bytes'", which when it converts to bytearray, you're doing bytearray("b\'bytes\'"), which is wrong.
Also this behavior is specific between JSON parsers, and different JSON parsers might interpret differently
Either strip the b'' that's wrapping your bytestring, or do an ast.literal_eval()
def test_view(request):
with open("test_json.json") as imageFile:
b = bytearray(
ast.literal_eval(
json.load(imageFile)["IMAGE_DATA"], 'utf8')
print(b)
image = Image.open(io.BytesIO(b))
response = HttpResponse(content_type="image/jpeg")
image.save(response, "JPEG")
return response

Saving a json request as an Image in Flask

I have a flask get request like this
content = {"vname":"","myPhoto":{"fieldname":"myPhoto","originalname":"flower-purple-lical-blosso.jpg","encoding":"7bit","mimetype":"image/jpeg","buffer":{"type":"Buffer","data":[255,216,255,224,0,...]}
My image file is with the key data.
data = content['myphoto']['buffer']['data']
I am unable to save it as a jpeg file.
I am not sure how to decode this object as an image as its currently a list.

If I correctly understood your question, it could be done like this:
#!/usr/bin/env python3
content = {"vname":"","myPhoto":{"fieldname":"myPhoto","originalname":"flower-purple-lical-blosso.jpg","encoding":"7bit","mimetype":"image/jpeg","buffer":{"type":"Buffer","data":[255,216,255,224,0,20]}}}
data = content['myPhoto']['buffer']['data']
str_data = ''.join(chr(d) for d in data) # build string using list comprehension
bytes_data = str_data.encode() # build bytes array from string
with open('output.jpg', 'wb') as f: # open file for writing bytes
f.write(bytes_data) # write bytes array to file
Code of course is not perfect and could be used as starting point.

How to store Gzipped content in Aerospike (cache)?

I'm using aerospike python client here. I don't find any problem while storing plain string in cache because when i retrieve the key, I get the same string I saved when I do conn.get(key)
import aerospike
CONN_AEROSPIKE_CONFIG = {'hosts': [('127.0.0.1', 3000)]}
conn = aerospike.client(CONN_AEROSPIKE_CONFIG).connect()
key = ('namspace_name', 'set_name', 'key_name')
value = "some big string"
conn.put(key, {'value': value})
If I want to save gzipped content in the place of value, I don't find any error but I'm unable to get back the exact content.
from gzip import GzipFile
from io import BytesIO
def compress_string(s):
zbuf = BytesIO()
with GzipFile(mode='wb', compresslevel=6, fileobj=zbuf, mtime=0) as zfile:
zfile.write(s)
return zbuf.getvalue()
put_value = compress_string(value)
conn.put(key, {'value': put_value})
_, _, get_value = conn.get(key)
I checked printing values of put_value, get_value. They don't match, and I need gzipped content, because my content exceeds 1MB and I need gzipped content only. Please guide me where I'm doing wrong.
I know that we can break the content into smaller chunks for storing and concatenate after getting them, but again I need to Gzip the content after getting the data. So I thought why not store directly Gzipped content, but that doesn't seems to be working for me. Any leads are appreceated, Thank you.

If put_value is a string, it will be truncated on the first \0 character that is encountered. Try converting put_value to a bytearray before sending it down:
conn.put(key, {'value': bytearray(put_value,"utf-8")})

KineticJS toDataURL() gives incorrect padding error in Python

I have a base64 string that I've acquired from KineticJS toDataURL(). It's a valid base64 string. This fiddle shows that it is: http://jsfiddle.net/FakRe/
My Problem: I'm sending this dataURI up to my server, decoding using python, and saving on the server. However, I'm getting a padding error. Here's my code:
def base64_to_file(self, image_data, image_name):
extension = re.search('data\:image\/(\w+)\;', image_data)
if extension:
ext = extension.group(1)
image_name = '{0}.{1}'.format(image_name, ext)
else:
# it's not in a format we understand
return None
image_data = image_data + '=' #fix incorrect padding
image_path = os.path.join('/my/image/path/', image_name)
image_file = open(image_path, 'w+')
image_file.write(image_data.decode('base64'))
image_file.close()
return image_file
I can get around this padding error by doing this at the top of my function:
image_data = image_data + '=' #fixes incorrect padding
After I add the arbitrary padding, it decodes successfully and writes to the filesystem. However, whenever I try and view the image, it doesn't render, and gives me the 'broken image' icon. No 404, just a broken image. What gives? Any help would be greatly appreciated. I've already spent way too much time on this as it is.
Steps to recreate (May be pedantic but I'll try and help)
Grab the base64 string from the JSFiddle
Save it to a text file
Open up the file in python, read in the data, save to variable
Change '/path/to/my/image' in the function to anywhere on your machine
Pass in your encoded text variable into the function with an name
See the output
Again, any help would be awesome. Thanks.

If you need to add padding, you have the wrong string. Do make sure you are parsing the data URI correctly; the data:image/png;base64, section is metadata about the encoded value, only the part after the comma is the actual Base64 value.
The actual data portion is 223548 characters long:
>>> len(image_data)
223548
>>> import hashlib
>>> hashlib.md5(image_data).hexdigest()
'03918c3508fef1286af8784dc65f23ff'
If your URI still includes the data: prefix, do parse that out:
from urllib import unquote
if image_data.startswith('data:'):
params, data = image_data.split(',', 1)
params = params[5:] or 'text/plain;charset=US-ASCII'
params = params.split(';')
if not '=' in params[0] and '/' in params[0]:
mimetype = params.pop(0)
else:
mimetype = 'text/plain'
if 'base64' in params:
# handle base64 parameters first
data = data.decode('base64')
for param in params:
if param.startswith('charset='):
# handle characterset parameter
data = unquote(data).decode(param.split('=', 1)[-1])
This then lets you make some more informed decisions about what extension to use for image URLs, for example, as you now have the mimetype parsed out as well.