When I try to JPEG-Decompress (old-style JPEG compression, not JPEG-LS and not JPEG2000) the RAW data, I get following error:
Traceback (most recent call last):
File "raw-reader.py", line 766, in <module>
raw_image_data = imageio.imread(io.BytesIO(raw_packed_image_data))
File "/home/ian/.local/lib/python3.6/site-packages/imageio/core/functions.py", line 206, in imread
reader = read(uri, format, 'i', **kwargs)
File "/home/ian/.local/lib/python3.6/site-packages/imageio/core/functions.py", line 129, in get_reader
return format.get_reader(request)
File "/home/ian/.local/lib/python3.6/site-packages/imageio/core/format.py", line 168, in get_reader
return self.Reader(self, request)
File "/home/ian/.local/lib/python3.6/site-packages/imageio/core/format.py", line 217, in __init__
self._open(**self.request.kwargs.copy())
File "/home/ian/.local/lib/python3.6/site-packages/imageio/plugins/pillow.py", line 398, in _open
pilmode=pilmode, as_gray=as_gray)
File "/home/ian/.local/lib/python3.6/site-packages/imageio/plugins/pillow.py", line 122, in _open
self._im = factory(self._fp, '')
File "/home/ian/.local/lib/python3.6/site-packages/PIL/JpegImagePlugin.py", line 780, in jpeg_factory
im = JpegImageFile(fp, filename)
File "/home/ian/.local/lib/python3.6/site-packages/PIL/ImageFile.py", line 102, in __init__
self._open()
File "/home/ian/.local/lib/python3.6/site-packages/PIL/JpegImagePlugin.py", line 339, in _open
handler(self, i)
File "/home/ian/.local/lib/python3.6/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF
raise SyntaxError("cannot handle %d-bit layers" % self.bits)
SyntaxError: cannot handle 14-bit layers
The RAW data in the image is 14-bit JPEG data, and imageio isn't able to read it. When I tried using pillow, it didn't even recognize the data as JPEG. My question now is: How can I decompress the data without writing my own JPEG decompressor, while keeping in mind that the data is 14 bits?
My code:
import io
import imageio
allbytes = open("raw_data.dat", "rb").read()
raw_packed_image_data = allbytes
raw_image_data = imageio.imread(io.BytesIO(raw_packed_image_data))
The file raw_data.dat is a file containing purely the RAW-Image data compressed with JPEG. Link to raw_data.dat
raw_data.dat is a JPEG Lossless, Nonhierarchical file with 2 frames and a precision > 8-bit, a very rare format.
The imagecodecs package can read the file (assuming that the _imagecodecs Cython extension is present):
>>> from imagecodecs import jpegsof3_decode
>>> data = open('raw_data.dat', 'rb').read()
>>> image = jpegsof3_decode(data)
>>> image.shape
(3528, 2640, 2)
>>> image.dtype
dtype('uint16')
The LEADTOOLS SDK should also be able to read the file (not tested).
I believe the issue can be solved using a different library to load the image. Similar underlying issue has been posted here How to combine 3 high range JPEG2000 images into single RGB one?. You can use something along these lines:
import matplotlib.image as mpimg
img_red = mpimg.imread('raw_data.dat')
Then you can use the read bitmap for further manipulation as if it was loaded via PIL/Pillow/imageio.
Related
I can't find a way of reading the Minecraft world files in a way that i could use in python
I've looked around the internet but can find no tutorials and only a few libraries that claim that they can do this but never actually work
from nbt import *
nbtfile = nbt.NBTFile("r.0.0.mca",'rb')
I expected this to work but instead I got errors about the file not being compressed or something of the sort
Full error:
Traceback (most recent call last):
File "C:\Users\rober\Desktop\MinePy\MinecraftWorldReader.py", line 2, in <module>
nbtfile = nbt.NBTFile("r.0.0.mca",'rb')
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\site-packages\nbt\nbt.py", line 628, in __init__
self.parse_file()
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\site-packages\nbt\nbt.py", line 652, in parse_file
type = TAG_Byte(buffer=self.file)
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\site-packages\nbt\nbt.py", line 99, in __init__
self._parse_buffer(buffer)
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\site-packages\nbt\nbt.py", line 105, in _parse_buffer
self.value = self.fmt.unpack(buffer.read(self.fmt.size))[0]
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\gzip.py", line 276, in read
return self._buffer.read(size)
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\gzip.py", line 463, in read
if not self._read_gzip_header():
File "C:\Users\rober\AppData\Local\Programs\Python\Python36-32\lib\gzip.py", line 411, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'\x00\x00')
Use anvil parser. (Install with pip install anvil-parser)
Reading
import anvil
region = anvil.Region.from_file('r.0.0.mca')
# You can also provide the region file name instead of the object
chunk = anvil.Chunk.from_region(region, 0, 0)
# If `section` is not provided, will get it from the y coords
# and assume it's global
block = chunk.get_block(0, 0, 0)
print(block) # <Block(minecraft:air)>
print(block.id) # air
print(block.properties) # {}
https://pypi.org/project/anvil-parser/
According to this page, the .mca files is not totally kind of of NBT file. It begins with an 8KiB header which includes the offsets of chunks in the region file itself and the timestamps for the last updates of those chunks.
I recommend you to see the offical announcement and this page for more information.
I am trying to load a pickle file which i created using joblib.dump()
The dumping code looks like:
from sklearn.externals import joblib
with open('sample.pickle','wb') as f:
joblib.dump([x,y],f)
This works fine and my sample.pickle is saved successfully. But when i try to load this file:
with open('sample.pickle', 'rb') as f:
x, y = joblib.load(f)
I get the following error:
ValueError: EOF: reading array data, expected 1200 bytes got 0
The full error log looks like this:
Traceback (most recent call last):
File "model.py", line 16, in <module>
vec_x, vec_y = joblib.load(f)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 568, in load
obj = _unpickle(fobj)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 508, in _unpickle
obj = unpickler.load()
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\pickle.py", line 1050, in load
dispatch[key[0]](self)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 341, in load_build
self.stack.append(array_wrapper.read(self))
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 184, in read
array = self.read_array(unpickler)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 135, in read_array
read_size, "array data")
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle_utils.py", line 646, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 1200 bytes got 0
By the way, i am very new to dumping and pickling.
Any help will be appreciated. Thanks in advance.
Pillow looks to be the solution to a lot of my problems, but I can't get it to work correctly. I think there might be something that I'm missing but I can't find any solutions that work.
from PIL import Image
img = Image.open("base64 (1).png")
print(img.size)
print(img.format)
img.show()
I have this set up to load an image, print its size and format, and then show the image. When I run the code, I get this:
(1920, 1080)
PNG
Traceback (most recent call last):
File "C:/Users/##########/PycharmProjects/weatherCanvas/imageTest.py", line 7, in <module>
img.show()
File "C:\Python27\lib\site-packages\PIL\Image.py", line 1736, in show
_show(self, title=title, command=command)
File "C:\Python27\lib\site-packages\PIL\Image.py", line 2449, in _show
_showxv(image, **options)
File "C:\Python27\lib\site-packages\PIL\Image.py", line 2454, in _showxv
ImageShow.show(image, title, **options)
File "C:\Python27\lib\site-packages\PIL\ImageShow.py", line 51, in show
if viewer.show(image, title=title, **options):
File "C:\Python27\lib\site-packages\PIL\ImageShow.py", line 75, in show
image = image.convert(base)
File "C:\Python27\lib\site-packages\PIL\Image.py", line 844, in convert
self.load()
File "C:\Python27\lib\site-packages\PIL\ImageFile.py", line 250, in load
raise_ioerror(e)
File "C:\Python27\lib\site-packages\PIL\ImageFile.py", line 59, in raise_ioerror
raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
The same thing happens if I replace img.show() with saving it. I know that PIL and Pillow save a temporary file in order to do the show() method, so I think the problem is somewhere in that.
Is there something I'm missing here?
I need to read a few xls files into Python.The sample data file can be found through Link:data.file. I tried:
import pandas as pd
pd.read_excel('data.xls',sheet=1)
But it gives an error message:
ERROR *** codepage 21010 -> encoding 'unknown_codepage_21010' ->
LookupError: unknown encoding: unknown_codepage_21010 Traceback (most
recent call last):
File "", line 1, in
pd.read_excel('data.xls',sheet=1)
File "C:\Anaconda3\lib\site-packages\pandas\io\excel.py", line 113,
in read_excel
return ExcelFile(io, engine=engine).parse(sheetname=sheetname, **kwds)
File "C:\Anaconda3\lib\site-packages\pandas\io\excel.py", line 150,
in init
self.book = xlrd.open_workbook(io)
File "C:\Anaconda3\lib\site-packages\xlrd__init__.py", line 435, in
open_workbook
ragged_rows=ragged_rows,
File "C:\Anaconda3\lib\site-packages\xlrd\book.py", line 116, in
open_workbook_xls
bk.parse_globals()
File "C:\Anaconda3\lib\site-packages\xlrd\book.py", line 1170, in
parse_globals
self.handle_codepage(data)
File "C:\Anaconda3\lib\site-packages\xlrd\book.py", line 794, in
handle_codepage
self.derive_encoding()
File "C:\Anaconda3\lib\site-packages\xlrd\book.py", line 775, in
derive_encoding
_unused = unicode(b'trial', self.encoding)
File "C:\Anaconda3\lib\site-packages\xlrd\timemachine.py", line 30,
in
unicode = lambda b, enc: b.decode(enc)
LookupError: unknown encoding: unknown_codepage_21010
Anyone could help with this problem?
PS: I know if I open the file in windows excel, and resave it, the code could work, but I am looking for a solution without manual adjustment.
using the ExcelFile class, I was successfully able to read the file into python.
let me know if this helps!
import xlrd
import pandas as pd
xls = pd.ExcelFile(’C:\data.xls’)
xls.parse(’Index Constituents Data’, index_col=None, na_values=[’NA’])
The below worked for me.
import xlrd
my_xls = xlrd.open_workbook('//myshareddrive/something/test.xls',encoding_override="gb2312")
I am new to python. I have a file data.pkl. What I would like to do is get the data from the file. I looked at http://docs.python.org/library/pickle.html, 11.1.7 example and tried exactly that.
My code looks like this:
import pprint, pickle
pkl_file = open('data.pkl', 'rb')
data1 = pickle.load(pkl_file)
pprint.pprint(data1)
pkl_file.close()
But it is giving me error:
Traceback (most recent call last):
File "/home/sadiksha/workspace/python/test.py", line 5, in <module>
data1 = pickle.load(pkl_file)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 966, in load_string
raise ValueError, "insecure string pickle"
Can anyone please tell me what am I doing wrong here?
It seems that your pickle file was either not written correctly (specifying 'wb') or the file was somehow corrupted. Try creating your own pickle file and reading that back in. That should do the trick.
As for the pickle file specified, it is definitely corrupted.