Develop Reference

Is there any way to use arithmetic ops on FITS files in Python?

I'm fairly new to Python, and I have been trying to recreate a working IDL program to Python, but I'm stuck and keep getting errors. I haven't been able to find a solution yet.
The program requires 4 FITS files in total (img and correctional images dark, flat1, flat2). The operations are as follows:
flat12 = (flat1 + flat2)/2
img1 = (img - dark)/flat12
The said files have dimensions (1024,1024,1). I have resized them to (1024,1024) to be able to even use im_show() function.
I have also tried using cv2.add(), but I get this:
TypeError: Expected Ptr for argument 'src1'
Is there any workaround for this? Thanks in advance.

To read your FITS files use astropy.io.fits: http://docs.astropy.org/en/latest/io/fits/index.html
This will give you Numpy arrays (and FITS headers if needed, there are different ways to do this, as explained in the documentation), so you could do something like:
>>> from astropy.io import fits
>>> img = fits.getdata('image.fits', ext=0) # extension number depends on your FITS files
>>> dark = fits.getdata('dark.fits') # by default it reads the first "data" extension
>>> darksub = img - dark
>>> fits.writeto('out.fits', darksub) # save output
If your data has an extra dimension, as shown with the (1024,1024,1) shape, and if you want to remove that axis, you can use the normal Numpy array slicing syntax: darksub = img[0] - dark[0].
Otherwise in the example above it will produce and save a (1024,1024,1) image.

Reading pre-processed cr2 RAW image data in python

I am trying to read raw image data from a cr2 (canon raw image file). I want to read the data only (no header, etc.) pre-processed if possible (i.e pre-bayer/the most native unprocessed data) and store it in a numpy array. I have tried a bunch of libraries such as opencv, rawkit, rawpy but nothing seems to work correctly.
Any suggestion on how I should do this? What I should use? I have tried a bunch of things.
Thank you

Since libraw/dcraw can read cr2, it should be easy to do. With rawpy:
#!/usr/bin/env python
import rawpy
raw = rawpy.imread("/some/path.cr2")
bayer = raw.raw_image # with border
bayer_visible = raw.raw_image_visible # just visible area
Both bayer and bayer_visible are then a 2D numpy array.

You can use rawkit to get this data, however, you won't be able to use the actual rawkit module (which provides higher level APIs for dealing with Raw images). Instead, you'll want to use mostly the libraw module which allows you to access the underlying LibRaw APIs.
It's hard to tell exactly what you want from this question, but I'm going to assume the following: Raw bayer data, including the "masked" border pixels (which aren't displayed, but are used to calculate various things about the image). Something like the following (completely untested) script will allow you to get what you want:
#!/usr/bin/env python
import ctypes
from rawkit.raw import Raw
with Raw(filename="some_file.CR2") as raw:
raw.unpack()
# For more information, see the LibRaw docs:
# http://www.libraw.org/docs/API-datastruct-eng.html#libraw_rawdata_t
rawdata = raw.data.contents.rawdata
data_size = rawdata.sizes.raw_height * rawdata.sizes.raw_width
data_pointer = ctypes.cast(
rawdata.raw_image,
ctypes.POINTER(ctypes.c_ushort * data_size)
)
data = data_pointer.contents
# Grab the first few pixels for demonstration purposes...
for i in range(5):
print('Pixel {}: {}'.format(i, data[i]))
There's a good chance that I'm misunderstanding something and the size is off, in which case this will segfault eventually, but this isn't something I've tried to make LibRaw do before.
More information can be found in this question on the LibRaw forums, or in the LibRaw struct docs.
Storing in a numpy array I leave as an excersize for the user, or for a follow up answer (I have no experience with numpy).

Best dtype for creating large arrays with numpy

I am looking to store pixel values from satellite imagery into an array. I've been using
np.empty((image_width, image_length)
and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.
I load the image values into the array using a loop and opening the image with gdal
img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()
but when I include print img_array I end up with just zeros.
I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.
Is numpy unable to load this many values or is there a way to optimize the array?
I am working with 8-bit tiff images that contain NDVI (decimal) values.
Thanks

Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:
dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2
But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy
If you give some more info on the type of images you want to read in, i can maybe help more specifically
Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)
A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()
Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.

PyQt/PySide: How do I convert QImage into OpenCV's MAT format

I'm looking to create a function for converting a QImage into OpenCV's (CV2) Mat format from within the PyQt.
How do I do this? My input images I've been working with so far are PNGs (either RGB or RGBA) that were loaded in as a QImage.
Ultimately, I want to take two QImages and use the matchTemplate function to find one image in the other, so if there is a better way to do that than I'm finding now, I'm open to that as well. But being able to convert back and forth between the two easily would be ideal.
Thanks for your help,

After much searching on here, I found a gem that got me a working solution. I derived much of my code from this answer to another question: https://stackoverflow.com/a/11399959/1988561
The key challenge I had was in how to correctly use the pointer. The big thing I think I was missing was the setsize function.
Here's my imports:
import cv2
import numpy as np
Here's my function:
def convertQImageToMat(incomingImage):
''' Converts a QImage into an opencv MAT format '''
incomingImage = incomingImage.convertToFormat(4)
width = incomingImage.width()
height = incomingImage.height()
ptr = incomingImage.bits()
ptr.setsize(incomingImage.byteCount())
arr = np.array(ptr).reshape(height, width, 4) # Copies the data
return arr

I tried the answer given above, but couldn't get the expected thing. I tried this crude method where i saved the image using the save() method of the QImage class and then used the image file to read it in cv2
Here is a sample code
def qimg2cv(q_img):
q_img.save('temp.png', 'png')
mat = cv2.imread('temp.png')
return mat
You could delete the temporary image file generated once you are done with the file.
This may not be the right method to do the work, but still does the required job.

Get Data from OpenGL glReadPixels(using Pyglet)

I'm using Pyglet(and OpenGL) in Python on an application, I'm trying to use glReadPixels to get the RGBA values for a set of pixels. It's my understanding that OpenGL returns the data as packed integers, since that's how they are stored on the hardware. However for obvious reasons I'd like to get it into a normal format for working with. Based on some reading I've come up with this: http://dpaste.com/99206/ , however it fails with an IndexError. How would I go about doing this?

You must first create an array of the correct type, then pass it to glReadPixels:
a = (GLuint * 1)(0)
glReadPixels(x, y, 1, 1, GL_RGB, GL_UNSIGNED_INT, a)
To test this, insert the following in the Pyglet "opengl.py" example:
#window.event
def on_mouse_press(x, y, button, modifiers):
a = (GLuint * 1)(0)
glReadPixels(x, y, 1, 1, GL_RGB, GL_UNSIGNED_INT, a)
print a[0]
Now you should see the color code for the pixel under the mouse cursor whenever you click somewhere in the app window.

I was able to obtain the entire frame buffer using glReadPixels(...), then used the PIL to write out to a file:
# Capture image from the OpenGL buffer
buffer = ( GLubyte * (3*window.width*window.height) )(0)
glReadPixels(0, 0, window.width, window.height, GL_RGB, GL_UNSIGNED_BYTE, buffer)
# Use PIL to convert raw RGB buffer and flip the right way up
image = Image.fromstring(mode="RGB", size=(window.width, window.height), data=buffer)
image = image.transpose(Image.FLIP_TOP_BOTTOM)
# Save image to disk
image.save('jpap.png')
I was not interested in alpha, but I'm sure you could easily add it in.
I was forced to use glReadPixels(...), instead of the Pyglet code
pyglet.image.get_buffer_manager().get_color_buffer().save('jpap.png')
because the output using save(...) was not identical to what I saw in the Window. (Multisampling buffers missed?)

You can use the PIL library, here is a code snippet which I use to capture such an image:
buffer = gl.glReadPixels(0, 0, width, height, gl.GL_RGB,
gl.GL_UNSIGNED_BYTE)
image = Image.fromstring(mode="RGB", size=(width, height),
data=buffer)
image = image.transpose(Image.FLIP_TOP_BOTTOM)
I guess including the alpha channel should be pretty straight forward (probably just replacing RGB with RGBA, but I have not tried that).
Edit: I wasn't aware that the pyglet OpenGL API is different from the PyOpenGL one. I guess one has to change the above code to use the buffer as the seventh argument (conforming to the less pythonic pyglet style).

If you read the snippet you link to you can understand that the simplest and way to get the "normal" values is just accessing the array in the right order.
That snippet looks like it's supposed to do the job. If it doesn't, debug it and see what's the problem.

On further review I believe my original code was based on some C specific code that worked because the array is just a pointer, so my using pointer arithmetic you could get at specific bytes, that obviously doesn't translate to Python. Does anyone how to extract that data using a different method(I assume it's just a matter of bit shifting the data).