I have downloaded large amounts of MODIS hdf files and extracted needed bands from each file.How can I mosaic all the hdf files together using GDAL lib in Python and reproject it?Thanks!
Use two GDAL command line tools:
gdal_merge.py to mosaic the bands into one raster
gdalwarp to reproject the raster
You could try also pyModis [0], it is using Python GDAL in the development version (from today, I need to test it more).
gdal_merge.py doesn't work with MODIS HDF (probably it is a bug), this is the returned error
Traceback (most recent call last):
File "/usr/local/bin/gdal_merge.py", line 510, in <module>
sys.exit(main())
File "/usr/local/bin/gdal_merge.py", line 390, in main
file_infos = names_to_fileinfos( names )
File "/usr/local/bin/gdal_merge.py", line 122, in names_to_fileinfos
if fi.init_from_name( name ) == 1:
File "/usr/local/bin/gdal_merge.py", line 147, in init_from_name
self.band_type = fh.GetRasterBand(1).DataType
AttributeError: 'NoneType' object has no attribute 'DataType'
[0] https://github.com/lucadelu/pyModis
Related
I'm attempting to gather temperature, humidity, and pm2.5 data for a project related to heat vulnerability. I'm using the following dataset from NASA to gather temperature data: MERRA-2 inst6_3d_ana_Nv: 3d,6-Hourly,Instantaneous,Model-Level,Analysis,Analyzed Meteorological Fields V5.12.4 (M2I6NVANA) https://disc.gsfc.nasa.gov/datasets/M2I6NVANA_5.12.4/summary
I've downloaded the .nc4 files using curl using these instructions: https://disc.gsfc.nasa.gov/data-access#mac_linux_curl
I'm using the netcdf4 python library to read and understand the data, but I'm being met with this error:
-bash-4.2$python pyScripts/nasaData.py
Traceback (most recent call last):
File "pyScripts/nasaData.py", line 5, in <module>
f = netCDF4.Dataset('/home/uja4bs/NASA_Data/MERRA2_400.inst6_3d_ana_Nv.20200101.nc4', 'r')
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: b'/home/uja4bs/NASA_Data/MERRA2_400.inst6_3d_ana_Nv.20200101.nc4'
Any pointers would be greatly appreciated!
I need to extract a single file (~10kB) from many very large RAR files (>1Gb). The code below shows a basic implementation of how I'm doing this.
from rarfile import RarFile
rar_file='D:\\File.rar'
file_of_interest='Folder 1/Subfolder 2/File.dat'
output_folder='D:/Output'
rardata = RarFile(rar_file)
rardata.extract(file_of_interest, output_folder)
rardata.close()
However, the extract instruction is returning the following error: rarfile.BadRarFile: Failed the read enough data: req=16384 got=52
When I open the file using WinRAR, I can extract the file successfully, so I'm sure the file isn't corrupted.
I've found some similar questions, but not a definite answer that worked for me.
Can someone help me to solve this error?
Additional info:
Windows 10 build 1909
Spyder 5.0.0
Python 3.8.1
Complete traceback of the error:
Traceback (most recent call last):
File "D:\Test\teste_rar_2.py", line 27, in <module>
rardata.extract(file_of_interest, output_folder)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 826, in extract
return self._extract_one(inf, path, pwd, True)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 912, in _extract_one
return self._make_file(info, dstfn, pwd, set_attrs)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 927, in _make_file
shutil.copyfileobj(src, dst)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 79, in copyfileobj
buf = fsrc.read(length)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 2197, in read
raise BadRarFile("Failed the read enough data: req=%d got=%d" % (orig, len(data)))
BadRarFile: Failed the read enough data: req=16384 got=52
import os
import numpy as np
import mne
data_folder = '/Users/sheetal/Desktop/emotions'
data_raw_file = os.path.join(data_folder,"sub-01","eeg","sub-01_task-ImaginedEmotion_eeg.set")
raw = mne.io.read_raw_eeglab(data_raw_file)
OUTPUT
runfile('/Users/sheetal/Desktop/emotions/sub-01/eeg/code1.py', wdir='/Users/sheetal/Desktop/emotions/sub-01/eeg')
Traceback (most recent call last):
File "/Users/sheetal/Desktop/emotions/sub-01/eeg/code1.py", line 15, in <module>
raw = mne.io.read_raw_eeglab(data_raw_file)
File "/Users/sheetal/opt/anaconda3/lib/python3.8/site-packages/mne/io/eeglab/eeglab.py", line 219, in read_raw_eeglab
return RawEEGLAB(input_fname=input_fname, preload=preload,
File "<decorator-gen-222>", line 24, in __init__
File "/Users/sheetal/opt/anaconda3/lib/python3.8/site-packages/mne/io/eeglab/eeglab.py", line 317, in __init__
eeg = _check_load_mat(input_fname, uint16_codec)
File "/Users/sheetal/opt/anaconda3/lib/python3.8/site-packages/mne/io/eeglab/eeglab.py", line 59, in _check_load_mat
eeg = read_mat(fname, uint16_codec=uint16_codec)
File "/Users/sheetal/opt/anaconda3/lib/python3.8/site-packages/mne/externals/pymatreader/pymatreader.py", line 78, in read_mat
raise IOError('The file %s does not exist.' % (filename,))
OSError: The file /Users/sheetal/Desktop/emotions/sub-01/eeg/sub-01_task-ImaginedEmotion_eeg.set does not exist.
I am new to use MNE python package for EEG signal processing and getting error reading .set file
I occasionally encounter this problem.
The .set & corresponding .fdt files are actually there, but unfortunately MNE cannot read those files, resulting in file not found error.
It seemed that the issue is related to the generating files from the EEGLab, which, due to its unstable development, often produce a lot of errors.
Suggestion: re-generating those files from EEGLab.
UPDATE
Currently, I found that my problem is relating to the uncompatible version of EEGLab and Python-MNE.
Previously, I used EEGLab 2019.1 to generate .set files and Python-MNE 0.20.4 to read those files.
Later, EEGLab released the latest version 2021.1 and disable 2019.1 which forced us to use the latest one.
The problems arrive when I used the EEGLab 2021.1 with MNE 0.20.4 which doesn't support the latest version of EEGLab.
Updating MNE to the latest 0.23.4 solves my issue.
I have the following piece of python code to convert PDF to JPG.
with Img(filename=pdfName, resolution=300) as pic:
pic.compression_quality = self.compressionQuality
pic.background_color = Color("white")
pic.alpha_channel = 'remove'
pic.save(filename=output)
My problem is, with a large PDF file (10mb) I have the following error :
File "/home/nathan/PycharmProjects/oc_for_maarch/worker.py", line 44, in <module>
launch(args)
File "/home/nathan/PycharmProjects/oc_for_maarch/src/main.py", line 105, in launch
q = process(args, path + file, Log, Separator, Config, Image, Ocr, Locale, WebService, q)
File "/home/nathan/PycharmProjects/oc_for_maarch/src/process/OCForMaarch.py", line 48, in process
Image.pdf_to_jpg(file + '[0]')
File "/home/nathan/PycharmProjects/oc_for_maarch/src/classes/Images.py", line 36, in pdf_to_jpg
self.save_img_with_wand(pdfName, self.jpgName)
File "/home/nathan/PycharmProjects/oc_for_maarch/src/classes/Images.py", line 46, in save_img_with_wand
with Img(filename=pdfName, resolution=300) as pic:
File "/home/nathan/Documents/OpenCV/lib/python3.7/site-packages/wand/image.py", line 6406, in __init__
self.read(filename=filename, resolution=resolution)
File "/home/nathan/Documents/OpenCV/lib/python3.7/site-packages/wand/image.py", line 6799, in read
raise WandRuntimeError(msg)
wand.exceptions.WandRuntimeError: MagickReadImage returns false, but did raise ImageMagick exception. This can occurs when a delegate is missing, or returns EXIT_SUCCESS without generating a raster.
I checked a little on Internet, and for what I've seen the problems was related to ghostscript but It's installed
I have the problem on Debian 10 and Ubuntu 19.04 using Python 3.7
EDIT : If I put the resolution to 100 instead of 300, I didn't have the issue
When you rasterize at a high density, you will make a potentially very large dimension image from your PDF. So it sounds like you may be running out of RAM. If so, then you need to edit your ImageMagick policy.xml file to allow for more ram or map space. See policy.xml at https://imagemagick.org/script/resources.php. It controls your resources which you can view with the command line command:
convert -list resource
I am trying to convert jpeg files to eps ones. I am using the following code:
fp=open("test.jpg",'rb')
im=Image.open(fp)
outf=open('test2.eps','wb')
im.save(outf, 'EPS')
However, I ma getting the following error:
Traceback (most recent call last):
File "im2eps2.py", line 11, in <module> im.save(outf, 'EPS')
File "C:\Python26\Lib\site-packages\PIL\Image.py", line 1465, in save
save_handler(self, fp, filename)
File "C:\Python26\Lib\site-packages\PIL\EpsImagePlugin.py", line 353, in _save
fp = io.TextIOWrapper(NoCloseStream(fp), encoding='latin-1')
File "C:\Python26\Lib\io.py", line 1429, in __init__
self._seekable = self._telling = self.buffer.seekable()
File "C:\Python26\Lib\site-packages\PIL\EpsImagePlugin.py", line 348, in __getattr__
return getattr(self.fp, name)
AttributeError: 'file' object has no attribute 'seekable'
I shall be thankful for suggestions.
Thanks
PS: I reinstall PIL from its main page i.e. http://www.pythonware.com/products/pil/ and it worked :) Earlier, I used windows installer provided by http://www.lfd.uci.edu/~gohlke/pythonlibs. I think the problem was with the binary that I installed earlier. Currently, I have 1.1.7 PIL and it is working fine.
Thanks
According to cgohlke this was a bug in Pillow and has been fixed in 2.4.0, which was released last month (1st April 2014).
From the bug report:
io.TextIOWrapper expects an object with io.IOBase interface, not a Python 2 file object. Using fh = io.open('test.eps', 'wb') passes this stage but io.TextIOWrapper.write() expects unicode strings; the example fails with TypeError: can't write str to text stream.