Unknown file format while dealing with netcdf files from NASA (Python) - python

I'm attempting to gather temperature, humidity, and pm2.5 data for a project related to heat vulnerability. I'm using the following dataset from NASA to gather temperature data: MERRA-2 inst6_3d_ana_Nv: 3d,6-Hourly,Instantaneous,Model-Level,Analysis,Analyzed Meteorological Fields V5.12.4 (M2I6NVANA) https://disc.gsfc.nasa.gov/datasets/M2I6NVANA_5.12.4/summary
I've downloaded the .nc4 files using curl using these instructions: https://disc.gsfc.nasa.gov/data-access#mac_linux_curl
I'm using the netcdf4 python library to read and understand the data, but I'm being met with this error:
-bash-4.2$python pyScripts/nasaData.py
Traceback (most recent call last):
File "pyScripts/nasaData.py", line 5, in <module>
f = netCDF4.Dataset('/home/uja4bs/NASA_Data/MERRA2_400.inst6_3d_ana_Nv.20200101.nc4', 'r')
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: b'/home/uja4bs/NASA_Data/MERRA2_400.inst6_3d_ana_Nv.20200101.nc4'
Any pointers would be greatly appreciated!

Related

Read s2p Z parameter files in Python

I am trying to read s2p files in using Python scikit-rf library. The s2p file in not in S parameter. I converted it to Z parameter. Looks like scikit-rf can read only S parameter file. Is there any other library or way to read s2p files with Z parameter. Below is how I am reading the file:
import skrf as rf
ntwk = rf.Network('ZTEST5_SHORTED.s10p')
s = ntwk.s
print(s)
Below is the error:
Traceback (most recent call last):
File "C:\Users\pujai\PycharmProjects\SIIntegration\practice.py", line 26, in <module>
ntwk = rf.Network('ZTEST5_SHORTED_DCfitted.s10p')
File "C:\Users\pujai\PycharmProjects\SIIntegration\venv\lib\site-packages\skrf\network.py", line 457, in __init__
self.read_touchstone(filename, self.encoding)
File "C:\Users\pujai\PycharmProjects\SIIntegration\venv\lib\site-packages\skrf\network.py", line 2050, in read_touchstone
raise NotImplementedError('only s-parameters supported for now.')
NotImplementedError: only s-parameters supported for now.
If I try to read an S parameter file, it works fine. Any help?

BadRarFile when extracting single file using RarFile in Python

I need to extract a single file (~10kB) from many very large RAR files (>1Gb). The code below shows a basic implementation of how I'm doing this.
from rarfile import RarFile
rar_file='D:\\File.rar'
file_of_interest='Folder 1/Subfolder 2/File.dat'
output_folder='D:/Output'
rardata = RarFile(rar_file)
rardata.extract(file_of_interest, output_folder)
rardata.close()
However, the extract instruction is returning the following error: rarfile.BadRarFile: Failed the read enough data: req=16384 got=52
When I open the file using WinRAR, I can extract the file successfully, so I'm sure the file isn't corrupted.
I've found some similar questions, but not a definite answer that worked for me.
Can someone help me to solve this error?
Additional info:
Windows 10 build 1909
Spyder 5.0.0
Python 3.8.1
Complete traceback of the error:
Traceback (most recent call last):
File "D:\Test\teste_rar_2.py", line 27, in <module>
rardata.extract(file_of_interest, output_folder)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 826, in extract
return self._extract_one(inf, path, pwd, True)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 912, in _extract_one
return self._make_file(info, dstfn, pwd, set_attrs)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 927, in _make_file
shutil.copyfileobj(src, dst)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 79, in copyfileobj
buf = fsrc.read(length)
File "C:\Users\bernard.kusel\AppData\Local\Continuum\anaconda3\lib\site-packages\rarfile.py", line 2197, in read
raise BadRarFile("Failed the read enough data: req=%d got=%d" % (orig, len(data)))
BadRarFile: Failed the read enough data: req=16384 got=52

How to write .wav file from data using wavio api

I'm using WAVIO python API to write data to a .wav file.
In order to test this api, First I read data .wav file and then I want to generate .wav containing the same data of the input .wav file, but I got an error while writing data to the output wav file.
import numpy as np
import wavio
wa = wavio.read("s24.wav") #Read a .wav file
print("x= "+str(wa.data)) #Data
print("rate= "+str(wa.rate)) #Rate
print("sampwidth= "+str(wa.sampwidth)) #sampwidth
wavio.write("sine24_output", wa.data, wa.rate,wa.sampwidth) #Error is here
Error is:
Traceback (most recent call last):
File "C:/Users/user/Desktop/Wav2Bytes/wavii.py", line 24, in <module>
wavio.write("sine24_output", wa.data, wa.rate,wa.sampwidth) #Write a .wav file. Error is here
File "C:\Program Files (x86)\Python\lib\site-packages\wavio.py", line 363, in write
vmin, vmax = scale
TypeError: cannot unpack non-iterable int object
You are missing the argument's name sampwidth. According to the docs, there are 5 arguments, if you don't specify the name sampwidth, it will instead set scale to wa.sampwidth. Anyway just use this:
wavio.write("sine24_output", wa.data, wa.rate,sampwidth=wa.sampwidth)

Mykrobe predictor JSON to TSV Converter

I wanted to ask a question regarding file conversion.
I have a JSON file (after AMR prediction execution) that I want to covert to a TSV file based on Mykrobe-predictor scripts (json_to_tsv.py) and this is my JSON output (result_TB.json).
./json_to_tsv.py /path/to/JSON_file
When I pasted a command into the terminal, I got a IndexError at Line 78.
https://github.com/iqbal-lab/Mykrobe-predictor/blob/master/scripts/json_to_tsv.py#L78
def get_sample_name(f):
return f.split('/')[-2]
And here is the error I get:
mykrobe_version file plate_name sample drug phylo_group species lineage phylo_group_per_covg species_per_covg lineage_per_covg phylo_group_depth species_depth lineage_depth susceptibility variants (gene:alt_depth:wt_depth:conf) genes (prot_mut-ref_mut:percent_covg:depth)
Traceback (most recent call last):
File "./json_to_tsv.py", line 157, in <module>
sample_name = get_sample_name(f)
File "./json_to_tsv.py", line 78, in get_sample_name
return f.split('/')[-2]
IndexError: list index out of range
Any suggestions would be appreciated.
Looking at the code I guess they expect to call the converter with something like:
python json_to_tsv.py plate/sample1/sample1.json
Try copying your JSON file to a directory called sample1 inside a directory called plate and see if you get the same error when you call it like in the example above.
Update
The problem is indeed as described above.
Doesn't work:
python json_to_tsv.py result_TB.json
mykrobe_version file plate_name sample drug phylo_group species lineage phylo_group_per_covg species_per_covg lineage_per_covg phylo_group_depth species_depth lineage_depth susceptibility variants
(gene:alt_depth:wt_depth:conf) genes
(prot_mut-ref_mut:percent_covg:depth)
Traceback (most recent call last): File "json_to_tsv.py", line 157, in <module>
sample_name = get_sample_name(f) File "json_to_tsv.py", line 78, in get_sample_name
return f.split('/')[-2] IndexError: list index out of range
Works:
python json_to_tsv.py plate/sample/result_TB.json
mykrobe_version file plate_name sample drug phylo_group species lineage phylo_group_per_covg species_per_covg lineage_per_covg phylo_group_depth species_depth lineage_depth susceptibility variants (gene:alt_depth:wt_depth:conf) genes (prot_mut-ref_mut:percent_covg:depth)
-1 result_TB plate sample NA

How to mosaic MODIS hdf files using Python GDAL?

I have downloaded large amounts of MODIS hdf files and extracted needed bands from each file.How can I mosaic all the hdf files together using GDAL lib in Python and reproject it?Thanks!
Use two GDAL command line tools:
gdal_merge.py to mosaic the bands into one raster
gdalwarp to reproject the raster
You could try also pyModis [0], it is using Python GDAL in the development version (from today, I need to test it more).
gdal_merge.py doesn't work with MODIS HDF (probably it is a bug), this is the returned error
Traceback (most recent call last):
File "/usr/local/bin/gdal_merge.py", line 510, in <module>
sys.exit(main())
File "/usr/local/bin/gdal_merge.py", line 390, in main
file_infos = names_to_fileinfos( names )
File "/usr/local/bin/gdal_merge.py", line 122, in names_to_fileinfos
if fi.init_from_name( name ) == 1:
File "/usr/local/bin/gdal_merge.py", line 147, in init_from_name
self.band_type = fh.GetRasterBand(1).DataType
AttributeError: 'NoneType' object has no attribute 'DataType'
[0] https://github.com/lucadelu/pyModis

Categories

Resources