I am trying to read data from hdf5 file in Python. I can read the hdf5 file using h5py, but I cannot figure out how to access data within the file.
My code
import h5py
import numpy as np
f1 = h5py.File(file_name,'r+')
This works and the file is read. But how can I access data inside the file object f1?
Read HDF5
import h5py
filename = "file.hdf5"
with h5py.File(filename, "r") as f:
# Print all root level object names (aka keys)
# these can be group or dataset names
print("Keys: %s" % f.keys())
# get first object name/key; may or may NOT be a group
a_group_key = list(f.keys())[0]
# get the object type for a_group_key: usually group or dataset
print(type(f[a_group_key]))
# If a_group_key is a group name,
# this gets the object names in the group and returns as a list
data = list(f[a_group_key])
# If a_group_key is a dataset name,
# this gets the dataset values and returns as a list
data = list(f[a_group_key])
# preferred methods to get dataset values:
ds_obj = f[a_group_key] # returns as a h5py dataset object
ds_arr = f[a_group_key][()] # returns as a numpy array
Write HDF5
import h5py
# Create random data
import numpy as np
data_matrix = np.random.uniform(-1, 1, size=(10, 3))
# Write data to HDF5
with h5py.File("file.hdf5", "w") as data_file:
data_file.create_dataset("dataset_name", data=data_matrix)
See h5py docs for more information.
Alternatives
JSON: Nice for writing human-readable data; VERY commonly used (read & write)
CSV: Super simple format (read & write)
pickle: A Python serialization format (read & write)
MessagePack (Python package): More compact representation (read & write)
HDF5 (Python package): Nice for matrices (read & write)
XML: exists too *sigh* (read & write)
For your application, the following might be important:
Support by other programming languages
Reading / writing performance
Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
Reading the file
import h5py
f = h5py.File(file_name, mode)
Studying the structure of the file by printing what HDF5 groups are present
for key in f.keys():
print(key) #Names of the root level object names in HDF5 file - can be groups or datasets.
print(type(f[key])) # get the object type: usually group or dataset
Extracting the data
#Get the HDF5 group; key needs to be a group name from above
group = f[key]
#Checkout what keys are inside that group.
for key in group.keys():
print(key)
# This assumes group[some_key_inside_the_group] is a dataset,
# and returns a np.array:
data = group[some_key_inside_the_group][()]
#Do whatever you want with data
#After you are done
f.close()
you can use Pandas.
import pandas as pd
pd.read_hdf(filename,key)
Here's a simple function I just wrote which reads a .hdf5 file generated by the save_weights function in keras and returns a dict with layer names and weights:
def read_hdf5(path):
weights = {}
keys = []
with h5py.File(path, 'r') as f: # open file
f.visit(keys.append) # append all keys to list
for key in keys:
if ':' in key: # contains data if ':' in key
print(f[key].name)
weights[f[key].name] = f[key].value
return weights
https://gist.github.com/Attila94/fb917e03b04035f3737cc8860d9e9f9b.
Haven't tested it thoroughly but does the job for me.
To read the content of .hdf5 file as an array, you can do something as follow
> import numpy as np
> myarray = np.fromfile('file.hdf5', dtype=float)
> print(myarray)
Use below code to data read and convert into numpy array
import h5py
f1 = h5py.File('data_1.h5', 'r')
list(f1.keys())
X1 = f1['x']
y1=f1['y']
df1= np.array(X1.value)
dfy1= np.array(y1.value)
print (df1.shape)
print (dfy1.shape)
Preferred method to read dataset values into a numpy array:
import h5py
# use Python file context manager:
with h5py.File('data_1.h5', 'r') as f1:
print(list(f1.keys())) # print list of root level objects
# following assumes 'x' and 'y' are dataset objects
ds_x1 = f1['x'] # returns h5py dataset object for 'x'
ds_y1 = f1['y'] # returns h5py dataset object for 'y'
arr_x1 = f1['x'][()] # returns np.array for 'x'
arr_y1 = f1['y'][()] # returns np.array for 'y'
arr_x1 = ds_x1[()] # uses dataset object to get np.array for 'x'
arr_y1 = ds_y1[()] # uses dataset object to get np.array for 'y'
print (arr_x1.shape)
print (arr_y1.shape)
from keras.models import load_model
h= load_model('FILE_NAME.h5')
If you have named datasets in the hdf file then you can use the following code to read and convert these datasets in numpy arrays:
import h5py
file = h5py.File('filename.h5', 'r')
xdata = file.get('xdata')
xdata= np.array(xdata)
If your file is in a different directory you can add the path in front of'filename.h5'.
What you need to do is create a dataset. If you take a look at the quickstart guide, it shows you that you need to use the file object in order to create a dataset. So, f.create_dataset and then you can read the data. This is explained in the docs.
Using bits of answers from this question and the latest doc, I was able to extract my numerical arrays using
import h5py
with h5py.File(filename, 'r') as h5f:
h5x = h5f[list(h5f.keys())[0]]['x'][()]
Where 'x' is simply the X coordinate in my case.
use this it works fine for me
weights = {}
keys = []
with h5py.File("path.h5", 'r') as f:
f.visit(keys.append)
for key in keys:
if ':' in key:
print(f[key].name)
weights[f[key].name] = f[key][()]
return weights
print(read_hdf5())
if you are using the h5py<='2.9.0'
then you can use
weights = {}
keys = []
with h5py.File("path.h5", 'r') as f:
f.visit(keys.append)
for key in keys:
if ':' in key:
print(f[key].name)
weights[f[key].name] = f[key].value
return weights
print(read_hdf5())
Related
I have downloaded an .h5 file which has various objects (most of them of the format data.dat) including one named History.txt. Upon accessing it, it shows <HDF5 dataset "History.txt": shape (), type "|O">. I am not able to access the text inside this object. Here it says that type "|O" is a reference file. Converting it to np.array is showing an output in which all the lines and text are squashed together. Is there a way to extract/read the text which is there in this object?
The code is as follows:
data_0180 = h5py.File('file.h5', 'r+')
data_0180['OutermostExtraction.dir'].keys()
Output of this has many keys, I've written the first few:
<KeysViewHDF5 ['History.txt', 'Y_l2_m-1.dat', 'Y_l2_m-2.dat', 'Y_l2_m0.dat']>
These .dat keys contain data, while this History.txt contains some kind of information about the file and that data. I want to read that information. When I try to print it:
print(data_0180['OutermostExtraction.dir/History.txt'])
it shows the following output:
<HDF5 dataset "History.txt": shape (), type "|O">
Converting it to np.array shows the following output (I have mentioned only the first couple of lines, the output is large)
array('WaveformModes_4872 = scri.SpEC.read_from_h5("SimulationAnnex/Catalog/NonSpinningSurrogate/
BBH_CFMS_d18_q1_sA_0_0_0_sB_0_0_0/Lev4/rhOverM_Asymptotic_GeometricUnits.h5/Extrapolated_N2.dir",
**{})\n# # WaveformBase.ensure_validity(WaveformModes_4872, alter=True, assertions=True)\n# WaveformModes.ensure_validity(WaveformModes_4872, alter=True,
assertions=True)\n# hostname = kingcrab\n#
cwd = /mnt/raid-project/nr/woodford\n# datetime = 2017-01-16T19:24:54.304846\n# scri.__version__ = 2016.10.10.devc5096f2\n# spherical_functions.__version__ = 2016.08.30.dev77191149\n',
dtype=object)
with the shape of the array as (). How do I extract/read the text in this object?
As mentioned in my comments, you have a variable length string. NumPy doesn't have a dtype to support this data type, so h5py stores with an object dtype (along with some metadata). Adding to the "fun", the array is a scalar array, so you can't access elements with typical NumPy indexing. This is where .item() comes to the rescue. Add .decode() to finish the process. Your code should look something like this:
data_str = np.array(data_0180['OutermostExtraction.dir/History.txt']).item().decode('utf-8')
To demonstrate the behavior (start to finish), I wrote an example starting from the variable length string example in the h5py documentation. First it creates a file ('foo.hdf5') with a dataset of variable length string data -- I used the first 3 lines from your output.
After writing, the file is closed and reopened in read-only mode. The dataset is read into an array (arr), then .item() is used to access the element and .decode('utf-8') to decode it (assumes it was encoded as 'utf-8'). The last line shows how to pull all these methods together in a single Python statement that returns a decoded string (variable data_str). Once you have the string, you can parse it as needed.
Example Below:
import h5py
import numpy as np
vlstr = 'WaveformModes_4872 = scri.SpEC.read_from_h5("SimulationAnnex/Catalog/NonSpinningSurrogate/' + \
'BBH_CFMS_d18_q1_sA_0_0_0_sB_0_0_0/Lev4/rhOverM_Asymptotic_GeometricUnits.h5/Extrapolated_N2.dir",' + \
'**{})\n# # WaveformBase.ensure_validity(WaveformModes_4872, alter=True, assertions=True)\n# WaveformModes.ensure_validity(WaveformModes_4872, alter=True,'
dt = h5py.string_dtype(encoding='utf-8')
with h5py.File('foo.hdf5','w') as h5f:
ds = h5f.create_dataset('VLDS', dtype=dt, data=vlstr)
with h5py.File('foo.hdf5','r') as h5f:
ds = h5f['VLDS']
print('dtype=',ds.dtype)
print('string dtype=',h5py.check_string_dtype(ds.dtype))
arr = np.array(ds)
print('\n',arr.item().decode('utf-8'))
data_str = np.array(h5f['VLDS']).item().decode('utf-8')
print('\n',data_str)
I'm having issues loading (I think storing is working – a file is being created and contains data) a dictionary (string key and array/list value) from a HDF5 file. I'm receiving the following error:
ValueError: malformed node or string: < HDF5 dataset "dataset_1": shape (), type "|O" >
My code is:
import h5py
def store_table(self, filename):
table = dict()
table['test'] = list(np.zeros(7,dtype=int))
with h5py.File(filename, "w") as file:
file.create_dataset('dataset_1', data=str(table))
file.close()
def load_table(self, filename):
file = h5py.File(filename, "r")
data = file.get('dataset_1')
print(ast.literal_eval(data))
I've read online using the ast method literal_eval should work but it doesn't appear to help... How do I 'unpack' the HDF5 so it's a dictionary again?
Any ideas would be appreciated.
It's not clear to me what you really want to accomplish. (I suspect your dictionaries have more than seven zeros. Otherwise, HDF5 is overkill to store your data.) If you have a lot of very large dictionaries, it would be better to covert the data to a NumPy array then either 1) create and load the dataset with data= or 2) create the dataset with an appropriate dtype then populate. You can create datasets with mixed datatypes, which is not addressed in the previous solution. If those situations don't apply, you might want to save the dictionary as attributes. Attributes can be associated to a group, a dataset, or the file object itself. Which is best depends on your requirements.
I wrote a short example to show how to load dictionary key/value pairs as attribute names/value pairs tagged to a group. For this example, I assumed the dictionary has a name key with the group name for association. The process is almost identical for a dataset or file object (just change the object reference).
import h5py
def load_dict_to_attr(h5f, thisdict) :
if 'name' not in thisdict:
print('Dictionary missing name key. Skipping function.')
return
dname = thisdict.get('name')
if dname in h5f:
print('Group:' + dname + ' exists. Skipping function.')
return
else:
grp = h5f.create_group(dname)
for key, val in thisdict.items():
grp.attrs[key] = val
###########################################
def get_grp_attrs(name, node) :
grp_dict = {}
for k in node.attrs.keys():
grp_dict[k]= node.attrs[k]
print (grp_dict)
###########################################
car1 = dict( name='my_car', brand='Ford', model='Mustang', year=1964,
engine='V6', disp=260, units='cu.in' )
car2 = dict( name='your_car', brand='Chevy', model='Camaro', year=1969,
engine='I6', disp=250, units='cu.in' )
car3 = dict( name='dads_car', brand='Mercedes', model='350SL', year=1972,
engine='V8', disp=4520, units='cc' )
car4 = dict( name='moms_car', brand='Plymouth', model='Voyager', year=1989,
engine='V6', disp=289, units='cu.in' )
a_truck = dict( brand='Dodge', model='RAM', year=1984,
engine='V8', disp=359, units='cu.in' )
garage = dict(my_car=car1,
your_car=car2,
dads_car=car3,
moms_car=car4,
a_truck=a_truck )
with h5py.File('SO_61226773.h5','w') as h5w:
for car in garage:
print ('\nLoading dictionary:', car)
load_dict_to_attr(h5w, garage.get(car))
with h5py.File('SO_61226773.h5','r') as h5r:
print ('\nReading dictionaries from Group attributes:')
h5r.visititems (get_grp_attrs)
If I understand what you are trying to do, this should work:
import numpy as np
import ast
import h5py
def store_table(filename):
table = dict()
table['test'] = list(np.zeros(7,dtype=int))
with h5py.File(filename, "w") as file:
file.create_dataset('dataset_1', data=str(table))
def load_table(filename):
file = h5py.File(filename, "r")
data = file.get('dataset_1')[...].tolist()
file.close();
return ast.literal_eval(data)
filename = "file.h5"
store_table(filename)
data = load_table(filename)
print(data)
My preferred solution is just to convert them to ascii and then store this binary data.
import h5py
import json
import itertools
#generate a test dictionary
testDict={
"one":1,
"two":2,
"three":3,
"otherStuff":[{"A":"A"}]
}
testFile=h5py.File("test.h5","w")
#create a test data set containing the binary representation of my dictionary data
testFile.create_dataset(name="dictionary",shape=(len([i.encode("ascii","ignore") for i in json.dumps(testDict)]),1),dtype="S10",data=[i.encode("ascii","ignore") for i in json.dumps(testDict)])
testFile.close()
testFile=h5py.File("test.h5","r")
#load the test data back
dictionary=testFile["dictionary"][:].tolist()
dictionary=list(itertools.chain(*dictionary))
dictionary=json.loads(b''.join(dictionary))
The two key parts are:
testFile.create_dataset(name="dictionary",shape=(len([i.encode("ascii","ignore") for i in json.dumps(testDict)]),1),dtype="S10",data=[i.encode("ascii","ignore") for i in json.dumps(testDict)])
Where
data=[i.encode("ascii","ignore") for i in json.dumps(testDict)])
Converts the dictionary to a list of ascii charecters (The string shape may also be calculated from this)
Decoding back from the hdf5 container is a little simpler:
dictionary=testFile["dictionary"][:].tolist()
dictionary=list(itertools.chain(*dictionary))
dictionary=json.loads(b''.join(dictionary))
All that this is doing is loading the string from the hdf5 container and converting it to a list of bytes. Then I coerce this into a bytes object which I can convert back to a dictionary with json.loads
If you are ok with the extra library usage (json, ittertools) I think this offers a somewhat more pythonic solution (which in my case wasnt a problem since I was using them anyway).
Good evening,
I am trying to store a dictionary with strings and numbers in an hdf5 file, but not successfully. I have already stored some datasets with numpy but only with numbers!I have made a research but I can't find a tutorial or something like this.
Thank you very much for your time in advance
The following code saves a dict with some strings and numbers to a hdf5 file.
import numpy as np
import h5py
# Write
dic = {'a': 'astring',
'b': np.arange(10),
'c': 15}
path = 'item/'
with h5py.File('test.h5', 'w') as h5file:
for key, item in dic.items():
# note that not all variable types are supported but string and int are
h5file[path + key] = item
# Read
file = h5py.File('test.h5', 'r')
item = file['item']
keys = item.keys()
out_dict = {}
for key in keys:
out_dict[key] = item[key][()]
This code is based on this stackexchange post. The code in that post makes it possible to store dicts in dicts hierarchically.
Naturally JSON is a very nice hierarchical file format as well, but hdf5 was developed with incremental load/save in mind, which makes it more suitable for my current project.
I recently changed my religion from Matlab to Python for my data-analysis. I often need to import a lot of data-files which I used to store in a structure array in Matlab. This way I could both store a lot of different data in a single structure-array where the parameters to which the data belongs will be stored within this structure. I would end up with structures like:
data(i).parameterx = a
data(i).parametery = b
data(i).data1 = [3, 5, 7, 8, 423, 2, 56, 6]
data(i).data2 = [4, 2, 5; 4, 6, 2; 5, 1, 3]
The length of data is the number of data-files I imported. This way I can retrieve all the parameters of each data-set based on the index of data.
So I want to do the same in Python. I tried using class but it seems that I can't store it in an array.
What I tried was the following:
#!/usr/bin/env python
# Import modules
import numpy as np
import fnmatch
import os
# Set up all elements of the path
class path:
main, dir = 'root/dir', 'data_dir'
complete = os.path.join(main,dir)
ext = 'odt' # Extension of the file to look for
# Initialize data array and data-class
all_data=[]
class odt_data:
pass
# Loop through all files in path.complete and find all ODT-files
for (root, dirnames, filenames) in os.walk(path.complete):
for filename in fnmatch.filter(filenames, '*.'+path.ext):
# Only ODT-files
# Extract parameters from filename
parameters = filename.split('__')
odt_data.parameterx = parameters[0]; odt_data.parametery = parameters[2]
# Import data from file and assign to correct attribute of 'odt_data'
file=os.path.join(root, filename)
data=np.loadtxt(file)
odt_data.data1 = data[:,0]; odt_data.data2 = data[:,1]
# Append new data to array of all data
all_data.append(odt_data)
The problem here is that is doesn't really save the data in odt_data but rather the reference to it. Hence when I do print(all_data) the result is:
[<class '__main__.odt_data'>, <class '__main__.odt_data'>, <class '__main__.odt_data'>]
And therefore only the last imported data is stored: all_data[0] is exactly the same as all_data[1] and so on. Hence, I'm not able to access the data of the first or second imported file, only that of the last one. So when I call all_data[1].parameterx I get the parameterx value of the last imported file, not the second!
NOTE: I don't care about how it prints, I only care about accessing all the imported data
Anybody knows a way to store the actual data in class odt_data in an array (or possibly use something else than a class)
You just need to initialize the class in the loop.
However, this isn't how you would store such data in Python. You don't benefit from using a class here (in either case). Python actually has structured arrays, but it is overkill in this situation. The simplest solution would be to use a list of dicts:
# Import modules
import numpy as np
import os
main = 'root/dir'
dir = 'data_dir'
complete = os.path.join(main,dir)
ext = '.odt' # Extension of the file to look for
# Initialize data array
all_data = []
for root, _, filenames in os.walk(path.complete):
for filename in filenames:
if os.path.splitext(filename)[-1] != ext:
continue
odt_data = {}
# Only ODT-files
# Extract parameters from filename
odt_data['parameterx'], odt_data['parametery'] = filename.split('__')
# Import data from file and assign to correct attribute of 'odt_data'
fname = os.path.join(root, filename)
data = np.loadtxt(fname)
odt_data['data1'] = data[:,0]
odt_data['data2'] = data[:,1]
# Append new data to array of all data
all_data.append(odt_data)
I saved a cell array as a .mat file in Matlab as follows:
test = {'hello'; 'world!'};
save('data.mat', 'test', '-v7.3')
How can I import it as the list of strings in Python with H5py?
I tried
f = h5py.File('data.mat', 'r')
print f.get('test')
print f.get('test')[0]
This prints out:
<HDF5 dataset "test": shape (1, 2), type "|O8">
[<HDF5 object reference> <HDF5 object reference>]
How can I dereference it to get the list of strings ['hello', 'world!'] in Python?
Writing in Matlab:
test = {'Hello', 'world!'; 'Good', 'morning'; 'See', 'you!'};
save('data.mat', 'test', '-v7.3') % v7.3 so that it is readable by h5py
Reading in Python (works for any number or rows or columns, but assumes that each cell is a string):
import h5py
import numpy as np
data = []
with h5py.File("data.mat") as f:
for column in f['test']:
row_data = []
for row_number in range(len(column)):
row_data.append(''.join(map(unichr, f[column[row_number]][:])))
data.append(row_data)
print data
print np.transpose(data)
Output:
[[u'Hello', u'Good', u'See'], [u'world!', u'morning', u'you!']]
[[u'Hello' u'world!']
[u'Good' u'morning']
[u'See' u'you!']]
This answer should be seen as an addition to Franck Dernoncourt's answer, which totally suffices for all cell arrays that contain 'flat' data (for mat files of version 7.3 and probably above).
I encountered a case where I had nested data (e.g. 1 row of cell arrays inside a named cell array). I managed to get my hands on the data by doing the following:
# assumption:
# idx_of_interest specifies the index of the cell array we are interested in
# (at the second level)
with h5py.File(file_name) as f:
data_of_interest_reference = f['cell_array_name'][idx_of_interest, 0]
data_of_interest = f[data_of_interest_reference]
Reason this works for nested data:
If you look at the type of the dataset you want to retrieve at a deeper level, it says 'h5py.h5r.Reference'. In order to actually retrieve the data the reference points to, you need to provide that reference to the file object.
I know this is an old question. But I found a package to scratch that itch:
hdf5storage
It can be installed by pip and works nicely on python 3.6 for both pre and post 7.3 matlab files. For older files it calls scipy.io.loadmat according to the docs.