Develop Reference

How to store a dictionary with strings and numbers in a hdf5 file?

Good evening,
I am trying to store a dictionary with strings and numbers in an hdf5 file, but not successfully. I have already stored some datasets with numpy but only with numbers!I have made a research but I can't find a tutorial or something like this.
Thank you very much for your time in advance

The following code saves a dict with some strings and numbers to a hdf5 file.
import numpy as np
import h5py
# Write
dic = {'a': 'astring',
'b': np.arange(10),
'c': 15}
path = 'item/'
with h5py.File('test.h5', 'w') as h5file:
for key, item in dic.items():
# note that not all variable types are supported but string and int are
h5file[path + key] = item
# Read
file = h5py.File('test.h5', 'r')
item = file['item']
keys = item.keys()
out_dict = {}
for key in keys:
out_dict[key] = item[key][()]
This code is based on this stackexchange post. The code in that post makes it possible to store dicts in dicts hierarchically.
Naturally JSON is a very nice hierarchical file format as well, but hdf5 was developed with incremental load/save in mind, which makes it more suitable for my current project.

Error: Set not JSON Serializable while converting TSV File into JSON Format using Python

I'm looking to convert a TSV File I have into JSON Format for mapping (google fusion maps doesn't support mapping multiple objects in the same location so I'm converting it to JSON format to try on Mapbox). Here's my TSV file if you're curious:
https://github.com/yongcho822/Movies-in-the-park/blob/master/MovieParksGeocodeTest.tsv
And here is my corresponding python code thus far:
import json
import csv
def create_map(datafile):
geo_map = {"type":"FeatureCollection"}
item_list = []
with open(datafile, 'r') as tsvfile:
reader = csv.DictReader(tsvfile, delimiter = '\t')
for i, line in enumerate(reader):
data = {}
data['type'] = 'Feature'
data['id'] = i
data['properties']={'title': line['Movie Title'],
'description': line['Amenities'],
'date': line['Date']}
data['name'] = {line['Location']}
data['geometry'] = {'type':'Point',
'coordinates':(line['Lat'], line['Lng'])}
item_list.append(data)
#print item_list
for point in item_list:
geo_map.setdefault('features', []).append(point)
print 'CHECKPOINT'
with open("thedamngeojson.geojson", 'w') as f:
f.write(json.dumps(geo_map))
create_map('MovieParksGeocodeTest.tsv')
It's throwing me an error at the end (after it prints CHECKPOINT), saying
TypeError: set(['Edgebrook Park, Chicago ']) is not JSON serializable
I figure the last two lines is where the error is.. but what's wrong and how do I fix it??

JSON is designed to be a very simple, very portable format; the only kinds of values it understands are strings, numbers, booleans, null (like Python None), objects (like a Python dict) and arrays (like a Python list).
But at least one of the values in your dictionary is a set:
data['name'] = {line['Location']}
Since JSON doesn't have a set type, you get an error telling you that set … is not JSON serializable.
If you don't actually need this to be a set instead of a list (which you probably don't—if it really only ever have one element, who cares which collection type it is?), the easy answer is to change it to be a list:
data['name'] = [line['Location']]
(In fact, even when you do need this to be a set during processing, you usually don't need it to be a set during storage/interchange. If the consumer of your file needs to use it as a set, it can always convert it back later.)

Storing python lmfit object in hdf5 file

I'm trying to store the results of a fit I made using the lmfit package for python in an hdf5 file using the h5py package for python.
Currently I find myself recreating the structure of the data object by hand (i.e. loop over all keys in dictionary, get values and save them).
I have the feeling there has to be a more efficient/pythonic way of saving such an object in an hdf5 file similar to how a pickle of an object would work.
Could anyone help me find a way to efficiently store the information contained in an lmfit.model.ModelFit or lmfit.parameter.Parameters object in an hdf5 file?
edited to show currently used.
def add_analysis_datagroup_to_file(self, group_name='Analysis'):
try:
self.analysis_group= self.f.create_group(group_name)
except ValueError:
print 'Datagroup name "%s" already exists in hdf5 file' %group_name
self.analysis_group = self.f[group_name]
def save_fitted_parameters(self, fit_results=None):
if fit_results is None:
fit_results = self.fit_results
try:
fit_grp = self.analysis_group.create_group('Fitted Params')
except:
fit_grp = self.analysis_group['Fitted Params']
for parname, par in self.fit_results.params.iteritems():
try:
par_group = fit_grp.create_group(parname)
except:
par_group = fit_grp[parname]
par_dict = vars(par)
for val_name, val in par_dict.iteritems():
if val_name == '_val':
val_name = 'value'
if val_name == 'correl' and val is not None:
try:
correl_group = par_group.create_group(val_name)
except:
correl_group = par_group[val_name]
for cor_name, cor_val in val.iteritems():
correl_group.attrs.create(name=cor_name, data=cor_val)
else:
try:
par_group.attrs.create(name=val_name, data=val)
except:
pass

This is quite an old post, but I just had the same problem so hopefully this answer will help someone...
You can use the built in method dumps() in the ModelResult class in lmfit to convert the file into json format which can then be saved as an hdf5 string. You can also use list comprehension to store an array of json files if you want to store multiple fit_results in place. the Parameter class as well as the Model class also have dumps methods.
To reload use loads() (Again, Parameters and Models can be reloaded as they have loads methods as well)
f = h5py.File('fit_result_example.hdf5','w')
grp = f.create_group('group1')
dt = h5py.special_dtype(vlen=str)
fit_results = np.asarray([fit_results.dumps()], dtype=dt)
grp.create_dataset('fit_results', data=fit_results)
f.close()

How to read HDF5 files in Python

I am trying to read data from hdf5 file in Python. I can read the hdf5 file using h5py, but I cannot figure out how to access data within the file.
My code
import h5py
import numpy as np
f1 = h5py.File(file_name,'r+')
This works and the file is read. But how can I access data inside the file object f1?

Read HDF5
import h5py
filename = "file.hdf5"
with h5py.File(filename, "r") as f:
# Print all root level object names (aka keys)
# these can be group or dataset names
print("Keys: %s" % f.keys())
# get first object name/key; may or may NOT be a group
a_group_key = list(f.keys())[0]
# get the object type for a_group_key: usually group or dataset
print(type(f[a_group_key]))
# If a_group_key is a group name,
# this gets the object names in the group and returns as a list
data = list(f[a_group_key])
# If a_group_key is a dataset name,
# this gets the dataset values and returns as a list
data = list(f[a_group_key])
# preferred methods to get dataset values:
ds_obj = f[a_group_key] # returns as a h5py dataset object
ds_arr = f[a_group_key][()] # returns as a numpy array
Write HDF5
import h5py
# Create random data
import numpy as np
data_matrix = np.random.uniform(-1, 1, size=(10, 3))
# Write data to HDF5
with h5py.File("file.hdf5", "w") as data_file:
data_file.create_dataset("dataset_name", data=data_matrix)
See h5py docs for more information.
Alternatives
JSON: Nice for writing human-readable data; VERY commonly used (read & write)
CSV: Super simple format (read & write)
pickle: A Python serialization format (read & write)
MessagePack (Python package): More compact representation (read & write)
HDF5 (Python package): Nice for matrices (read & write)
XML: exists too *sigh* (read & write)
For your application, the following might be important:
Support by other programming languages
Reading / writing performance
Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python

Reading the file
import h5py
f = h5py.File(file_name, mode)
Studying the structure of the file by printing what HDF5 groups are present
for key in f.keys():
print(key) #Names of the root level object names in HDF5 file - can be groups or datasets.
print(type(f[key])) # get the object type: usually group or dataset
Extracting the data
#Get the HDF5 group; key needs to be a group name from above
group = f[key]
#Checkout what keys are inside that group.
for key in group.keys():
print(key)
# This assumes group[some_key_inside_the_group] is a dataset,
# and returns a np.array:
data = group[some_key_inside_the_group][()]
#Do whatever you want with data
#After you are done
f.close()

you can use Pandas.
import pandas as pd
pd.read_hdf(filename,key)

Here's a simple function I just wrote which reads a .hdf5 file generated by the save_weights function in keras and returns a dict with layer names and weights:
def read_hdf5(path):
weights = {}
keys = []
with h5py.File(path, 'r') as f: # open file
f.visit(keys.append) # append all keys to list
for key in keys:
if ':' in key: # contains data if ':' in key
print(f[key].name)
weights[f[key].name] = f[key].value
return weights
https://gist.github.com/Attila94/fb917e03b04035f3737cc8860d9e9f9b.
Haven't tested it thoroughly but does the job for me.

To read the content of .hdf5 file as an array, you can do something as follow
> import numpy as np
> myarray = np.fromfile('file.hdf5', dtype=float)
> print(myarray)

Use below code to data read and convert into numpy array
import h5py
f1 = h5py.File('data_1.h5', 'r')
list(f1.keys())
X1 = f1['x']
y1=f1['y']
df1= np.array(X1.value)
dfy1= np.array(y1.value)
print (df1.shape)
print (dfy1.shape)
Preferred method to read dataset values into a numpy array:
import h5py
# use Python file context manager:
with h5py.File('data_1.h5', 'r') as f1:
print(list(f1.keys())) # print list of root level objects
# following assumes 'x' and 'y' are dataset objects
ds_x1 = f1['x'] # returns h5py dataset object for 'x'
ds_y1 = f1['y'] # returns h5py dataset object for 'y'
arr_x1 = f1['x'][()] # returns np.array for 'x'
arr_y1 = f1['y'][()] # returns np.array for 'y'
arr_x1 = ds_x1[()] # uses dataset object to get np.array for 'x'
arr_y1 = ds_y1[()] # uses dataset object to get np.array for 'y'
print (arr_x1.shape)
print (arr_y1.shape)

from keras.models import load_model
h= load_model('FILE_NAME.h5')

If you have named datasets in the hdf file then you can use the following code to read and convert these datasets in numpy arrays:
import h5py
file = h5py.File('filename.h5', 'r')
xdata = file.get('xdata')
xdata= np.array(xdata)
If your file is in a different directory you can add the path in front of'filename.h5'.

What you need to do is create a dataset. If you take a look at the quickstart guide, it shows you that you need to use the file object in order to create a dataset. So, f.create_dataset and then you can read the data. This is explained in the docs.

Using bits of answers from this question and the latest doc, I was able to extract my numerical arrays using
import h5py
with h5py.File(filename, 'r') as h5f:
h5x = h5f[list(h5f.keys())[0]]['x'][()]
Where 'x' is simply the X coordinate in my case.

use this it works fine for me
weights = {}
keys = []
with h5py.File("path.h5", 'r') as f:
f.visit(keys.append)
for key in keys:
if ':' in key:
print(f[key].name)
weights[f[key].name] = f[key][()]
return weights
print(read_hdf5())
if you are using the h5py<='2.9.0'
then you can use
weights = {}
keys = []
with h5py.File("path.h5", 'r') as f:
f.visit(keys.append)
for key in keys:
if ':' in key:
print(f[key].name)
weights[f[key].name] = f[key].value
return weights
print(read_hdf5())

file IO with defaultdict

I'm attempting to:
load dictionary
update/change the dictionary
save
(repeat)
Problem: I want to work with just 1 dictionary (players_scores)
but the defaultdict expression creates a completely seperate dictionary.
How do I load, update, and save to one dictionary?
Code:
from collections import defaultdict#for manipulating dict
players_scores = defaultdict(dict)
import ast #module for removing string from dict once it's called back
a = {}
open_file = open("scores", "w")
open_file.write(str(a))
open_file.close()
open_file2 = open("scores")
open_file2.readlines()
open_file2.seek(0)
i = input("Enter new player's name: ").upper()
players_scores[i]['GOLF'] = 0
players_scores[i]['MON DEAL'] = 0
print()
scores_str = open_file2.read()
players_scores = ast.literal_eval(scores_str)
open_file2.close()
print(players_scores)

You are wiping your changes; instead of writing out your file, you read it anew and the result is used to replace your players_scores dictionary. Your defaultdict worked just fine before that, even if you can't really use defaultdict here (ast.literal_eval() does not support collections.defaultdict, only standard python literal dict notation).
You can simplify your code by using the json module here:
import json
try:
with open('scores', 'r') as f:
player_scores = json.load(f)
except IOError:
# no such file, create an empty dictionary
player_scores = {}
name = input("Enter new player's name: ").upper()
# create a complete, new dictionary
players_scores[name] = {'GOLF': 0, 'MON DEAL': 0}
with open('scores', 'w') as f:
json.dump(player_scores, f)
You don't need defaultdict here at all; you are only creating new dictionary for every player name anyway.

I think one problem is that to index the data structure the way you want, something like a defaultdict(defaultdict(dict)) is what's really needed — but which unfortunately it's impossible to specify one directly like that. However, to workaround that, all you need to do is define a simple intermediary factory function to pass to the upper-level defaultdict:
from collections import defaultdict
def defaultdict_factory(*args, **kwargs):
""" Create and return a defaultdict(dict). """
return defaultdict(dict, *args, **kwargs)
Then you can use players_scores = defaultdict(defaultdict_factory) to create one.
However ast.literal_eval() won't work with one that's been converted to string representation because it's not one of the simple literal data types the function supports. Instead I would suggest you consider using Python's venerable pickle module which can handle most of Python's built-in data types as well custom classes like I'm describing. Here's an example of applying it to your code (in conjunction with the code above):
import pickle
try:
with open('scores', 'rb') as input_file:
players_scores = pickle.load(input_file)
except FileNotFoundError:
print('new scores file will be created')
players_scores = defaultdict(defaultdict_factory)
player_name = input("Enter new player's name: ").upper()
players_scores[player_name]['GOLF'] = 0
players_scores[player_name]['MON DEAL'] = 0
# below is a shorter way to do the initialization for a new player
# players_scores[player_name] = defaultdict_factory({'GOLF': 0, 'MON DEAL': 0})
# write new/updated data structure (back) to disk
with open('scores', 'wb') as output_file:
pickle.dump(players_scores, output_file)
print(players_scores)