Hdf5ObjectSaver
Related Doc:
package hdf5
How the individual array elements are named in the HDF5 object naming space
How the individual array elements are named in the HDF5 object naming space
How the size of an array is tagged in the HDF5 object naming space
How the size of an array is tagged in the HDF5 object naming space
Close object store, preventing further writes.
Close object store, preventing further writes.
Create a group and make it the group used by any group-relative commands.
Create a group and make it the group used by any group-relative commands.
The most recently created or opened group.
The most recently created or opened group.
The fileId of the opened (ObjectRestorer) or created (ObjectSaver) file
The fileId of the opened (ObjectRestorer) or created (ObjectSaver) file
The sequence of opened hdf5 groups, maintained as a stack.
The sequence of opened hdf5 groups, maintained as a stack.
Close the last created/opened group, making the previous group the one used by any group-relative commands.
Close the last created/opened group, making the previous group the one used by any group-relative commands.
Open an existing group and make it the group used by any group-relative commands.
Open an existing group and make it the group used by any group-relative commands.
Open an hdf5 group given its absolute pathname.
Open an hdf5 group given its absolute pathname.
Write a Float to the object store.
Write a Float to the object store.
Write an Array[Float] to the object store.
Write an Array[Float] to the object store.
Write an Int to the object store.
Write an Int to the object store.
Write an Array[Int] to the object store.
Write an Array[Int] to the object store.
Write an Long to the object store.
Write an Long to the object store.
Write an Array[Long] to the object store.
Write an Array[Long] to the object store.
Write a non-primitive "Saveable" object to the object store.
Write a non-primitive "Saveable" object to the object store.
Write an Array of non-primitive "Saveable" object to the object store.
Write an Array of non-primitive "Saveable" object to the object store.
Write a String to the object store.
Write a String to the object store.
Write an Array[String] to the object store.
Write an Array[String] to the object store.
Cog ComputeGraph saver based on HDF5
First realize that HDF5 provides considerable flexibility in how ones "web of objects" gets mapped to the HDF5 concepts of groups, datasets, attributes, etc. We have taken the approach of not using attributes and mapping the objects to groups and datasets as follows:
- Primitive values (Ints, Floats, Strings, etc.) are stored each in their own dataset, which is named by the writeXXX() call that wrote the primitive. - Arrays of primitive values are similarly stored each in their own named dataset. Only 1D Arrays are currently supported and the length of the array is stored and can be retrieved from the dataspace of the dataset. - Single Objects are created as a group (which is like a directory in the filesystem analogy) with the name given in the writeObject() call. The primitive data members of the object are stored in datasets that are within the object's group. - Arrays of Objects are created with a two-level group hierarchy. First a group is created that has the name provided by the writeObjectArray() call (e.g. "fields"). Then a special dataSet called "fieldsSize" is created within this group to hold the integer length of the array. Finally, each object of the array is stored in the group under a name that includes the array index (e.g. "fields[3]"). Note that the storing of the individual array objects will add a second level to the group hierarchy.
Note that this scheme permits a nesting of Objects and Arrays[Objects].
By example, assume you had the following class:
class Datapoint { val time: Int, val data: Float }
and instance:
val points = new Array[Datapoint](3)
Assume further that the Datpoint class is set up as a "Saveable" that wrote out its fields with names "time" and "data".
Then the call to writeObjectArray("points", points) would generate the following hdf5 datasets:
points/pointsSize points/points[0]/time points/points[0]/data points/points[1]/time points/points[1]/data points/points[2]/time points/points[2]/data
Note that under the hdf5 root group "/", the following groups were created:
points points/points[0] points/points[1] points/points[2]