/** @defgroup yaml_serialization YAML Serialization
@ingroup technical_notes
Overview
Drake provides infrastructure for reading YAML files into C++ structs, and
writing C++ structs into YAML files. These functions are often used to read
or write configuration data, but may also be used to serialize runtime data
such as Diagram connections or OutputPort traces. Any C++ struct to be
serialized must provide a @ref implementing_serialize "Serialize()" function
to enumerate its fields.
Examples
Given a struct definition:
@code{cpp}
struct MyData {
...
double foo{0.0};
std::vector bar;
};
@endcode
Loading
Given a YAML data file:
@code{yaml}
foo: 1.0
bar: [2.0, 3.0]
@endcode
We can use LoadYamlFile() to load the file:
@code{cpp}
int main() {
const MyData data = LoadYamlFile("filename.yaml");
std::cout << fmt::format("foo = {:.1f}\n", data.foo);
std::cout << fmt::format("bar = {:.1f}\n", fmt::join(data.bar, ", "));
}
@endcode
Output:
@code{txt}
foo = 1.0
bar = 2.0, 3.0
@endcode
Saving
We can use SaveYamlFile() to save to a file:
@code{cpp}
int main() {
MyData data{4.0, {5.0, 6.0}};
SaveYamlFile("filename.yaml", data);
}
@endcode
Output file:
@code{yaml}
foo: 4.0
bar: [5.0, 6.0]
@endcode
The following sections explain each of these steps in more detail, along with
the customization options that are available for each one.
@anchor implementing_serialize
Implementing Serialize
Any C++ struct to be serialized must provide a templated `Serialize()` function
that enumerates the fields. Typically, `Serialize()` will be implemented via a
member function on the struct, but if necessary it can also be a free function
obtained via argument-dependent lookup.
Here is an example of implementing a Serialize member function:
@code{cpp}
struct MyData {
template
void Serialize(Archive* a) {
a->Visit(DRAKE_NVP(foo));
a->Visit(DRAKE_NVP(bar));
}
double foo{0.0};
std::vector bar;
};
@endcode
Structures can be arbitrarily nested, as long as each `struct` has a
`Serialize()` function:
@code{cpp}
struct MoreData {
template
void Serialize(Archive* a) {
a->Visit(DRAKE_NVP(baz));
a->Visit(DRAKE_NVP(quux));
}
std::string baz;
std::map quux;
};
@endcode
For background information about visitor-based serialization, see also the
Boost.Serialization Tutorial, which served as the inspiration for Drake's
design.
Style guide for Serialize
By convention, we place the Serialize function prior to the data members per
the
styleguide rule. Each data member has a matching `Visit` line in the
Serialize function, in the same order as the member fields appear.
By convention, we declare all of the member fields as public, since they are
effectively so anyway (because anything that calls the Serialize function
receives a mutable pointer to them). The typical way to do this is to declare
the data as a `struct`, instead of a `class`.
However, if
the
styleguide rule for struct vs class points towards using a `class` instead,
then we follow that advice and make it a `class`, but we explicitly label the
member fields as `public`. We also omit the trailing underscore from the field
names, so that the Serialize API presented to the caller of the class is
indifferent to whether it is phrased as a `struct` or a `class`.
See drake::schema::Gaussian for an example of this situation.
If the member fields have invariants that must be immediately enforced during
de-serialization, then we add invariant checks to the end of the `Serialize()`
function to enforce that, and we mark the class fields private (adding back the
usual trailing underscore). See drake::math::BsplineBasis for an example of
this situation.
Built-in types
Drake's YAML I/O functions provide built-in support for many common types:
- bool
- double
- float
- int32_t
- int64_t
- uint32_t
- uint64_t
- std::array
- std::map
- std::optional
- std::string
- std::unordered_map
- std::variant
- std::vector
- Eigen::Matrix (including 1-dimensional matrices, i.e., vectors)
YAML correspondence
The simple types (`std::string`, `bool`, floating-point number, integers) all
serialize to a Scalar
node in YAML.
The array-like types (`std::array`, `std::vector`, `Eigen::Matrix`) all
serialize to a Sequence
node in YAML.
User-defined structs and the native maps (`std::map`, `std::unordered_map`) all
serialize to a Mapping
node in YAML.
For the treatment of `std::optional`, refer to
@ref serialize_nullable "Nullable types", below.
For the treatment of `std::variant`, refer to
@ref serialize_variant "Sum types", below.
Reading YAML files
Use LoadYamlFile() or LoadYamlString() to de-serialize YAML-formatted string
data into C++ structure.
It's often useful to write a helper function to load using a specific schema,
in this case the `MyData` schema:
@code{cpp}
MyData LoadMyData(const std::string& filename) {
return LoadYamlFile(filename);
}
int main() {
const MyData data = LoadMyData("filename.yaml");
std::cout << fmt::format("foo = {:.1f}\n", data.foo);
std::cout << fmt::format("bar = {:.1f}\n", fmt::join(data.bar, ", "));
}
@endcode
Sample data in `filename.yaml`:
@code{yaml}
foo: 1.0
bar: [2.0, 3.0]
@endcode
Sample output:
@code{txt}
foo = 1.0
bar = 2.0, 3.0
@endcode
There is also an option to load from a top-level child in the document:
@code{yaml}
data_1:
foo: 1.0
bar: [2.0, 3.0]
data_2:
foo: 4.0
bar: [5.0, 6.0]
@endcode
@code{cpp}
MyData LoadMyData2(const std::string& filename) {
return LoadYamlFile(filename, "data_2");
}
@endcode
Sample output:
@code{txt}
foo = 4.0
bar = 5.0, 6.0
@endcode
Defaults
The LoadYamlFile() function offers a `defaults = ...` argument. When provided,
the yaml file's contents will overwrite the provided defaults, but any fields
that are not mentioned in the yaml file will remain intact at their default
values.
When merging file data atop any defaults, any `std::map` or `std::unordered_map`
collections will merge the contents of the file alongside the existing map
values, keeping anything in the default that is unchanged. Any other
collections such as `std::vector` are entirely reset, even if they already
had some values in place (in particular, they are not merely appended to).
Merge keys
YAML's "merge keys" (https://yaml.org/type/merge.html) are supported during
loading. (However, the graph-aliasing relationship implied by nominal YAML
semantics is not implemented; the merge keys are fully deep-copied.)
Example:
@code{yaml}
_template: &common_foo
foo: 1.0
data_1:
<< : *common_foo
bar: [2.0, 3.0]
data_2:
<< : *common_foo
bar: [5.0, 6.0]
@endcode
Writing YAML files
Use SaveYamlFile() or SaveYamlString() to output a YAML-formatted serialization
of a C++ structure.
The serialized output is always deterministic, even for unordered datatypes such
as `std::unordered_map`.
@code{cpp}
struct MyData {
template
void Serialize(Archive* a) {
a->Visit(DRAKE_NVP(foo));
a->Visit(DRAKE_NVP(bar));
}
double foo{0.0};
std::vector bar;
};
int main() {
MyData data{1.0, {2.0, 3.0}};
std::cout << SaveYamlString(data, "root");
return 0;
}
@endcode
Output:
@code{yaml}
root:
foo: 1.0
bar: [2.0, 3.0]
@endcode
Document root
Usually, YAML reading or writing requires a serializable struct that matches
the top-level YAML document. However, sometimes it's convenient to parse the
document in the special case of a C++ `std::map` at the top level, without
the need to define an enclosing struct.
@code{yaml}
data_1:
foo: 1.0
bar: [2.0, 3.0]
data_2:
foo: 4.0
bar: [5.0, 6.0]
@endcode
@code{cpp}
std::map LoadAllMyData(const std::string& filename) {
return LoadYamlFile>(filename);
}
@endcode
@anchor serialize_nullable
Nullable types (std::optional)
When a C++ field of type `std::optional` is present, then:
- when saving its enclosing struct as YAML data, if the optional is nullopt,
then no mapping entry will be emitted.
- when load its enclosing struct from YAML data, if no mapping entry is present
in the YAML, then it is not an error.
@anchor serialize_variant
Sum types (std::variant)
When reading into a std::variant<>, we match its
YAML tag
to the shortened C++ class name of the variant selection. For example, to read
into this sample struct:
@code{cpp}
struct Foo {
template
void Serialize(Archive* a) {
a->Visit(DRAKE_NVP(data));
}
std::string data;
};
struct Bar {
template
void Serialize(Archive* a) {
a->Visit(DRAKE_NVP(value));
}
std::variant value;
};
@endcode
Some valid YAML examples are:
@code{yaml}
# For the first type declared in the variant<>, the tag is optional.
bar:
value: hello
# YAML has built-in tags for string, float, int.
bar2:
value: !!str hello
# For any other type within the variant<>, the tag is required.
bar3:
value: !!float 1.0
# User-defined types use a single exclamation point.
bar4:
value: !Foo
data: hello
@endcode
*/