ocean.util.serialize.contiguous

Format Explained

This package implements a binary serialization format useful for efficient
struct representation in monomorphic environment. All servers are expected to
have similar enough h/w architecture and software to have identical in-memory
representation of D structures. It doesn work as a generic cross-platform
serialization format.

Essential idea here is storing all struct instance data (including all data
transitively accessible via arrays / pointers) in a single contiguous memory
buffer. Which is exactly the reason why the package is named like that. That
way deserialization is very fast and doesn need any memory allocation for
simple cases - all the deserializer needs to do is to iterate through the
memory chunk and update internal pointers.

``contiguous.Deserializer`` returns a memory buffer wrapped in
``Contiguous!(S)`` struct. Such wrapper is guaranteed to conform to the
contiguity expectation explained above. It is recommended to use it in your
application instead of plain ``void[]`` for added type safety.

There are certain practical complications with it that are explained as part of
``contiguous.Serializer`` and ``contiguous.Deserializer`` API docs. Those should
not concern most applications and won be mentioned in the overview.

Available Decorators

`contiguous.VersionDecorator` adds struct versioning information to the basic binary serialization format. It expects struct definitions with additional meta-information available at compile-time and prepends a version number byte before the actual data buffer. Upon loading the serialized data, the stored version number is compared against the expected one and automatic struct conversion is done if needed. It only allows conversion through one version increment/decrement at a time.

`contiguous.MultiVersionDecorator` is almost identical to plain `VersionDecorator` but allows the version increment range to be defined in the constructor. Distinct classes are used so that, if incoming data accidentally is too old, performance-critical applications will emit an error rather than wasting CPU cycles converting through multiple versions. For other aplications multi-version implementation should be more convenient.

API

All methods that do deserialization of data (``Deserializer.deserialize``,
``VersionDecorator.load``, ``VersionDecorator.loadCopy``) return
``Contiguous!(S)`` struct. Lifetime of such struct is identical to lifetime
of buffer used for deserialization. For 1-argument methods it is that argument,
for 2-argument ones it is the destination argument.

To get a detailed overview of serializer API check the modules:

``ocean.util.serialize.contiguous.Serializer``
``ocean.util.serialize.contiguous.Deserializer``

To get a detailed overview of decorator API check the mixins used for its
generation:

``ocean.util.serialize.model.VersionDecoratorMixins``
``ocean.util.serialize.contiguous.model.LoadCopyMixin``

Serializer methods are static because they work only with argument state.
Decorators need to be created as persistent objects because they need an
intermediate state for version conversions.

If a method refers to ``DeserializerReturnType!(Deserializer, S)`` as return
type, you can substitute it with ``Contiguous!(S)`` as it is the return type
used by existing contiguous deserializer.

There is also the ``ocean.util.serialize.contiguous.Util`` module which provides
higher level ``copy`` functions for optimized deep copying of data between
contiguous structs as well as from normal structs to contiguous ones.

Recommended Usage

The contiguous serialization format and the version decorator are primarily designed as a way to interchange D structs between different applications that may expect different versions of those struct layout. It is recommended to completely strip the version information with the help of the decorator upon initially reading a record, and to use the resulting raw contiguous buffer internally in the application (i.e. in a cache). That way you can use any of the serialization/deserialization utilities in the application without thinking about the version meta-data

Typical code pattern for a cache:

1. Define a `Cache` of `Contiguous!(S)` elements.

2. When receiving external data we do `version_decorator.loadCopy!(S)(dht_data, cache_element)`

3. Use `contiguous.Util.copy(cache_element, contiguous_instance)` for copying the struct instance if needed

4. Use `contiguous_instance.ptr` to work with deserialized data as if it was `S*`

It is likely that you will need to change the code to use strongly-typed `Contiguous!(S)` persistent buffer instead of raw `void[]` buffer. Possibly multiple such buffers if the old one was reused for different struct types. This may result in small memory footprint increase at the benefit of better type safety and optimized copy operations.

You can completely abandon the resulting the `Contiguous!(S)` instance and use/store the plain `S*` instead (retrieved via .ptr getter) which will work as long as underlying buffer persists. This is however slightly discouraged because if you will need to copy the data later, it will be impossible to use optimized version without unsafe explicit casts.