Projections
The XOOM LATTICE way to project state, Domain Events, and other Source types into a CQRS Query Model.
Last updated
The XOOM LATTICE way to project state, Domain Events, and other Source types into a CQRS Query Model.
Last updated
Projections are an important part of the component set required to support CQRS, whether or not you are using Event Sourcing. Recall that in CQRS there are two models, one called the Command Model and another called the Query Model.
The Command Model is often referred to as the "Write Model" and the Query Model as the "Read Model." These AKAs are quite inaccurate because you actually read from and write to both models. We prefer the names Command Model and Query Model because these adhere to the pattern name, CQRS, and express the purpose of each of the two models.
A projection is a component that takes one set of states and transforms those into another state. This is typically how it works:
The Command Model causes a state mutation and emits one or more Domain Events
The new Domain Events are persisted into a Journal
The emitted Domain Events are dispatched/routed to a set of Projection components based on their interest in a given Domain Event type
The Projection components interpret how each Domain Event should be reflected into the portion of the Query Model for which they are responsible, and project that interpretation into the persistent Query Model
In all of the above steps, you can replace "Domain Event" with "entity state," because you may decide not to use Domain Events in your Command Model. In other words, the full state of a Command Model entity may be interpreted for projection into the Query Model. Of course if you use Event Sourcing you will always project Domain Events into the Query Model.
With the XOOM LATTICE projection components, we provide the means to project an entity's state and various concrete Source<?> types—DomainEvent and Command, for example—into a CQRS query model.
There are two steps required to use projections. You must implement one or more projections and you must configure those projections to be used within your service/application. These two topics are covered below.
The components used to manage the data filtering, routing, and the actual projections, must be created. The following are the component types.
The ProjectToDescription is a description of how data and events are filtered and selected. You may configure a ProjectToDescription for projecting State<T> and also for projecting Source<T> instances, such as DomainEvent types. Both are described below.
The Dispatcher and ProjectionDispatcher to be registered with the storage mechanism is created by means of an actor that implements both protocols. Two such are the BinaryProjectionDispatcherActor and TextProjectionDispatcherActor, and used when a binary store or text store is used, respectively.
The store is created and Dispatcher registered with it.
The following describes each of the above steps.
ProjectToDescription for State<T>A ProjectToDescription may be defined for state data and for events. The following shows a description defined for a state object.
State data may have associated Metadata, and the Metadata instance may have a named operation that caused the state to transition to the current value. The above description indicates that if the state data contains an operation that matches "User:new", "User:contact", or "User:name", the dispatcher will route the data to the UserProjectionActor.
The following demonstrates how the operation name is associated with the new state. In this case the command is to replace the user's name with a different name. The transitioned state and the descriptive operation is applied. After the state is persisted, the Dispatcher is given the opportunity to route the new state to a projection. As seen in the description of the previous code snippet, matching the "User:name" operation will cause the new state to be routed to the UserProjectionActor.
We provide a comprehensive example of using this full-state projection technique, known as the vlingo-http-frontservice.
ProjectToDescription for Source<T>A ProjectToDescription may be defined for state data and for events. The following shows a description defined for events.
The above description indicates that if a concrete DomainEvent is one of the types ForumStarted, ForumClosed, or ForumModeratorAssigned, the dispatcher will route the event to the ForumProjectionActor.
The following demonstrates how the ForumStarted event occurs. After the event is persisted, the Dispatcher is given the opportunity to route it to a projection. As seen in the description of the previous code snippet, matching the ForumStarted event will cause it to be routed to the ForumProjectionActor.
Note however that keeping the ProjectToDescription up to date with new and changing event types could be error prone by being overlooked. Instead of referencing each event type, you may instead reference the Java package that contains all of the events.
Dispatcher and ProjectionDispatcherThe following demonstrates how the ProjectToDescription, Dispatcher, ProjectionDispatcher, TextProjectionDispatcherActor, and Journal are wired for use.
You may want to have an isolated component to configure your projections, such as this ProjectionDispatcherProvider example.
There are three primary objects created.
Notable are the three ProjectToDescription instances created, which are formed into a single List<ProjectToDescription>. Each of the descriptions indicate how various command model changes will be dispatched into the actor instance whose type is the first parameter. For example, this DashboardProjectionActor instance will be created inside the ProjectionDispatcher and all DomainEvent types defined as inner classes of the com.dashdawn.model.dashboard.Events class, will be dispatched to it. The same goes for the WaveBoardDetailsProjectionActor and the inner Events types defined in its package.
The descriptions are then used to create the ProjectionDispatcher, which is implemented by the TextProjectionDispatcherActor provided by XOOM LATTICE. Actually this actor implements two protocols, both the Dispatcher and the ProjectionDispatcher. The Dispatcher is used by the Journal to dispatch newly appended events to the ProjectionDispatcher. (This is implemented by the abstract base ProjectionDispatcherActor, which is extended by the concrete actors, TextProjectionDispatcherActor and BinaryProjectionDispatcherActor). The two protocol references answered by actorFor() are captured in the Protocols type.
Lastly the ProjectionDispatcherProvider instance is created, where both protocol references are held and available to dependents.
The following is an example of how the Dispatcher is used, where the command model storage will use it to dispatch events into the projections. Note that the command model store knows nothing about projections, only the Dispatcher.
The above code may be placed in your Bootstrap or Startup class containing a main().
Projection InterfaceThe following is an example of a Projection designed to handle state mutation operations. It corresponds to the above section ProjectToDescription for State<T>.
The Projectable contains the data to be projected. The ProjectionControl is used to confirm that the Projectable has been projected so that the Dispatcher will not route it again. The code inside each of the cases would create or update the views effected by the state mutations.
The full example of the above projection approach is in the xoom-http-frontservice.
The following is an example of a Projection designed to handle events. This example corresponds to the above section ProjectToDescription for Source<T>.
Since there may be one or more events that were applied, persisted, dispatched, and routed to the projection, the projection must iterate over all entries so as not to miss any. The code inside each of the cases would create or update the views effected by the events. A much cleaner approach follows.
StateStoreProjectionActorThere is an Actor abstract base type for this: StateStoreProjectionActor. This is a much more powerful abstract than directly using the Projection interface. As the name indicates, this projection base is used to project into a StateStore.
Since object-relational mapping and other object persistence approaches are less efficient than key-value stores, we have decided not to support ORM-based projections at this time.
Here is the a condensed code listing of the StateStoreProjectionActor.
In the following documentation we explain a number of available overridable base class methods. Note that these are all optional, except that one of the two merge(...) methods should be overridden.
Override the following method if you are performing full-state projections:
protected T merge(
final T previousData,
final int previousVersion,
final T currentData,
final int currentVersion)
Override the following method if you are performing projections for Event Sourcing:
protected T merge(
final T previousData,
final int previousVersion,
final T currentData,
final int currentVersion,
final List> sources)
The default implementation behaviors of all non-merge methods will very likely serve your needs fully.
To create a new type of projection, extend StateStoreProjectionActor, as demonstrated next. This example overrides the merge(...) that is best used with Event Sourcing.
This projection is next examined in detail, starting with the constructor.
The DashboardProjectionActor constructor uses the super constructor that takes only the StateStore used to write and read DashboardView instances. This single-parameter constructor provides a default StateAdapter and EntryAdapter for the concrete projection to use. These are part of XOOM SYMBIO and are known as DefaultTextStateAdapter and DefaultTextEntryAdapter.
There is a second constructor for StateStoreProjectionActor that takes an additional two parameters, one for the StateAdapter and one for the EntryAdapter.
These adapters are used to adapt from persistent State and Entry types to the model types. When the single argument constructor is used you get the default StateAdapter and EntryAdapter.
Since there is only one EntryAdapter supported, but no doubt several or many DomainEvent types needed, your own custom EntryAdapter must internally hold the full number of EntryAdapter instances needed for all DomainEvent types. This might use a Map with keys of Class<? extends DomainEvent> and values of concrete EntryAdapter instances. The outer EntryAdapter registered with the StateStoreProjectionActor would operate by looking up the specific concrete EntryAdapter by DomainEvent type, dispatch to it, and answer its return value.
The DashboardProjectionActor implements the Projection protocol, but there is a default implementation of method projectWith() provided by the abstract base classStateStoreProjectionActor. The implementation of this method is quite simple for most uses. If you need a more involved implementation you may override the default provided by the abstract base class.
The simple step is to "upsert" (i.e. insert or update, depending on existence) to current Projectable, which may hold a state, or entries, or both.
The upsertFor() does the following.
Asks currentDataFor(projectable) for its notion of the current state, as in the state that is now being projected. You may override currentDataFor(projectable). This may or may not be useful, but if it is, you must return the value that is considered current from the Projectable. This method is generally used only for full-state projections rather than those based on DomainEvent instances.
Requests that the projection prepareForMergeWith(). The default behavior adapts any entries() in the Projectable to Source<?> instances, holding them in an internal List<Source<?>>. The concrete extender may get the adapted Source<?> instances using the sources() method. These are provided by default with the second merge(...) method that is used for Event Sourcing projections. If you have some alternative or additional preparation to do before the merge this method must be overridden.
Reads the current T value from the StateStore (see the last statement in the method) and passes a BiConsumer<T, Integer> upserter. When the read completes, the upserter is used to merge the previousData with the currentData and/or apply any Source<?> instances.
Writes the merged state back to the StateStore if alwaysWrite() is true or if the newly merged data is not equal to the previous data. This happens inside the upserter.
The DashboardProjectionActor implements currentDataFor() as follows.
As previously indicated, the currentDataFor() may or may not be useful in terms of providing a new state. This implementation answers an Empty instance of the DashboardView, which will be used by the merge() as an accumulator of modifications. This particular override example is unnecessary but demonstrates an alternative to the default implementation. Assume that we decide not to implement the above override and remove the Empty value declaration as well as the currentDataFor() method override.
If you are not projecting from events, but the full state of the domain object, an implementation of currentDataFor() and merge() similar to the previous section is necessary.
Next the DashboardProjectionActor is given the opportunity to prepare for the merge.
In this implementation the prepareForMergeWith() adapts all Entry instances into instances of the DomainEvent specialization IdentifiedDomainEvent. This gives the concrete event types the means to provide the identity of the event instance. As indicated above, this specific implementation of prepareForMergeWith() is redundant because it does the same that the default behavior already does. You would choose to override prepareForMergeWith() only if you required behavior beyond the above implementation.
Immediately following prepareForMergeWith() the upsert() calls dataIdFor().
As seen above, the dataId is provided by the first (and possibly only) instance of the IdentifiedDomainEvent in the events list. Recall that this events list was just previously populated by the prepareForMergeWith(). This example dataIdFor() method override is likely redundant because the default behavior provides a more thorough implementation, first checking for the Projectable dataId(), and if that is not available, looks for the identity from one of the first IdentifiedDomainEvent. If neither of these are available then the dataId will be a blank String, and an override of this method would be necessary to provide a custom value.
If you are using the default implementation of prepareForMergeWith(), the source/event instances are retrieved as follows. Here again, your DomainEvent instances must be of the extends IdentifiedDomainEvent type, which can provide the identity() of the entity that emitted them.
The following is the DashboardProjectionActor implementation of merge() and its helper method mergeInto(), assuming that the prepareForMergeWith() was also overridden and a local List<DomainEvent> of events was collected by it. This example shows an alternative to using the more appropriate merge(...) override that provides the List<Source<?>> parameter.
The merge() passes the currentData to be used by mergeInto() as an accumulator of the new state. Any preexisting state on the currentData that coincides with the interpretation of the DomainEvent will be overwritten/replaced.
Because the Projectable may have multiple entries (serialized DomainEvent instances), the mergeInto() iterates over the entire collection of events deserialized by the method prepareForMergeWith().
Assuming you are using the default prepareForMergeWith() implementation, you would iterate as follows, using the sources() method rather than a collection that you gathered on your own.
The switch uses an enum of DashboardViewProjectableType to match on each of the DomainEvent instances.
The DomainEvent type is matched to the enum by the match() method. A non-match produces the Unmatched type, which is ignored by mergeInto(), but logged as a warning.
Once a specific type is matched, the mergeInto() maps event attributes to DashboardView attributes. Any preexisting values that are mapped are overwritten on the view accumulator. The iteration and matching is done for any number of events, which may be included with the current Projectable or others received in the future.
Note that the design of the DashboardView to directly use public mutable attributes was done to keep the implementation simple, albeit somewhat "dangerous" according to opinions. It doesn't seem like a bad thing in this specific case because the view object is used in very specific limited ways. Feel free to include accessor methods for reading and writing in your designs.
When the merge() completes, the modified DashboardView is written to the StateStore.
The StateStoreProjectionActor<T> provides a default means to determine the current version of the data to be merged and then updated to the database. It works like this:
In other words, by default the data version provided by the Projectable is used as the version for the current data to be merged and updated to the database. This is how all of the above examples obtain the current data version for inserts/updates. It is based on a policy to always update the current merged data. That is, the data is always written no matter whether it is the same as the previous data, or different.
Yet, in some cases you may want to use a different versioning policy and also conditionally write the newly merged data instance only if it is different from the previous data. This is useful for idempotent operations, such as when the Projectable is redelivered due to a latent or failed delivery confirmation. In such cases the StateStoreProjectionActor<T> provides a secondary policy, which is to automatically increment the previous version by one and to also test the current merged data with the previous data to qualify the need to update the data to the database.
To use the secondary policy, you should override the alwaysWrite() method. By default it returns true. Your override should return false instead.
The second method involved in the policy is currentDataVersionFor(), which you need not override if the secondary policy works for your specific projection use case.
As you can see, the currentDataVersionFor() provides both policies in one line of code. If alwaysWrite() answers true, the projectable.dataVersion() is returned as was explained above. Otherwise, when alwaysWrite() answers false, the next sequential version from the one previously stored in the databased is returned. Note that in the case where there was no previous data in the database for the given identity, the previousVersion is -1. Thus, there is the one special case where 1 must be explicitly returned as the initial version.
As previously indicated, using the secondary version and write policy is useful when your use case must be idempotent. Yet, there is another case to consider. If you are projecting multiple entity states and/or events into a single projection data instance, the versions from the various instances will almost certainly collide with each other.
For example, ProductDefined from one Product entity will have version 1, and ProductDefined from a different Product entity will also have version 1. This is unavoidable because ProductDefined is always the first event emitted from a Product. If you attempt to merge the ProductDefined that is delivered after the first initially delivered ProductDefined, the merged/updated data will have version 1 just as the previous data did. Writing this version will cause an optimistic concurrency violation in the StateStore, because the version will make the new data look as if it is based on stale state. By overriding alwaysWrite() to answer false, your current/merged data version will always be one greater than the previous data version.
But what about the idempotent check? In the case where alwaysWrite() answers false, there will be a pre-update check requiring the current data to be different from the previous data.
Notice specifically this statement.
The data difference check is managed by using the equals() of the data's type. The updating write is performed only when the current and previous data are not equal. Note that the equals() implementation must check for value equality (compare type equality as well as equality of all corresponding data attributes/fields) rather than entity equality (compare ids only).
