Telemetry

The Reactive metrics collection toolkit for the VLINGO XOOM platform.

Use XOOM Telemetry for measure performance, throughput, and latency, for your project components. This is currently an experimental tool and at this time supports XOOM Actors by way of tapping mailboxes.

The following is a brief usage guide.

Usage Guide

XOOM Telemetry provides a telemetry registry that can be used within any actor to push metrics to your APM or monitoring system. By default, XOOM Telemetry uses an implementation backed by Micrometer, so most common backends are supported, including JMX, Prometheus, and StatsD.

Registering the Telemetry Registry

XOOM Telemetry provides a Telemetry object that will be used by actors to push metrics. The easiest way to register the Telemetry object is by using the DefaultTelemetryProvider, which will create a MicrometerTelemetry that exports all it's metrics to JMX. The snippet to configure telemetry follows.

// Probably you will already have a World defined with your own configuration :)
World world = World.startWithDefaults("measurements");

// The DefaultTelemetryProvider uses Micrometer
TelemetryProvider<MeterRegistry> telemetryProvider = new DefaultTelemetryProvider();

// Provides a Telemetry registry based on the created world
Telemetry<MeterRegistry> telemetry = telemetryProvider.provideFrom(world);

// Registers the telemetry object in the world
world.registerDynamic("telemetry", telemetry);

Sending Metrics From Actors

All actors hosted by XOOM Actors live in a Stage, each of which is part of a World. All actors through their World can access the Telemetry provider and send metrics to the registry asynchronously, thus reducing the performance impact of monitoring actors.

The Telemetry object is usually acquired by the actor during the instantiation, as it will be needed during the whole lifecycle of the actor. A single instance is all that is needed. To provide some context, consider the following example.

Here's a Counter protocol to be implemented by an actor. The actor will count seconds and, on each second, send a metric with the counter to the registry.

public interface Counter {
    void count();
}

The protocol now requires and actor to implement it in order to provide behavior. Create a new CounterActor class that, implementing Counter, performs the business of counting. First note the constructor.

public class CounterActor extends Actor implements Counter {
    private final Telemetry<?> telemetry;
    private final String name;

    public CounterActor(String name) {
        this.telemetry = Telemetry.from(this.stage().world()); // <-- getting the telemetry!
        this.name = name;
    }

    public void count() {
        // will focus later on this
    }
}

Because the Telemetry object lives in the same World as the actor, the instance is available from the actor's World reference. By getting the registered Telemetry, it's now possible to send metrics at will from inside the actor. Next consider the count() behavior.

public void count() {
    telemetry.count("Count", 1, Telemetry.Tag.of("Name", name));
}

That's it.

One-liners deserve explanation, even though the telemetry design is rather intuitive. The telemetry count method contains three parameters:

Parameter	Explanation	Example Value
Metric Name	The name of the metric that will be registered in the MetricRegistry. It should be unique.	"Count"
Delta	The amount to add to the metric in the registry. This value can be negative if the desire is to subtract an amount. For example, when the metric is 5, and a delta of 1 is given when using count(), the total in the registry will then be 6.	1
Tags	List of tags (vararg in Java) that will be sent to the metric registry. Depending on the metric registry, later, metrics can be grouped and filtered using these tags.	Telemetry.Tag.of("MyTag", "MyValue")

If you are using the default telemetry provider, as in this example, you can see the metrics using the JConsole that comes with your JDK. Connect to the local process using debugging.

JConsole targeting a local process with a Counter.

The Telemetry provide three different methods to send metrics, with different use cases. We've already seen count .

Method	Parameters	Description
count	String metric, int delta, Tag... tags	Increases the metric by "delta". Generates percentiles and averages.
gauge	String metric, int delta, Tag... tags	Increases the metric by delta. Does not generate any additional metric.
time	String metric, Callable callable, Tag... tags	Registers the amount of time required for the callable to finish and returns the result. It (still) does not support asynchronous methods. You will need to use gauge to measure them.

Integration With Actors

XOOM Actors provide a plugin mechanism that allows XOOM Telemetry to supply a mailbox provider, and thus automatically measure the amount of messages and processing time, per actor. This is really useful for monitoring systems in production, as the low number of messages and processing time in actors indicates that the system is healthy.

The recommended way to plug in the mailbox telemetry is before starting the World. The easiest way to accomplish that is by using the xoom-actors.properties file. Adding the plugin should be enough to have all the metrics ready to read through JMX.

plugin.name.telemetry = true
plugin.telemetry.classname = io.vlingo.xoom.telemetry.TelemetryPlugin

plugin.name.mailboxTelemetry = true
plugin.mailboxTelemetry.classname = io.vlingo.xoom.telemetry.plugin.mailbox.MailboxTelemetryPlugin

The plugin provides the following metrics aggregated by the actor class. All metrics are tagged with the address of the actor, so metrics can be gathered also by actor.

Metric	Description
pending	Number of messages pending in the mailbox
idle	Number of times the actor tried to query the mailbox but it was empty
failed.send.EXCEPTION_NAME	Number of times an actor could not send a message due to EXCEPTION_NAME
failed.deliver.EXCEPTION_NAME	Number of times XOOM Actors could not deliver a message due to EXCEPTION_NAME

Pending messages metric for our CounterActor in JConsole.

Using the properties file is recommended most of the times as it simplifies configuration and can be handled relatively easily and versioned without changing Java code. However, to use the xoom-actors.properties file, you will need to start your world with the start() method instead of startWithDefaults(). Any previously used plugin configuration using Java code is no longer needed as it is replaced by the properties configuration.

Because the VLINGO XOOM platform is built entirely on actors, all internal actors are also monitored, so metrics for the DeadLettersActor (for example) can also be monitored as easy as server/application actors. This will be plugged in for all actors: Journal instances, XOOM HTTP servers, queue consumers, and all others can be monitored with no additional effort.

Using a Custom Micrometer Provider

XOOM Telemetry, by default, exports all the information to JMX. However, in production setups, you might desire to export metrics to more reliable platforms, such as Prometheus, StatsD, CloudWatch, etc. Micrometer already works out of the box with all of the aforementioned, so XOOM Telemetry can be adapted to export your service/application metrics as well.

To do so, create a new TelemetryProvider and implement the provideFrom() method to return a new Telemetry object that is set up for the desired observation console. Here's an example for DataDog.

public class DatadogTelemetryProvider implements TelemetryProvider<MeterRegistry> {
    @Override
    public Telemetry<MeterRegistry> provideFrom(World world) {
        DatadogConfig config = new DatadogConfig() {
            @Override
            public Duration step() {
                return Duration.ofSeconds(10);
            }

            @Override
            public String apiKey() {
                return "YOUR_API_KEY";
            }

            @Override
            public String get(String k) {
                return null; // accept the rest of the defaults
            }
        };

        MeterRegistry registry = new DatadogMeterRegistry(config, Clock.SYSTEM);
        return new MicrometerTelemetry(registry);
    }
}

XOOM Telemetry does not provide all Micrometer dependencies, only the core. If you want to use DataDog with Micrometer, you will need to add the dependency to your build.gradle or pom.xml as explained in the documentation for Micrometer. The same goes for other observation tools.

Given the use of DataDog, define the appropriate provider in the properties file.

plugin.name.telemetry = true
plugin.telemetry.classname = io.vlingo.xoom.telemetry.TelemetryPlugin
plugin.telemetry.providerClass = com.myapp.infrastructure.DatadogTelemetryProvider

Our team welcomes your feedback and contributions for improving the XOOM Telemetry functionality.