# Telemetry

Use XOOM Telemetry for measure performance, throughput, and latency, for your project components. This is currently an experimental tool and at this time supports XOOM Actors by way of tapping mailboxes.

The following is a brief usage guide.

## Usage Guide

XOOM Telemetry provides a telemetry registry that can be used within any actor to push metrics to your APM or monitoring system. By default, XOOM Telemetry uses an implementation backed by [Micrometer](https://micrometer.io/), so most common backends are supported, including JMX, Prometheus, and StatsD.

### Registering the Telemetry Registry

XOOM Telemetry provides a `Telemetry` object that will be used by actors to push metrics. The easiest way to register the `Telemetry` object is by using the `DefaultTelemetryProvider`, which will create a `MicrometerTelemetry` that exports all it's metrics to JMX. The snippet to configure telemetry follows.

```java
// Probably you will already have a World defined with your own configuration :)
World world = World.startWithDefaults("measurements");

// The DefaultTelemetryProvider uses Micrometer
TelemetryProvider<MeterRegistry> telemetryProvider = new DefaultTelemetryProvider();

// Provides a Telemetry registry based on the created world
Telemetry<MeterRegistry> telemetry = telemetryProvider.provideFrom(world);

// Registers the telemetry object in the world
world.registerDynamic("telemetry", telemetry);
```

### Sending Metrics From Actors

All actors hosted by XOOM Actors live in a `Stage`, each of which is part of a `World`. All actors through their `World` can access the `Telemetry` provider and send metrics to the registry asynchronously, thus reducing the performance impact of monitoring actors.

The `Telemetry` object is usually acquired by the actor during the instantiation, as it will be needed during the whole lifecycle of the actor. A single instance is all that is needed. To provide some context, consider the following example.

Here's a `Counter` protocol to be implemented by an actor. The actor will count seconds and, on each second, send a metric with the counter to the registry.

```java
public interface Counter {
    void count();
}
```

The protocol now requires and actor to implement it in order to provide behavior. Create a new `CounterActor` class that, implementing `Counter`, performs the business of counting. First note the constructor.

```java
public class CounterActor extends Actor implements Counter {
    private final Telemetry<?> telemetry;
    private final String name;

    public CounterActor(String name) {
        this.telemetry = Telemetry.from(this.stage().world()); // <-- getting the telemetry!
        this.name = name;
    }

    public void count() {
        // will focus later on this
    }
}
```

Because the `Telemetry` object lives in the same `World` as the actor, the instance is available from the actor's `World` reference. By getting the registered `Telemetry`, it's now possible to send metrics at will from inside the actor. Next consider the `count()` behavior.

```java
public void count() {
    telemetry.count("Count", 1, Telemetry.Tag.of("Name", name));
}
```

That's it.

One-liners deserve explanation, even though the telemetry design is rather intuitive. The telemetry count method contains three parameters:

| Parameter   | Explanation                                                                                                                                                                                                                                            | Example Value                          |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------- |
| Metric Name | The name of the metric that will be registered in the `MetricRegistry`. It should be unique.                                                                                                                                                           | `"Count"`                              |
| `Delta`     | The amount to add to the metric in the registry. This value can be negative if the desire is to subtract an amount. For example, when the metric is `5`, and a delta of `1` is given when using `count()`, the total in the registry will then be `6`. | `1`                                    |
| `Tags`      | List of tags (`vararg` in Java) that will be sent to the metric registry. Depending on the metric registry, later, metrics can be grouped and filtered using these tags.                                                                               | `Telemetry.Tag.of("MyTag", "MyValue")` |

{% hint style="success" %}
If you are using the default telemetry provider, as in this example, you can see the metrics using the [JConsole](https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html) that comes with your JDK. Connect to the local process using debugging.
{% endhint %}

![JConsole targeting a local process with a Counter.](https://3691394259-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LLB-V2sJmANuWISDmBf%2F-MPLpgP6eFy_VmNBYjXy%2F-MPLq3hLfjgra_pDaRfG%2Fimage.png?alt=media\&token=bf320f66-6177-477c-bb81-a9f769f6a3fa)

The `Telemetry` provide three different methods to send metrics, with different use cases. We've already seen `count` .

| Method  | Parameters                                      | Description                                                                                                                                                                              |
| ------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `count` | `String metric, int delta, Tag... tags`         | Increases the metric by "delta". Generates percentiles and averages.                                                                                                                     |
| `gauge` | `String metric, int delta, Tag... tags`         | Increases the metric by `delta`. Does not generate any additional metric.                                                                                                                |
| `time`  | `String metric, Callable callable, Tag... tags` | Registers the amount of time required for the callable to finish and returns the result. It (still) does not support asynchronous methods. You will need to use `gauge` to measure them. |

### Integration With Actors

XOOM Actors provide a plugin mechanism that allows XOOM Telemetry to supply a mailbox provider, and thus automatically measure the amount of messages and processing time, per actor. This is really useful for monitoring systems in production, as the low number of messages and processing time in actors indicates that the system is healthy.

The recommended way to plug in the mailbox telemetry is before starting the `World`. The easiest way to accomplish that is by using the `xoom-actors.properties` file. Adding the plugin should be enough to have all the metrics ready to read through JMX.

```
plugin.name.telemetry = true
plugin.telemetry.classname = io.vlingo.xoom.telemetry.TelemetryPlugin

plugin.name.mailboxTelemetry = true
plugin.mailboxTelemetry.classname = io.vlingo.xoom.telemetry.plugin.mailbox.MailboxTelemetryPlugin
```

The plugin provides the following metrics aggregated by the actor class.  All metrics are tagged with the address of the actor, so metrics can be gathered also by actor.

| Metric                          | Description                                                                     |
| ------------------------------- | ------------------------------------------------------------------------------- |
| `pending`                       | Number of messages pending in the mailbox                                       |
| `idle`                          | Number of times the actor tried to query the mailbox but it was empty           |
| `failed.send.EXCEPTION_NAME`    | Number of times an actor could not send a message due to `EXCEPTION_NAME`       |
| `failed.deliver.EXCEPTION_NAME` | Number of times XOOM Actors could not deliver a message due to `EXCEPTION_NAME` |

![Pending messages metric for our CounterActor in JConsole.](https://3691394259-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LLB-V2sJmANuWISDmBf%2F-MPM6kQJYpzjqk2Fxp6a%2F-MPM78Jf1T9Q2wFTFHl_%2Fimage.png?alt=media\&token=4e38c95d-563d-4296-95cf-6aaddc5d1c04)

{% hint style="success" %}
Using the properties file is recommended most of the times as it simplifies configuration and can be handled relatively easily and versioned without changing Java code. However, to use the `xoom-actors.properties` file, you will need to start your world with the `start()` method instead of `startWithDefaults()`. Any previously used plugin configuration using Java code is no longer needed as it is replaced by the properties configuration.
{% endhint %}

{% hint style="success" %}
Because the VLINGO XOOM platform is built entirely on actors, all internal actors are also monitored, so metrics for the `DeadLettersActor` (for example) can also be monitored as easy as server/application actors. This will be plugged in for ***all*** actors: `Journal` instances, XOOM HTTP servers, queue consumers, and all others can be monitored with no additional effort.
{% endhint %}

### Using a Custom Micrometer Provider

XOOM Telemetry, by default, exports all the information to JMX. However, in production setups, you might desire to export metrics to more reliable platforms, such as Prometheus, StatsD, CloudWatch, etc. Micrometer already works out of the box with all of the aforementioned, so XOOM Telemetry can be adapted to export your service/application metrics as well.

To do so, create a new `TelemetryProvider` and implement the `provideFrom()` method to return a new `Telemetry` object that is set up for the desired observation console. Here's an example for DataDog.

```java
public class DatadogTelemetryProvider implements TelemetryProvider<MeterRegistry> {
    @Override
    public Telemetry<MeterRegistry> provideFrom(World world) {
        DatadogConfig config = new DatadogConfig() {
            @Override
            public Duration step() {
                return Duration.ofSeconds(10);
            }

            @Override
            public String apiKey() {
                return "YOUR_API_KEY";
            }

            @Override
            public String get(String k) {
                return null; // accept the rest of the defaults
            }
        };

        MeterRegistry registry = new DatadogMeterRegistry(config, Clock.SYSTEM);
        return new MicrometerTelemetry(registry);
    }
}
```

{% hint style="warning" %}
XOOM Telemetry does not provide all Micrometer dependencies, only the core. If you want to use DataDog with Micrometer, you will need to add the dependency to your `build.gradle` or `pom.xml` as explained in the [documentation for Micrometer](https://micrometer.io/docs/registry/datadog). The same goes for other observation tools.
{% endhint %}

Given the use of DataDog, define the appropriate provider in the properties file.

```
plugin.name.telemetry = true
plugin.telemetry.classname = io.vlingo.xoom.telemetry.TelemetryPlugin
plugin.telemetry.providerClass = com.myapp.infrastructure.DatadogTelemetryProvider
```

Our team welcomes your feedback and contributions for improving the XOOM Telemetry functionality.