Getting started with QBit Microservices Stats KPI Metrics Monitoring

KPI Microservices Monitoring

There has been a lot written on the subject of Microservices Monitoring. Monitoring is a bit of an overloaded term. 
There is service health monitoring, which can be done with tools like Mesosphere/Marathon, Nomad, Consul, etc. There is also KPI monitoring, which is done with tools like Grafana, Graphite, InfluxDB, StatsD, etc. Then there is log monitoring and search with tools like the ELK stack (elastic-search, LogStash and Kibana) and Splunk, where you can easily trace logs down to the requests or client ID in a request header. And, then there is system monitoring (JVM, slow query logs, network traffic, etc.), with tools like SystemD, and more. 
You will want all of this when you are doing Microservices Development.
The more insight you have into your microservices, the easier to support and debug. Microservices imply async distributed development. Doing async distributed development without monitoring is like running with scissors.
To summarize Microservices Monitoring is:
  • KPI Monitoring (e.g., StatsD, Grafana, Graphite, InfluxDB, etc.)
  • Health Monitoring (e.g., Consul, Nomad, Mesosphere/Marathon, Heroku, etc.)
  • Log monitoring (e.g., ELK stack, Splunk, etc.)
QBit has support for ELK/Splunk by providing support for MDC. QBit has support for systems that can monitor health, e.g., Mesosphere/Marathon, Heroku, Consul, Nomad, etc. QBit has an internal health system that QBit's service actors all check-in with. These checks then gets rolled up to other systems like Mesosphere/Marathon, Heroku, Consul, Nomad, etc.
In this tutorial we are going to just cover KPI monitoring for microservices which is sometimes called Metrics Monitoring or Stats Monitoring. KPI stands for Key Performance Indicators. KPIs are the things you really care about to see if your microservice is up and running, and how often, and how its performing.
At the heart of the QBit KPI system is the Metrics collector. QBit uses the Metrik interface for tracking Microservice KPIs.

Metrik Interface for tracking KPIs

public interface MetricsCollector {

    default void increment(final String name) {
        recordCount(name, 1);
    }

    default void recordCount(String name, long count) {
    }

    default void recordLevel(String name, long level) {
    }

    default void recordTiming(String name, long duration) {
    }

}
The above interface is for recording counts (per time period), current level (or gauge at this instance in time) and timings which is how long did something take.

Demonstrating using QBit metrics

This guide assumes you have read through the main overview of QBit and have gone through the first tutorials, but you should be able to follow along if you have not. You just will be able to follow along better if you read the documentation (at least skimmed) and went through the first set of tutorials.
Let's show it. First we need to build. Use Gradle as follows:

gradle.build -- Gradle build system to build microservices monitoring

group 'qbit-ex'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'application'
mainClassName = "com.mammatustech.todo.TodoServiceMain"

compileJava {
    sourceCompatibility = 1.8
}

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'

    /* Used for reactive Java programming. */ 
    compile 'io.advantageous.reakt:reakt:2.5.0.RELEASE'

    /* Used to simplify QBit config and for integration with Consul, StatsD, Heroku, etc. */
    compile group: 'io.advantageous.qbit', name: 'qbit-admin', version: '1.3.0.RELEASE'

    /* Vertx is used as the network driver for QBit. */
    compile group: 'io.advantageous.qbit', name: 'qbit-vertx', version: '1.3.0.RELEASE'

}
Read the comments in all of the code listings.
In this example we will use StatsD but QBit is not limited to StatsD for Microservice KPI monitoring. In fact QBit can do amazing things with its StatsService like clustered rate limiting based on OAuth header info, but that is beyond this tutorial.
StatsD is a protocol that you can send KPI messages to over UDP. Digital Ocean has a really nice description of StatsD.
The easiest way to setup StatsD is to use Docker. Docker is a great tool for development, and using Docker with Nomad, CoreOS, Mesosphere/Marathon, etc. is a great way to deploy Docker containers, but at a minimum you should be using the Docker tools for development.
Please setup the StatsD server stack by using this public docker container.
QBit ships with StatsD support in the qbit-admin lib (jar). It has done this for a long time.
We will connect to StatsD with this URI.

URI to connect to with StatsD

final URI statsdURI = URI.create("udp://192.168.99.100:8125");
Depending on how you have Docker setup, your URI might look a bit different. If you are running Docker tools with a Mac, then the above should be your URI. (On Linux the above URI is likely to be localhost not 192.168.99.100, go through the docker tool tutorials if you are lost at this point. It will be worth your time.)
If you have not already, follow the instructions at statsD, grafana, influxdb docker container docs.

Running Docker

docker run -d \
  --name docker-statsd-influxdb-grafana \
  -p 3003:9000 \
  -p 3004:8083 \
  -p 8086:8086 \
  -p 22022:22 \
  -p 8125:8125/udp \
  samuelebistoletti/docker-statsd-influxdb-grafana
The above yields

Servers

Host        Port        Service

3003        9000            grafana        to see the results
8086        8086            influxdb       to store the results 
3004        8083            influxdb-admin to query the results
8125        8125            statsd         server that listens to statsD UPD messages
22022       22              sshd
If you want to see the metrics and see if this is working, go through the influxDB tutorial and look around at the measurements with the influx-admin. Influx is a time series database. Grafana allows you to see graphs and charts of the microservice KPIs that we are collecting. You will want to learn grafana to visualize the data. QBit makes the metrics data query-able so you can implement back pressure, cloud scaling triggers and rate limiting easily. 
We use the host and port of the URI to connect to the StatsD daemon that is running on the docker container.

Setting up StatsD by using QBit managedServiceBuilder

        managedServiceBuilder.getStatsDReplicatorBuilder().setHost(host).setPort(port);
        managedServiceBuilder.setEnableStatsD(true);
We covered using and setting up the managedServiceBuilder in the first tutorials, and the complete code listing is below.

Use the managedServiceBuilder to create the StatsCollector/MetricsCollector

        StatsCollector statsCollector = managedServiceBuilder.createStatsCollector();

        /* Start the service. */
        managedServiceBuilder.addEndpointService(new TodoService(reactor, statsCollector))
The QBit StatsCollector interface extends the Metrik MetricsCollector interface (from QBit 1.5 onwards).

Using the StatsCollector.

Then we just need to use it.

Using the StatsCollector to collect KPIs about our service

For kicks, we track the KPI todoservice.i.am.alive every three seconds. Then we collect metrics when the add method is called and when the remove method is called. Read through the code listings. QBit does a good job of capturing and batch sending the results to reduce overhead while collecting your microservice KPIs. 

Tracking KPI i.am.am.alive

@RequestMapping("/todo-service")
public class TodoService {
...
        /** Send stat count i.am.alive every three seconds.  */
        this.reactor.addRepeatingTask(Duration.ofSeconds(3),
                () -> statsCollector.increment("todoservice.i.am.alive"));

Tracking calls to add method

    @RequestMapping(value = "/todo", method = RequestMethod.POST)
    public void add(final Callback<Boolean> callback, final Todo todo) {

        /** Send KPI add.called every time the add method gets called. */
        statsCollector.increment("todoservice.add.called");
        todoMap.put(todo.getId(), todo);
        callback.accept(true);
    }

Tracking calls to remove method

    @RequestMapping(value = "/todo", method = RequestMethod.DELETE)
    public void remove(final Callback<Boolean> callback, final @RequestParam("id") String id) {

        /** Send KPI add.removed every time the remove method gets called. */
        statsCollector.increment("todoservice.remove.called");
        Todo remove = todoMap.remove(id);
        callback.accept(remove!=null);

    }

Managing callbacks and repeating tasks

    @QueueCallback({EMPTY, IDLE, LIMIT})
    public void process() {
        reactor.process();
    }

Complete example

Todo.java

package com.mammatustech.todo;

public class Todo {

    private  String id;

    private final String name;
    private final String description;
    private final long createTime;

    public Todo(String name, String description, long createTime) {
        this.name = name;
        this.description = description;
        this.createTime = createTime;

        this.id = name + "::" + createTime;
    }


    public String getId() {
        if (id == null) {
            this.id = name + "::" + createTime;
        }
        return id;
    }

    public String getName() {
        return name;
    }

    public String getDescription() {
        return description;
    }

    public long getCreateTime() {
        return createTime;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Todo todo = (Todo) o;

        if (createTime != todo.createTime) return false;
        return !(name != null ? !name.equals(todo.name) : todo.name != null);

    }

    @Override
    public int hashCode() {
        int result = name != null ? name.hashCode() : 0;
        result = 31 * result + (int) (createTime ^ (createTime >>> 32));
        return result;
    }
}

TodoService.java to show tracking KPIs.

package com.mammatustech.todo;

import io.advantageous.qbit.annotation.*;
import io.advantageous.qbit.reactive.Callback;
import io.advantageous.qbit.service.stats.StatsCollector;
import io.advantageous.reakt.reactor.Reactor;

import java.time.Duration;
import java.util.*;

import static io.advantageous.qbit.annotation.QueueCallbackType.*;


@RequestMapping("/todo-service")
public class TodoService {


    private final Map<String, Todo> todoMap = new TreeMap<>();

    /** Used to manage callbacks and such. */
    private final Reactor reactor;

    /** Stats Collector for KPIs. */
    private final StatsCollector statsCollector;

    public TodoService(Reactor reactor, StatsCollector statsCollector) {
        this.reactor = reactor;
        this.statsCollector = statsCollector;

        /** Send stat count i.am.alive every three seconds.  */
        this.reactor.addRepeatingTask(Duration.ofSeconds(3),
                () -> statsCollector.increment("todoservice.i.am.alive"));

        this.reactor.addRepeatingTask(Duration.ofSeconds(1), statsCollector::clientProxyFlush);
    }


    @RequestMapping(value = "/todo", method = RequestMethod.POST)
    public void add(final Callback<Boolean> callback, final Todo todo) {

        /** Send KPI add.called every time the add method gets called. */
        statsCollector.increment("todoservice.add.called");
        todoMap.put(todo.getId(), todo);
        callback.accept(true);
    }



    @RequestMapping(value = "/todo", method = RequestMethod.DELETE)
    public void remove(final Callback<Boolean> callback, final @RequestParam("id") String id) {

        /** Send KPI add.removed every time the remove method gets called. */
        statsCollector.increment("todoservice.remove.called");
        Todo remove = todoMap.remove(id);
        callback.accept(remove!=null);

    }



    @RequestMapping(value = "/todo", method = RequestMethod.GET)
    public void list(final Callback<ArrayList<Todo>> callback) {

        /** Send KPI add.list every time the list method gets called. */
        statsCollector.increment("todoservice.list.called");

        callback.accept(new ArrayList<>(todoMap.values()));
    }


    @QueueCallback({EMPTY, IDLE, LIMIT})
    public void process() {
        reactor.process();
    }


}

TodoServiceMain.java showing how to configure StatsD QBit for MicroService KPI tracking

package com.mammatustech.todo;


import io.advantageous.qbit.admin.ManagedServiceBuilder;
import io.advantageous.qbit.service.stats.StatsCollector;
import io.advantageous.reakt.reactor.Reactor;

import java.net.URI;
import java.time.Duration;
import java.util.Objects;

public class TodoServiceMain {


    public static void main(final String... args) throws Exception {

        //To test locally use https://hub.docker.com/r/samuelebistoletti/docker-statsd-influxdb-grafana/
        final URI statsdURI = URI.create("udp://192.168.99.100:8125");

        //For timer
        final Reactor reactor = Reactor.reactor();


        /* Create the ManagedServiceBuilder which manages a clean shutdown, health, stats, etc. */
        final ManagedServiceBuilder managedServiceBuilder =
                ManagedServiceBuilder.managedServiceBuilder()
                        .setRootURI("/v1") //Defaults to services
                        .setPort(8888); //Defaults to 8080 or environment variable PORT

        enableStatsD(managedServiceBuilder, statsdURI);

        StatsCollector statsCollector = managedServiceBuilder.createStatsCollector();



        /* Start the service. */
        managedServiceBuilder.addEndpointService(new TodoService(reactor, statsCollector)) //Register TodoService
                .getEndpointServerBuilder()
                .build().startServer();

        /* Start the admin builder which exposes health end-points and swagger meta data. */
        managedServiceBuilder.getAdminBuilder().build().startServer();

        System.out.println("Todo Server and Admin Server started");

    }

    /**
     * Enable Stats D.
     *
     * @param host statsD host
     * @param port statsD port
     */
    public static void enableStatsD(ManagedServiceBuilder managedServiceBuilder, String host, int port) {
        if (port < 1) throw new IllegalStateException("StatsD port must be set");
        Objects.requireNonNull(host, "StatsD Host cannot be null");
        if (host.isEmpty()) throw new IllegalStateException("StatsD Host name must not be empty");
        managedServiceBuilder.getStatsDReplicatorBuilder().setHost(host).setPort(port);
        managedServiceBuilder.setEnableStatsD(true);
    }

    /**
     * Enable Stats D.
     *
     * @param uri for statsd
     */
    public static void enableStatsD(ManagedServiceBuilder managedServiceBuilder, URI uri) {
        if (!uri.getScheme().equals("udp")) throw new IllegalStateException("Scheme must be udp");
        enableStatsD(managedServiceBuilder, uri.getHost(), uri.getPort());
    }
}



Read more
Comments