KairosDB is a time series database built on top of Cassandra, one of the top NoSQL database. It has a very simple API and a nice Grafana plugin in order to display the time series.

Let’s start with the easy part. Building a 3 nodes Cassandra cluster and a Grafana server using docker via a docker-compose file. Read this post in order to get more information about Cassandra.

version: '2'
 services:

###############################
 cassandra0:
 image: cassandra:3.0.10
 container_name: cassandra0
 ports:
 - 9042:9042
 - 9160:9160
 - 7199:7199
 - 8778:8778
 environment:
 - CASSANDRA_START_RPC=true

###############################
 cassandra1:
 image: cassandra:3.0.10
 container_name: cassandra1
 ports:
 - 9142:9042
 links:
 - cassandra0:seed
 environment:
 - CASSANDRA_SEEDS=seed

###############################
 cassandra2:
 image: cassandra:3.0.10
 container_name: cassandra2
 ports:
 - 9242:9042
 links:
 - cassandra0:seed
 environment:
 - CASSANDRA_SEEDS=seed

##############################
 grafana:
 image: grafana/grafana:master
 container_name: grafana
 ports:
 - 3000:3000

Start the system via a docker-compose up -d command and the cluster should be ready.

Dockerizing KairosDB

There is no pre packaged version of Kairos DB. You can use mine which is hosted on docker hub. This is basically the same as the one from wangdrew but with a more recent version of Kairos.

The following lines must be added to the docker-compose.yml file.

   kairosdb:
    image:snuids/kairosdb:v1.1.3
    container_name: kairosdb
    environment:
      - CASSANDRA_HOST_LIST=cassandra0:9160
    ports:
      - "4242:4242"
      - "8083:8083"
    links:
      - cassandra0

In order to let KairosDB connect to Cassandra, the thrift port of Cassandra must be opened. This can be done using nodetool inside the container as shown in the following screenshot.

cassandrathrift

You can now start KairosDB via docker-compose and browse to the following url:

http://localhost:8083/

kairosdb2

Inserting data into KairosDB

It is possible to insert data into KairosDB via the Rest API. The following python 2 program will create a time series per bicycle station of the town of Brussels. Each time series will display the number of available bikes. (Code here)

import time
import urllib2
import json
import re
import requests

from datetime import datetime

def fetch_villo():
    url = 'http://opendata.bruxelles.be/api/records/1.0/search/?dataset=stations-villo-disponibilites-en-temps-reel&rows=1000&facet=banking&facet=bonus&facet=status&facet=contract_name'

    h = urllib2.urlopen(url)
    res= h.read()
    print res;
    res=res.replace("\u0","")
    data = json.loads(res)

    timepoints=[];
    for station in data["records"]:

        onepoint={}
        onepoint["name"]=station["fields"]["name"]
        onepoint["value"]=station["fields"]["available_bikes"]
        onepoint["timestamp"]=int(time.time()*1000)
        onepoint["tags"]={}
        onepoint["tags"]["banking"]=station["fields"]["banking"]
        onepoint["tags"]["status"]=station["fields"]["status"]

        timepoints.append(onepoint)

    print "Bulk ready."
    print json.dumps(timepoints)
    r = requests.post("http://localhost:8083/api/v1/datapoints", data=json.dumps(timepoints))
    print(r.status_code, r.reason)

    print "Bulk gone."

for i in range(0,100):
    print '*'*80
    fetch_villo();
    time.sleep(30)
    print '*'*80

It is now possible to use the internal KairosDB client to display the time series.

kairosdb

Using Grafana to view KairosDB data

It is possible to add a KairosDB plugin into Grafana. The easiest way to do it it to execute the following command inside the grafana container.

grafana-cli plugins install grafana-kairosdb-datasource

grafanakairos

Restart the Grafana container, and then create the new data source and a new dashboard.

This slideshow requires JavaScript.