KairosDB is a time series database built on top of Cassandra, one of the top NoSQL database. It has a very simple API and a nice Grafana plugin in order to display the time series.
Let’s start with the easy part. Building a 3 nodes Cassandra cluster and a Grafana server using docker via a docker-compose file. Read this post in order to get more information about Cassandra.
A newer version of this docker-compose using the version 3 of docker-compose is available here.
version: '2' services: ############################### cassandra0: image: cassandra:3.0.10 container_name: cassandra0 ports: - 9042:9042 - 9160:9160 - 7199:7199 - 8778:8778 environment: - CASSANDRA_START_RPC=true ############################### cassandra1: image: cassandra:3.0.10 container_name: cassandra1 ports: - 9142:9042 links: - cassandra0:seed environment: - CASSANDRA_SEEDS=seed ############################### cassandra2: image: cassandra:3.0.10 container_name: cassandra2 ports: - 9242:9042 links: - cassandra0:seed environment: - CASSANDRA_SEEDS=seed ############################## grafana: image: grafana/grafana:master container_name: grafana ports: - 3000:3000
Start the system via a docker-compose up -d command and the cluster should be ready.
Dockerizing KairosDB
There is no pre packaged version of Kairos DB. You can use mine which is hosted on docker hub. This is basically the same as the one from wangdrew but with a more recent version of Kairos.
The following lines must be added to the docker-compose.yml file.
kairosdb: image:snuids/kairosdb:v1.1.3 container_name: kairosdb environment: - CASSANDRA_HOST_LIST=cassandra0:9160 ports: - "4242:4242" - "8083:8083" links: - cassandra0
In order to let KairosDB connect to Cassandra, the thrift port of Cassandra must be opened. This can be done using nodetool inside the container as shown in the following screenshot.
You can now start KairosDB via docker-compose and browse to the following url:
Inserting data into KairosDB
It is possible to insert data into KairosDB via the Rest API. The following python 2 program will create a time series per bicycle station of the town of Brussels. Each time series will display the number of available bikes. (Code here)
import time import urllib2 import json import re import requests from datetime import datetime def fetch_villo(): url = 'http://opendata.bruxelles.be/api/records/1.0/search/?dataset=stations-villo-disponibilites-en-temps-reel&rows=1000&facet=banking&facet=bonus&facet=status&facet=contract_name' h = urllib2.urlopen(url) res= h.read() print res; res=res.replace("\u0","") data = json.loads(res) timepoints=[]; for station in data["records"]: onepoint={} onepoint["name"]=station["fields"]["name"] onepoint["value"]=station["fields"]["available_bikes"] onepoint["timestamp"]=int(time.time()*1000) onepoint["tags"]={} onepoint["tags"]["banking"]=station["fields"]["banking"] onepoint["tags"]["status"]=station["fields"]["status"] timepoints.append(onepoint) print "Bulk ready." print json.dumps(timepoints) r = requests.post("http://localhost:8083/api/v1/datapoints", data=json.dumps(timepoints)) print(r.status_code, r.reason) print "Bulk gone." for i in range(0,100): print '*'*80 fetch_villo(); time.sleep(30) print '*'*80
It is now possible to use the internal KairosDB client to display the time series.
Using Grafana to view KairosDB data
It is possible to add a KairosDB plugin into Grafana. The easiest way to do it it to execute the following command inside the grafana container.
grafana-cli plugins install grafana-kairosdb-datasource
Restart the Grafana container, and then create the new data source and a new dashboard.
2 Pingback