(MODAClouds) OpenTSDB

Overview

Initially built at StumbleUpon, OpenTSDB is a software service that provides both storage and querying of monitoring information, while keeping intact the inserted readings, and coping without special archival procedures. [OpenTSDB-1]

Model

According to the available, but succinct, documentation [OpenTSDB-2], a reading is modeled as follows:

name

A string in the form of <service>.<observable>, which denotes the type (or class) of a particular reading, but not the actual source. (Thus the term "name" is somewhat improperly used.)

timestamp

The usual "seconds since UNIX epoch".

value

Either an integer or double, quantifying the observable characteristic.

tags

Key-value pairs (both are strings), which should pin-point the source of the particular metric type. Thus with this tag the system is able to manage multiple metrics of the same type ("name"), from the same service, but from different contained entities, such as the space used for each individual database held inside the same server. Moreover it could be used to hierarchically organize the readings, such as application, cloud, region, availability, VM, etc.

Observations

Underneath OpenTSDB there lies the columnar-database HBase --- which itself runs over Hadoop --- therefore the scalability properties of OpenTSDB. However this also implies that in order to use OpenTSDB, we need to deploy, configure and maintain a Hadoop cluster, including the HDFS and HBase sub-systems, which could translate both in initial cost and deployment time overheads. For real-world application it might be the most suitable solution, especially for those with high monitoring reading rates.

On the other hand, besides the HBase requirement, running OpenTSDB is trivial, as the service acts only as a thin adaptation layer over HBase, being able to run with multiple instances --- as to fulfill the scalability requirements --- without any configuration or interaction between the instances. In fact [OpenTSDB-3] suggests running different instances of the service for read than for writes.

Interacting with OpenTSDB can be done either through an HTTP-based dialogue, a text-based dialogue over TCP, or a more efficient binary protocol over TCP (currently in development).

However, according to [OpenTSDB-3], reading data out of the storage, tends not to be as efficient, compared with the writing rate, even in the case of dedicated reader instances. This in turn could heavily impact our various ModaClouds tools that need to extract and interpret the data.

...

References