1.1   Introduction

In the context of the ModaClouds project, the monitoring data coming from either the application, its resources, and possibly the support services, are used mainly to support QoS monitoring, SLA monitoring and enforcement. Thus most of the employed algorithms and techniques use only "sufficiently recent" data --- for example a sliding window, or a ring buffer, fed from the monitoring data stream --- thus the value of "historical" monitoring data is less important. Moreover the from a technical point of view, the handling of monitoring data seems to move in the following direction: once the readings are collected they are quickly transfered to a C-SPARQL instance, which is the single data source to be used by all the tools requiring access to monitoring data.

However there might be use cases where the operator of an application might want to store, and possibly access on-line, monitoring data spanning over a large period of time --- for example more than a week --- which in turn would require a technical solution adapted for such a situation.

Fortunately there are a few out-of-the box solutions built specifically for such a purpose, which we shall briefly describe in what follows. Thus, if during the course of the project we deem as necessary to provide monitoring data archival, we have out our disposal a few potential candidates that could be quickly integrated in the ModaClouds ecosystem.

OpenTSDB

Initially built at StumbleUpon, and released as open source, it is one of the most mature solutions, capable of handling huge amounts of data (as in TBi). All this by leveraging Hadoop's HBase, itself possibly over Hadoop's HDFS or another distributed file system. As features OpenTSDB provides both data storage, data access, and minimal graphing support. [OpenTSDB-1]

Graphite

Starting as a replacement of the venerable RRDtool, it aimed at being semantically close to it, but technically more efficient, by allowing fast writes of many metrics from many sources (in the thousands), which with earlier versions of RRDtool was sub-optimal. However as with RRDtool the original data is not retained past a particular point in time, but aggregated into increasingly coarse grained statistics. On the upside it also provides a useful dashboard-like interface, allowing the operator to browse through various graphs of the available metrics. [Graphite-1]

Cube

Cube is another recent open-source project, that provides support for storage and query of monitoring data, by leveraging another NoSQL database, namely MongoDB. However it is still a young project, not deployed nearly as much as the previous solutions. [Cube-1]

Generic time series databases

Another option would be to turn towards time-series databases (like for example IBM's Informix), which were specifically built for such purposes. Unfortunately, to our knowledge, there are no production ready open-source solutions, thus it would prove impractical for our project to integrate such a solution. [SDS-Benchmarks]

Generic SQL, NoSQL, or other embedded databases

If everything else fails, we can fallback to general purpose solutions, but we would need to provide a wrapping service, and the scalability would be highly impacted. However embedded databases like Berkeley DB (Oracle), or LevelDB (Google) might prove the best candidates to start with, as would provide the greatest flexibility, although they aren't themselves distributed. Regarding relational databases, we would expect that they would perform much more poorly due to their inherent features, such as transactions or constraints, that are not strictly required for our usecase. [SDS-Benchmarks]

1.2   Dedicated solutions

1.3   Customized solutions

...