(MODAClouds) Execution sub-systems

Observations

• the ModaClouds run-time execution environment is the one providing support for various functionalities needed for controlling and monitoring the deployed application; it must be seen as distinct from the ModaClouds application environment which is most likely provided by a PaaS;

• below we use the term "observe" instead of "monitor", as it implies only health checks and bare minimum metrics (most likely provided by the OS itself);

• all the items below are services part of the ModaClouds run-time execution environment; some of them fulfill direct requirements by the ModaClouds project, some other fulfill low level operations on behalf of the higher level services and aren't directly exposed to the user or other tools;

• we define the "core services" as made of those basic or strictly required to run any other service; this includes:
  • object service;
  • logging service;
  • notifications service;
  • credentials service;
  • HTTP gateway;
• most of these services could also have the following extra functionalities:
  • provide a HTTP-based programmatic interface (e.g. RPC-style, SOAP/WSDL, or RESTful);
  • provide a HTTP-based web user interface to allow direct interaction with the operator;
• most of these services run in a constrained environment, and are isolated from the other services; see the "environment", for example:
  • native unconstrained: runs directly inside the host OS, and has complete access to it (i.e. it runs as root with full capabilities);
  • native constrained: runs directly inside the host OS, but doesn't have any special rights (most likely it runs as a separate user, possibly in a chroot);
  • container: it runs completely isolated from the other services (i.e. LXC);
• some of these services actually constitute distributed systems, in either a clustered or peer-to-peer manner (e.g. nodes controller, discovery and quorum, hub, etc.);

Services

Nodes controller

• description: controls and observes each VM part of the run-time execution environment;
• functionalities:
  • mediates interaction with some OS artifacts (i.e. firewall, DNS services, etc.);
  • bootstraps and starts the other services;
• interactions:
  • core services;
  • discovery service: to announce each VM availability;
• environment: native unconstrained;
• responsible: TBD (IeAT);
• technologies:
  • ... (???);
• priority: medium;

Resources controller

• description: controls and obverves resources from the cloud provider needed by the run-time execution environment; (it doesn't concert itself with the application resources but it could be extended to handle that;)
• functionalities:
  • creates and destroys resources (i.e. VM's, DFS like S3, etc.);
  • exposes various information (except monitoring) about the committed resources;
• interactions:
  • core services;
  • environment: container;
• responsible: Silviu (IeAT);
• references:
  • ... (Silviu???);
• technologies:
  • ... (Silviu???);
• priority: medium;

Services controller

• description: deploys, controls and observes the various services which compose the run-time execution environment;
• functionalities:
  • provides a frontend for controlling (creating, destroying, snapshotting, migrating, etc.) services;
  • exposes various information (including some basic monitoring information) about services;
• interactions:
  • container controller: for delegating the creation and destruction of the actual service OS-level environment;
  • core services;
• environment: native constrained;
• responsible: TBD (IeAT);
• technologies:
  • ... (???);
• priority: medium;
• observations:
  • it doesn't directly handles all the aspects, and it delegates most of the OS-level tasks to other services (like the container controller);

Application controller

• description: deploys, controls and observes the application, which most likely runs ontop of a PaaS provider;
• functionalities:
  • uses the RTPI to interact with the PaaS;
  • ...;
• interactions:
  • ...;
• environment: container;
• responsible: TBD (TBD);
• references:
  • Cloud4SOA (???);
• priority: ... (???);

Packages repository

• description: stores software artifacts, which are actually archives (most likely zip, tar or cpio), containing either binary application code and data (e.g. war for Java, egg files for Python, etc.), or application source code;

• functionalities:
  • provides upload / download of software artifact archives;
  • supports for mirroring between the repository hosted on the developers' workstation (or on a server inside the office) and the repository hosted in the cloud;
• interactions:
  • core services;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:
  • Git;
  • Go;
• priority: medium (???);
• references:

  • mOSAIC Notes -- Packages;

  • mosaic-packages-repository;

Snapshot repository

• description: stores (possibly large) binary unstructured data streams, used by other services (or the application itself) to store large state snapshots, backups, etc.;
• functionalities:
  • provides streamed upload / download of data streams;
  • supports mirroring between cloud providers;
• interactions:
  • core services;
• environment: container;
• responsible: TBD (IeAT);
• technologies:

  • Ceph (???);

• priority: medium (???);
• references:

  • Amazon S3;

  • Amazon Glacier;

Object store

• description: stores small structured data objects (preferably JSON, but YAML or XML should be fine), used by other services to retrieve configuration or store small state snapshots;
• priority: medium (???);
• functionalities:
  • providers get / set operations for objects;
  • supports mirroring between cloud providers;
• interactions:
  • core services;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:

  • CouchDB;

• references:

  • Basic Concepts About IndexedDB;

Discovery and quorum service

• description: ...;
• priority: high;
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: container;
• responsible: Silviu (IeAT);
• technologies:
  • ... (Silviu???);
• references:
  • ... (Silviu???);

DNS service

• description: manages a DNS zone;
• priority: high;
• functionalities:
  • behaves as an authoritative DNS server for a particular zone (or zones);
  • enables clients to manage (create, destroy, update) records for the managed zones;
• interacions:
  • object store: for storing the various items related with zones or records;
  • core services;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:
  • Bind;
  • tinydns;
  • CouchDB;
  • Amazon Route53;
• references:
  • .. (Ciprian???);

Various monitoring collectors

• description: collects monitoring metrics and events from various sources (the application itself, the PaaS, etc.) and then delegates this data, through the monitoring distribution, towards analysis agents;
• priority: ... (???);
• functionalities:
  • ...;
• interactions:
  • monitoring distribution: for the actual distribution of the monitoring data;
  • core services;
• environment: container;
• responsible: TBD (TBD);
• technologies:
  • ...;
• references:
  • ...;

Monitoring distribution

• description: distributes the monitoring metrics and events towards various interested parties, in a publish-subscribe manner;
• priority: ... (???);
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: container;
• responsible: TBD (IeAT);
• technologies:
  • ...;
• references:
  • ...;

Logging service

• description: stores and exposes messages and various events pushed by the various services; these items are meant to support troubleshooting or debugging actions done by the operator; (the monitoring, alerts and other events are handled by other services;)
• priority: low;
• functionalities:
  • ...;
• interactions:
  • core services;
• environment: container;
• responsible: TBD (IeAT);
• technologies:
  • ...;
• references:
  • ...;

Notifications service

• description: forwards alerts and other events, published by the various services, towards interested parties (most likely paging services or human operators via SMTP); these alerts are most likely meant to reach human operators; they are just forwarded (or stored only temporarily); (the monitoring, logging and other events are handled by other services;)
• priority: low;
• functionalities:
  • ...;
• interactions:
  • core services;
• environment: container;
• responsible: TBD (IeAT);
• technologies:
  • ...;
• references:

  • Amazon SNS;

Credentials service

• description: ...;
• priority: low;
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:
  • ... (Ciprian???);
• references:

  • Security in Plan 9;

  • mosaic-credentials-service;

Hub

• description: provides low-level RPC-like and publish-subscribe system for inter-service communication; it could be used for service life-cycle related operations (i.e. management tasks done by an operator), initial service discovery and high-level interface end-point negotiation;
• priority: ... (???);
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:
  • ... (Ciprian???);
• references:

  • mOSAIC Notes -- Hub;

  • Extended Process Registry for Erlang;

  • Introduction to D-BUS;

Container controller

• description: ...;
• priority: medium (???);
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: native unconstrained;
• responsible: TBD (IeAT);
• technologies:
  • ...;
• references:
  • ...;

File-system controller

• description: ...;
• priority: medium (???);
• functionalities:
  • ...;
• interactions:
  • ...;
• environment: native unconstrained;
• responsible: TBD (IeAT);
• technologies:
  • ...;
• references:
  • ...;

HTTP gateway

• description: because most services will expose HTTP-based interfaces, this service should behave as a "dispatcher" of requests towards all these services (actually it is a reverse proxy); although all these services could have been used without this, it can solve a couple of issues orthogonal to the HTTP-based interfaces like authentication, logging, etc.;
• priority: high;
• functionalities:
  • act as a reverse proxy for the other services running on the same node; (thus dispatching HTTP requests based on a base URL prefix;)
  • authenticating requests between the clients (which could be other services on the same node) and the service; (thus the service doesn't need to handle credentials, checking, etc.;)
  • securing requests from outside the node (SSL);
  • monitoring, logging and providing various metrics for requests (health checks, response time, etc.);
• interactions:
  • core services;
• environment: container;
• responsible: Ciprian (IeAT);
• technologies:
  • Mongrel2 or Nginx;
• references:
  • ...;

Other references

• Building blocks of scalable applications;