(MODAClouds) Supervised service deployment

Introduction

Notes

Note

The procedure described herein is tailored for a single host deployment, however splitting the various components on different hosts is possible by simply tweaking the endpoints.

Note

This procedure is only meant for experimentation, the final deployment being based on mOSAIC.

Requirements

  • OpenSUSE 13.1 (or derivative) environment; preferably mOS, available on IeAT's Eucalyptus, Amazon EC2, and Flexiant; (see the dedicated page for details);
  • x86-64 architecture;
  • "enough" RAM and CPU; (i.e. at minimum 1-2 GiB RAM, and 2 CPU's;)
  • in case of firewall-like solutions (either on the host like iptables, or in the cloud like security groups), open the required ports to be accessed from outside;

Endpoints

MODAClouds knowledgebase

MODAClouds monitoring manager

MODAClouds monitoring DDA

MODAClouds monitoring SDA Matlab

MODAClouds monitoring SDA Weka

MODAClouds metric explorer

MODAClouds metric importer

MODAClouds load-balancer controller

MODAClouds load-balancer reasoner

  • none;

mOSAIC object store

Snippets

Preparation (using mOS tools)

Attention!

These commands work only with the mOS 2.0.0 distribution.

Note

Execute the following commands in order, inside (the same) shell on the target host.

Switching as mos-operator

su - mos-operator

Refreshing repositories

mos-packages refresh

Updating existing packages

mos-packages update

Installing MODAClouds packages

mos-packages install \
        modaclouds-services-knowledgebase \
        modaclouds-services-monitoring-manager \
        modaclouds-services-monitoring-dda \
        modaclouds-services-monitoring-sda-matlab \
        modaclouds-services-monitoring-sda-weka \
        modaclouds-services-metric-explorer \
        modaclouds-services-metric-importer \
        modaclouds-services-load-balancer-controller \
        modaclouds-services-load-balancer-reasoner \
        mosaic-object-store \
#

Preparation (using Zypper)

Attention!

These commands work with any OpenSUSE 13.1 (x86-64) distribution.

Note

Execute the following commands in order, inside (the same) shell on the target host.

Registering mOS RPM repositories (hosted at IeAT)

zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        http://mos.repositories.mosaic-apps.eu/v2/packages/modaclouds/rpm \
        modaclouds
zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        http://mos.repositories.mosaic-apps.eu/v2/packages/mosaic/rpm \
        mosaic
zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        http://mos.repositories.mosaic-apps.eu/v2/packages/external/rpm \
        mosaic-external

Refreshing mOS RPM repositories (hosted at IeAT)

zypper \
            --non-interactive \
            --no-gpg-checks --gpg-auto-import-keys \
    refresh \
        modaclouds \
        mosaic \
        mosaic-external \
#

Registering mOS RPM repositories (hosted on S3) (only as backup)

zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        https://s3-eu-west-1.amazonaws.com/mos-packages/v2/modaclouds/rpm \
        modaclouds-s3
zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        https://s3-eu-west-1.amazonaws.com/mos-packages/v2/mosaic/rpm \
        mosaic-s3
zypper \
            --non-interactive --no-refresh \
            --no-gpg-checks --gpg-auto-import-keys \
    addrepo \
            --type rpm-md \
            --no-keep-packages --no-gpgcheck \
        https://s3-eu-west-1.amazonaws.com/mos-packages/v2/external/rpm \
        mosaic-external-s3

Refreshing mOS RPM repositories (hosted on S3) (only as backup)

zypper \
            --non-interactive \
            --no-gpg-checks --gpg-auto-import-keys \
    refresh \
        modaclouds-s3 \
        mosaic-s3 \
        mosaic-external-s3 \
#

Updating existing packages

zypper \
            --non-interactive \
            --no-gpg-checks --gpg-auto-import-keys \
    update

Installing MODAClouds packages

zypper \
          --non-interactive --no-refresh \
          --no-gpg-checks --gpg-auto-import-keys \
    install \
        modaclouds-services-knowledgebase \
        modaclouds-services-monitoring-manager \
        modaclouds-services-monitoring-dda \
        modaclouds-services-monitoring-sda-matlab \
        modaclouds-services-monitoring-sda-weka \
        modaclouds-services-metric-explorer \
        modaclouds-services-metric-importer \
        modaclouds-services-load-balancer-controller \
        modaclouds-services-load-balancer-reasoner \
        mosaic-object-store \
#

Environment

Attention!

The IP address below is the one configured on the hosts Ethernet card, not the one bound through the cloud infrastructure. (I.e. in case of Amazon EC2 / Eucalyptus / OpenStack infrastructures this is the "private" IP address, not the "public" one.)

MOS_NODE_IP="${mos_node_private_ip}"
RELEASE=0.7.0.10

Deployment

Attention!

Execute the following commands in order, each inside in a separate shell on the target host, because the services don't daemonize themselves in the background.

Attention!

Before executing each command, don't forget to set the MOS_NODE_IP variable (and others presented above).

Hint

It is advisable to execute these commands inside a screen, tmux or dtach session, to allow disconnecting and reconnecting over SSH without disrupting the execution of the services.

Note

In order to test the deployment open each of the links listed in the Endpoints section, and follow the monitoring platform procedures described in the deliverables.

MODAClouds knowledgebase
env \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_PORT=8081 \
    dtach -c /tmp/modaclouds-services-knowledgebase.dtach \
    /opt/modaclouds-services-knowledgebase-"${RELEASE}"/bin/modaclouds-services-knowledgebase--run-service
MODAClouds monitoring DDA
env \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_PORT=8083 \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_PORT=8081 \
    dtach -c /tmp/modaclouds-services-monitoring-dda.dtach \
    /opt/modaclouds-services-monitoring-dda-"${RELEASE}"/bin/modaclouds-services-monitoring-dda--run-service
MODAClouds monitoring SDA (written in Matlab)
env \
        MODACLOUDS_MONITORING_SDA_MATLAB_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_SDA_MATLAB_ENDPOINT_PORT=8084 \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_PORT=8083 \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_PORT=8081 \
    dtach -c /tmp/modaclouds-services-monitoring-sda-matlab.dtach \
    /opt/modaclouds-services-monitoring-sda-matlab-"${RELEASE}"/bin/modaclouds-services-monitoring-sda-matlab--run-service
MODAClouds monitoring SDA (written in Weka)
env \
        MODACLOUDS_MONITORING_SDA_WEKA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_SDA_WEKA_ENDPOINT_PORT=8085 \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_PORT=8083 \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_PORT=8081 \
    dtach -c /tmp/modaclouds-services-monitoring-sda-weka.dtach \
    /opt/modaclouds-services-monitoring-sda-weka-"${RELEASE}"/bin/modaclouds-services-monitoring-sda-weka--run-service
MODAClouds monitoring manager
env \
        MODACLOUDS_MONITORING_MANAGER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_MANAGER_ENDPOINT_PORT=8082 \
        MODACLOUDS_MONITORING_SDA_MATLAB_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_SDA_MATLAB_ENDPOINT_PORT=8084 \
        MODACLOUDS_MONITORING_SDA_WEKA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_SDA_WEKA_ENDPOINT_PORT=8085 \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_PORT=8083 \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_KNOWLEDGEBASE_ENDPOINT_PORT=8081 \
    dtach -c /tmp/modaclouds-services-monitoring-manager.dtach \
    /opt/modaclouds-services-monitoring-manager-"${RELEASE}"/bin/modaclouds-services-monitoring-manager--run-service
MODAClouds metric explorer
env \
        MODACLOUDS_METRIC_EXPLORER_DASHBOARD_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_EXPLORER_DASHBOARD_ENDPOINT_PORT=9010 \
        MODACLOUDS_METRIC_EXPLORER_QUERY_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_EXPLORER_QUERY_ENDPOINT_PORT=9011 \
        MODACLOUDS_METRIC_EXPLORER_PICKLE_RECEIVER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_EXPLORER_PICKLE_RECEIVER_ENDPOINT_PORT=9012 \
        MODACLOUDS_METRIC_EXPLORER_LINE_RECEIVER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_EXPLORER_LINE_RECEIVER_ENDPOINT_PORT=9013 \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_MONITORING_DDA_ENDPOINT_PORT=8083 \
    dtach -c /tmp/modaclouds-services-metric-explorer.dtach \
    /opt/modaclouds-services-metric-explorer-"${RELEASE}"/bin/modaclouds-services-metric-explorer--run-service
MODAClouds metric importer
env \
        MODACLOUDS_METRIC_IMPORTER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_IMPORTER_ENDPOINT_PORT=9019 \
        MODACLOUDS_METRIC_EXPLORER_LINE_RECEIVER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_METRIC_EXPLORER_LINE_RECEIVER_ENDPOINT_PORT=9013 \
    dtach -c /tmp/modaclouds-services-metric-importer.dtach \
    /opt/modaclouds-services-metric-importer-"${RELEASE}"/bin/modaclouds-services-metric-importer--run-service
MODAClouds load-balancer controller
env \
        MODACLOUDS_LOAD_BALANCER_CONTROLLER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_LOAD_BALANCER_CONTROLLER_ENDPOINT_PORT=8088 \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_PORT_MIN=8000 \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_PORT_MIN=8080 \
    dtach -c /tmp/modaclouds-services-load-balancer-controller.dtach \
    /opt/modaclouds-services-load-balancer-controller-"${RELEASE}"/bin/modaclouds-services-load-balancer-controller--run-service
MODAClouds load-balancer reasoner
env \
        MODACLOUDS_LOAD_BALANCER_CONTROLLER_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_LOAD_BALANCER_CONTROLLER_ENDPOINT_PORT=8088 \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_IP="${MOS_NODE_IP}" \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_PORT_MIN=8000 \
        MODACLOUDS_LOAD_BALANCER_GATEWAY_ENDPOINT_PORT_MIN=8080 \
    dtach -c /tmp/modaclouds-services-load-balancer-reasoner.dtach \
    /opt/modaclouds-services-load-balancer-reasoner-"${RELEASE}"/bin/modaclouds-services-load-balancer-reasoner--run-service
mOSAIC object store
env \
        MOSAIC_OBJECT_STORE_ENDPOINT_IP="${MOS_NODE_IP}" \
        MOSAIC_OBJECT_STORE_ENDPOINT_PORT=9020 \
    dtach -c /tmp/mosaic-object-store.dtach \
    /opt/mosaic-object-store-"${RELEASE}"/bin/mosaic-object-store--run-service