(mOSAIC) Descriptors

Contents

(mOSAIC) Descriptors

References

mOSAIC application deployment workflow (initial variant);
mOSAIC application deployment workflow (simplified proposal);
mOSAIC tools -- configuration files (on wiki);
mOSAIC tools -- configuration files (initial draft);
mOSAIC descriptors (from Ljubljana);
mOSAIC deploy request (from Silviu) (original);
mOSAIC deploy request (from Silviu) (continued);

Workflow

phase	inputs	tool	outputs
development	(idea)	Semantic Engine	SP Application Descriptor
development	SP Application Descriptor	(editor)	SP Assembly Descriptor
pre-deployment	SP Assembly Descriptor	SP Tool	SP Requirements Descriptor
pre-deployment	SP Requirements Descriptor	(provisioning)	SP Resources Descriptor
pre-deployment	SP Assembly + Resources Descriptor	SP Tool	SP Deployment Descriptor
deployment	SP Deployment Descriptor	SP Tool	(application)

Generic descriptor

Summary:

this is not a separate descriptor, it just highlights which fields should appear in all other descriptors;
all fields are mandatory (even if they allow a null value), except,
only those fields that allow an undefined value are optional;

Proposed syntax:

<<generic-descriptor>> ::= {
    "schema" : /^([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?):([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?)$/,
    "version" : <<positive-integer>>,
    "identifier" : <<hex-data>>,
    "revision" : <<hex-data>>,
        // current revision
    "revision/previous" : <<hex-data>> | null,
        // previous revision
    "annotation" : <<object>> | null | undefined
}

<<hex-data>> ::= /^[0-9a-f]{2}+$/

SP application descriptor

It has an identical syntax with the assembly descriptor, just that it serves as a template (generated by the semantic engine) for the user to edit in order to get a descriptor for his application.

SP assembly descriptor

Summary:

purpose:
- describes the raw architecture of the application;
- provides a list of needed components and resources;
- provides configuration for the various components or resources;
- describes the relation between the various components;
provided by:
- the semantic engine and then edited by the editor -- if the developer follows the entire mOSAIC workflow;
- the developer;
consumed by:
- the SP tool -- to generate various other descriptors;
edited with:
- the Eclipse editor;
- any text editor;

Current syntax:

(see also this page);

<<assembly-descriptor>> ::= {
    <<component-alias>> : <<component-descriptor>>,
    ...
}

<<component-descriptor>> ::= {
    "type" : /^#([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?):([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?)$/,
    "configuration" : <<object>> | null,
    "order" : <<positive-integer>>,
    "delay" : <<timeout>>,
    "annotation" : <<object>> | null,
}

<<component-alias>> ::= /^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$/

<<timeout>> ::= <<positive-integer>>
    // the number of milliseconds

Proposed syntax:

<<assembly-descriptor>> ::= {
    <<generic-descriptor>>,
        // include the same fields as the generic one
    "components" : {
        <<alias>> : <<component-descriptor>>,
        ...
    },
    "resources" : {
        <<alias>> : <<resource-destriptor>>,
        ...
    },
}

<<component-descriptor>>, <<resource-descriptor>> ::= {
    "type" : /^([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?):([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?)$/,
    "configuration" : <<object>> | null,
    "dependencies" : <<dependencies>> | null | undefined,
    "multiplicity" : <<multiplicity>>,
    "annotation" : <<object>> | null | undefined,
}

<<alias>> ::= /^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$/
    // it must be unique in the scope of the same descriptor
    // i.e. the components and resources are considered part of the same set, thus the alias must be unique in this set

<<dependencies>> ::=
    | {
        <<alias>> : <<dependency>>,
        ...
    }
    | null | undefined
        // i.e. none

<<dependency>> ::= true
    // currently only `true` is accepted

<<multiplicity>> ::=
    | "singleton"
        // i.e. only one component of this type can exist
    | {
        "minimum" : <<positive-integer>> | null | undefined,
        "maximum" : <<positive-integer>> | null | undefined,
        "initial" : <<positive-integer>> | null | undefined,
    }

Example:

{
    "schema" : "mosaic-platform:assembly-descriptor",
    "version" : 1,
    "identifier" : "67da4b3b0b1e34c41e3c9b4474574a3e",
    "revision" : "5e2e9daba78dc623b6776dd000e3a16c",
    "revision/previous" : null,
    
    "components" : {
        "httpg" : {
            "type" : "mosaic-components:http-gateway",
            "configuration" : null,
            "dependencies" : {
                "rabbitmq" : true
            }
            "multiplicity" : "singleton",
        },
        "cloudlet" : {
            "type" : "mosaic-components:java-cloudlet-container",
            "configuration" : {
                ...
            },
            "dependencies" : {
                "httpg" : true,
                "amqp-driver" : true,
                "kv-driver" : true,
            },
            "multiplicity" : {
                "minimum" : 2,
                "maximum" : 20,
                "initial" : 2
            },
        },
        "amqp-driver" : {
            ...
        },
        "kv-driver" : {
            ...
        }
    },
    
    "resources" : {
        "rabbitmq" : {
            "type" : "mosaic-components:rabbitmq",
            "configuration" : null,
            "multiplicity" : "singleton",
        },
        "riak" : {
            "type" : "mosaic-components:riak-kv",
            "configuration" : null,
            "multiplicity" : {
                "minimum" : 3,
                "maximum" : 9,
                "initial" : 3
            },
        }
    },
}

SP requirements descriptor

Summary:

purpose:
- describes the raw infrastructure requirements of the application;
- provides a list of required resources like VM's, together with needed features (memory, CPU, etc.);
provided by:
- the SP tool -- generated via an heuristic based on the number of components;
- the developer -- if he wants to customize certain aspects of the resources;
consumed by:
- the "provisioning" tool -- in our case the CA which should acquire resources matching the requirements;
edited with:
- the Eclipse editor;
- any text editor;

Current syntax:

none so far;
but see the prototype available in Silviu's resource allocator (see this page);

Proposed syntax:

<<requirements-descriptor>> ::= {
    <<generic-descriptor>>,
        // include the same fields as the generic one
    "compute-nodes" : {
        <<alias>> : <<compute-node-descriptor>>,
            // this alias has no meaning for the provisioner
            // this alias has meaning only for the developer
        ...
    }
}

<<compute-node-descriptor>> ::= {
    "type" : <<compute-node-type>>,
        // this type has no meaning for the provisioner
        // this type has meaning only for the platform and should be "pasted" back for each instance of this resource type in the resulting resources descriptor
    "image" : <<compute-node-image>>,
    "cores" : {
        "minimum" : <positive-integer> | null | undefined,
        "maximum" : <positive-integer> | null | undefined,
        "preferred" : <positive-integer> | null | undefined,
    },
    "memory" : {
        // same as above, the number being in MiB
    },
    "storage" : {
        // same as above
    },
    "count" : {
        "minimum" : <positive-integer> | null | undefined,
        "maximum" : <positive-integer> | null | undefined,
        "preferred" : <positive-integer> | null | undefined,
    },
    "annotation" : <object> | null | undefined,
}

<<compute-node-type>> ::= /^([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?):([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?)$/
<<compute-node-image>> ::= /^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$/

Example:

{
    "schema" : "mosaic-platform:requirements-descriptor",
    "version" : 1,
    "identifier" : "67da4b3b0b1e34c41e3c9b4474574a3e",
    "revision" : "5e2e9daba78dc623b6776dd000e3a16c",
    "revision/previous" : null,
    
    "compute-nodes" : {
        "executors" : {
            "type" : "mosaic-platform:executor",
            "image" : ""mosaic-platform:mos-v0.11.0",
            "cores" : {"minimum" : 2, "maximum" : 8, "preferred" : 4},
            "memory" : {...},
            "storage" : {...},
            "count" : {"minimum" : 1, "maximum" : 9, "preferred" : 3}
        },
        "control" : {
            "type" : "mosaic-platform:control",
            "image" : ""mosaic-platform:mos-v0.11.0",
            "cores" : {"minimum" : 1, "maximum" : 4, "preferred" : 2},
            "memory" : {...},
            "storage" : {...},
            "count" : {"minimum" : 1, "maximum" : 4, "preferred" : 2}
        }
    }

SP resources descriptor

Summary:

purpose:
- describes the raw infrastructure available to the application;
- provides a list of available resource like VM's, together with identification and connection information (identifiers, FQDN, user, etc.);
provided by:
- the "provisioning" tool -- in our case the CA which after acquiring the resources provides the complete list;
- the developer -- if it manually provisions resources;
consumed by:
- the SP tool -- to generate the deployment descriptor;

Current syntax:

none so far;
as the resources are manually (or through some external tool) started by the developer himself;
all that is needed is the URL of one of the application controllers running on one of the available VM's;

Proposed syntax:

<<resources-descriptor>> ::= {
    <<generic-descriptor>>,
        // include the same fields as the generic one
    "compute-nodes" : {
        <<compute-node-identifier>> : <<compute-node-descriptor>>,
        ...
    }
}

<<compute-node-descriptor>> ::= {
    "type" : <<compute-node-type>>,
    "endpoints" : {
        "fqdn" : {
            "public" : <<fqdn>> | null | undefined,
            "private" : <<fqdn>> | null | undefined,
        } | null | undefined,
        "ip-v4" : {
            "public" : [<<ip-v4-address>>, ...],
                // better said "extra-cluster", i.e. an address accessible from the "outside", to be used by nodes outside the group to talk to the ones inside;
            "private" : [<<ip-v4-address>>, ...],
                // better said "intra-cluster", i.e. an address to be used by nodes inside a group to talk to each other;
                // it is required at least one address for each type;
                // the same address can be reused for both types;
                // (i.e. you can use a public address as private if the provider doesn't provide on);
        },
        "ssh" : {
            "credentials" : {
                <<alias>> : {
                    "role" : "administrator" | "operator",
                    "login" : <<string>>,
                    "password" : <<string>> | null | undefined,
                    "public-key" : <<ssh-key-object>> | null | undefined,
                    "private-key" : <<ssh-key-object>> | null | undefined,
                    "public-key-fingerprint" : <<ssh-key-fingerprint>> | null | undefined,
                        // it is required one of the following field combinations (besides `login`):
                        // * `password`;
                        // * `private-key` (and optionally `public-key`);
                        // * `public-key-fingerprint`; (not for now;)
                    "annotation" : <<object>> | null | undefined,
                },
                ...
            },
            "service" : {
                "public-key" : <<ssh-key-object>> | null | undefined,
                "public-key-fingerprint" : <<ssh-key-fingerprint>> | null | undefined,
                "port" : <<port>> | null | undefined,
            } | null | undefined,
        } | null | undefined,
    },
    "annotation" : null | undefined,
}

<<compute-node-identifier>> ::= /^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$/
<<compute-node-type>> ::= /^([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?):([a-z0-9](?:[a-z0-9.-]*[a-z0-9])?)$/

<<fqdn>> ::= ...
<<ip-v4-address>> ::= ...
<<pkcs8-pem>> ::= ...
<ssh-key-object>> ::= {
    "encoding" : "pkcs8-pem/private" | "pkcs8-pem/public",
    "data" : <<base64>>,
        // the content of the key file encoded in Base64 without any newlines and padding
}
<<ssh-key-fingerprint>> ::= ...

Example:

{
    "schema" : "mosaic-platform:resources-descriptor",
    "version" : 1,
    "identifier" : "67da4b3b0b1e34c41e3c9b4474574a3e",
    "revision" : "5e2e9daba78dc623b6776dd000e3a16c",
    "revision/previous" : null,
    
    "compute-nodes" : {
        "aws--us-east-1--i94f3a7" : {
            "type" : "mosaic-platform:executor",
            "endpoints" : {
                "fqdn" : {
                    "public" : ["i94f3a7.compute-1.amazonaws.com"]    
                },
                "ip-v4" : {
                    "public" : ["59.23.5.94"],
                    "private" : ["10.1.1.94"]
                },
                "ssh" : {
                    "credentials" : {
                        "role" : "administartor",
                        "login" : "root",
                        "private-key" : {
                            "encoding" : "pkcs8-pem/private",
                            "data" : "..."
                        }
                    }
                }
            }
        },
        "aws--us-east-1--i66f3a7" : {
            "type" : "mosaic-platform:executor",
            "endpoints" : {
                "fqdn" : {
                    "public" : ["i66f3a7.compute-1.amazonaws.com"]
                },
                "ip-v4" : {
                    "public" : ["59.23.5.66"],
                    "private" : ["10.1.1.66"]
                },
                "ssh" : {
                    // ...
                }
            }
        },
        "aws--us-east-1--i33f3a7" : {
            "type" : "mosaic-platform:control",
            "endpoints" : {
                "fqdn" : {
                    "public" : "i33f3a7.compute-1.amazonaws.com"
                },
                "ip-v4" : {
                    "public" : ["59.23.5.33"],
                    "private" : ["10.1.1.33"]
                },
                "ssh" : {
                    // ...
                }
            }
        }
    }
}

SP deployment descriptor

Summary:

this descriptor is completely internal to the deployment tool;
purpose:
- subsumes all the required information to deploy the application on the acquired resources;
- it includes information about:
  - the components;
  - startup order and configuration;
  - available resources;
provided by:
- the SP tool;
consumed by:
- the SP tool;

Current syntax:

none so far;
as the components are started by feeding the current application assembly descriptor directly to the URL of the application controller;

Proposed syntax:

it should be almost a merge between the assembly and the resources descriptor;

<<deployment-descriptor>> ::= {
    <<generic-descriptor>>,
        // include the same fields as the generic one
    "platform" : {
        "components" : {
            <<alias>> : <<component-descriptor>>,
                // a clone of the same section from the assembly descriptor
            ...
        },
        "resources" : {
            <<alias>> : <<resource-destriptor>>,
                // a clone of the same section from the assembly descriptor
            ...
        },
    },
    "infrastructure" : {
        "compute-nodes" : {
            <<compute-node-identifier>> : <<compute-node-descriptor>>,
                // a clone of the same section from the resources descriptor
            ...
        }
    },
}