(mOSAIC) mOS package builder -- Guide

Important

This document is a draft!

1   Overview

Error

FIXME: To be written!

2   Concepts

2.1   Packages

Error

FIXME: To be written!

2.2   Definitions

Error

FIXME: To be written!

2.3   Overlays

Error

FIXME: To be written!

2.4   Resources

Error

FIXME: To be written!

3   Descriptors

3.1   Package

Error

FIXME: To be written!

3.2   Definitions

Error

FIXME: To be written!

3.3   Dependencies

Error

FIXME: To be written!

3.4   Overlays

Error

FIXME: To be written!

3.5   Resources

Error

FIXME: To be written!

4   Examples

4.1   mOSAIC RabbitMQ (annotated)

Note

The following example is taken from the source code repository, and is available at https://github.com/mosaic-cloud/mosaic-mos-package-builder/blob/development/examples/example-01.json.

{
      // This part is constant.  It identifies the JSON structure.
      "_schema" : "tag:ieat.ro,2014:mosaic:v2:mos-package-builder:descriptors:composite-package",
      "_schema/version" : 1,

      // These are the the main package information tags.
      //   See: https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-specfile-syntax.html
      // The values are accessible as @{package:<name>}, etc.
      "package" : {
              "name" : "@{definitions:package:name}",
              "version" : "@{definitions:package:version}",
              "release" : "@{definitions:package:release}",
              "architecture" : "i686",
              "root" : "/opt/@{package:identifier}"
      },

      // These are the package dependency tags.  (See the link as above.)
      "dependencies" : {
              "provides" : ["@{definitions:package:name}"],
              "requires" : [
                      "mosaic-rt-erlang-r15b-32bit",
                      "glibc-32bit"
              ]
      },

      // These are the other (less important) package information tags.
      "miscellaneous" : {
              "license" : "apache-2.0",
              "url" : "http://mosaic.ieat.ro/",
              "summary" : "mOSAIC components: RabbitMQ"
      },

      // This section describes how the contents of the package is to be created.
      "overlays" : [
              {
                      // This overlay states that some folders should be created.
                      "generator" : "folders",
                      // This `target` states that the folders to be created should be prefixed with the specified value.
                      "target" : "@{package:root}",
                      // This in effect states that `@{package:root}` should be created.
                      "folders" : ["/"]
              },
              {
                      // This overlay states that the contents of the resource file should be unarchived.
                      "generator" : "unarchiver",
                      // This states that the contents should be placed under the specified path.
                      "target" : "@{package:root}",
                      // This states the name of the rource to be used.  (See the next section.)
                      "resource" : "mosaic-components-rabbitmq.cpio.gz",
                      // This states that the archive is to be extracted with `cpio` after `gunzip` has been run.
                      "format" : "cpio+gzip"
              }
      ],

      // This section describes resource files that are needed to create the package.
      "resources" : {
              "mosaic-components-rabbitmq.cpio.gz" : {
                      // This states how the resource file is to be obtained, i.e. fetching it from an URL.
                      "generator" : "fetcher",
                      // The URL of the resource (see below the URL base)!
                      "uri" : "@{definitions:resources:url:base}/mosaic-components-rabbitmq--@{definitions:resources:url:suffix}.cpio.gz",
                      // This value is inferred automatically.  It should not be usually specified!
                      "cache" : "mosaic-components-rabbitmq--@{definitions:resources:url:suffix}.cpio.gz"
              }
      },

      // These are "variables" that allow some customization.
      // The values are accessible as @{definitions:<name>}, etc.
      "definitions" : {
              "package:name" : "mosaic-components-rabbitmq",
              "package:version" : "0.7.0_dev",
              "package:release" : "@{execution:timestamp}",
              "resources:url:base" : "http://data.volution.ro/ciprian/public/mosaic/packages",
              "resources:url:suffix" : "@{definitions:package:version}"
      }
}

4.2   MODAClouds monitoring SDA Matlab (annotated)

Note

The following example is taken from the source code repository, and is available at https://github.com/mosaic-cloud/mosaic-mos-package-builder/blob/development/examples/example-02.json.

{
      "_schema" : "tag:ieat.ro,2014:mosaic:v2:mos-package-builder:descriptors:composite-package",
      "_schema/version" : 1,

      "package" : {
              "name" : "@{definitions:package:name}",
              "version" : "@{definitions:package:version}",
              "release" : "@{definitions:package:release}",
              "architecture" : "i686",
              "root" : "/opt/@{package:identifier}"
      },

      "dependencies" : {
              "provides" : ["@{definitions:package:name}"],
              "requires" : [
                      "modaclouds-rt-matlab-mcr-r2013a",
                      "mosaic-rt-jre-7"
              ]
      },

      "miscellaneous" : {
              "license" : "apache-2.0",
              "url" : "http://modaclouds.eu/",
              "summary" : "MODAClouds services: monitoring SDA Matlab"
      },

      "overlays" : [
              {
                      "generator" : "folders",
                      "target" : "@{package:root}",
                      "folders" : [
                              "/",
                              "/etc",
                              "/bin",
                              "/lib",
                              "/etc/sda-matlab",
                              "/lib/sda-matlab",
                              "/lib/scripts"
                      ]
              },
              {
                      "generator" : "unarchiver",
                      "target" : "@{package:root}/lib/sda-matlab",
                      "resource" : "sda-matlab-distribution.tar.gz",
                      "format" : "tar+gzip",
                      "options" : {
                              // This states that the paths within the archive should be rewritten by removing the first component.
                              //   I.e. `a/b/c` becomes `b/c`.
                              "strip-components" : 1
                      }
              },
              {
                      // This overlay states that a file should be created from a template, possibly with expansion of definitions.
                      "generator" : "file-creator",
                      // This states the path of the destination file.
                      "target" : "@{package:root}/lib/scripts/run.bash",
                      // This states the resource file used as a template.
                      "resource" : "service-run.bash",
                      // This states that the destination file should be marked as executable.
                      "executable" : true,
                      // This states that definition expansion should be done.
                      "expand" : true
              },
              {
                      // This overlay states that some symlinks should be created.
                      "generator" : "symlinks",
                      // Like in the `folders` generator case, the symlinks are created under this folder.
                      "target" : "@{package:root}/lib/sda-matlab",
                      "links" : {
                              // The key is the path (relative to `target`) of the symlink, and the value is the exact contents of the symlink.
                              "dda.properties" : "../../etc/sda-matlab/dda.properties",
                              "kb.properties" : "../../etc/sda-matlab/kb.properties",
                              "port.txt" : "../../etc/sda-matlab/port.txt"
                      }
              },
              {
                      "generator" : "symlinks",
                      "target" : "@{package:root}/bin",
                      "links" : {
                              "@{package:name}--run-service" : "../lib/scripts/run.bash"
                      }
              }
      ],

      "resources" : {
              "service-run.bash" : {
                      // This states that the resource file is to be taken from the sources folder.
                      "generator" : "sources",
                      "path" : "service-run.bash"
              },
              "sda-matlab-distribution.tar.gz" : {
                      "generator" : "fetcher",
                      "uri" : "@{definitions:resources:modaclouds:url:dependencies}/modaclouds-monitoring-sda-matlab-@{definitions:versions:modaclouds:sda-matlab}-distribution.tar.gz"
              }
      },

      "definitions" : {
              "package:name" : "modaclouds-services-monitoring-sda-matlab",
              "package:version" : "0.7.0_dev",
              "package:release" : "@{execution:timestamp}",
              "versions:modaclouds:sda-matlab" : "1.0.2",
              "resources:modaclouds:url:base" : "http://data.volution.ro/ciprian/public/modaclouds",
              "resources:modaclouds:url:dependencies" : "@{definitions:resources:modaclouds:url:base}/dependencies",
              // These variables are used in the scripts.
              "environment:SDA_MATLAB_HOME" : "@{package:root}/lib/sda-matlab",
              "environment:SDA_MATLAB_CONF" : "@{package:root}/etc/sda-matlab",
              "environment:MCR_HOME" : "/opt/modaclouds-rt-matlab-mcr-r2013a/v81",
              "environment:JAVA_HOME" : "/opt/mosaic-rt-jre-7"
      }
}

4.3   mOSAIC platform packages

All the package descriptors for the mOSAIC platform can be found at the following link: https://github.com/cipriancraciun/mosaic-mos-platform-packages/tree/development/packages.

4.4   MODAClouds platform packages

All the package descriptors for the MODAClouds platform can be found at the following link: https://github.com/cipriancraciun/modaclouds-mos-platform-packages/tree/development/packages.

5   Package building

Error

FIXME: To be written!

5.1   Environment preparation

File-system variables

By default the tool will use ${TMPDIR} for the temporary directory, and no resource cache directory. It is advised to use a memory-backed file system for the temporary directory, and a disk backed file system for the resource cache directory.

export mpb_resource_cache=/tmp/mpb-resources
export mpb_temporary=/tmp/mpb-temporary
mkdir -- "${mpb_resource_cache}" "${mpb_temporary}"

Execution variables

## NOTE: If one wants to see all the details:
export mpb_verbose=true
## NOTE: If one only wants to see what would execute (without actually executing anything):
export mpb_execute=false

Script fetching

This variable is not used by the tool itself (it is only used in these examples).

export mpb_script=/tmp/mbp.py
curl -L --retry 3 -o "${mpb_script}" -- 'https://raw.githubusercontent.com/mosaic-cloud/mosaic-mos-package-builder/development/sources/mos-package-builder.py'
chmod +x -- "${mpb_script}"

5.2   Package building

Using a "workbench" argument

If one has a folder, say workbench, where either a sources directory exists (containing a package.json descriptor file and possibly other source files), or a package.json descriptor file exists, then one can use the following command; the resulting RPM package is placed under the designated folder as package.rpm.

"${mpb_script}" ./workbench

Using a "sources" folder and "package.rpm" arguments

If one has a folder, say sources, which contains a package.json descriptor file and possibly other source files, then one can use the following command by specifying that sources folder and where the resulting RPM package should be placed.

"${mpb_script}" ./sources ./package.rpm

Using a "sources" archive and "package.rpm" arguments

If one has an archive (either .zip, .tar or .cpio), say sources.zip, which contains a package.json descriptor file and possibly other source files, then one can use the following command by specifying that sources archive and where the resulting RPM package should be placed.

"${mpb_script}" ./sources.zip ./package.rpm

Using a "package.json" descriptor file and "package.rpm" arguments

If one has a package descriptor file, say package.json, and no other source files, then one can use the following command by specifying that descriptor file and where the resulting RPM package should be placed.

"${mpb_script}" ./package.json ./package.rpm

6   Package building by using Jenkins

Error

FIXME: To be written!

7   References

Error

FIXME: To be written!