(mOSAIC) Object store -- Guide

Important

This document is a draft!

1   Overview

Error

FIXME: To be written!

2   Concepts

2.1   Objects

An object is nothing more than a keyed container which aggregates various attributes that refer to the same subject. For example one could have an object to hold the configuration parameters of a given service (or class of services); or perhaps to hold the end-point (and other protocol parameters) where a given service can be contacted.

An object's attributes, which are described in the next sections, are: data, indices, links, annotations, and attachments.

Similarities with other databases

In the case of document-oriented databases an object would map to a document (such as in CouchDB or MongoDB), in key-value stores it would map to a key-value pair (such as in Riak or Memcache). Meanwhile, in the case of relational databases, it would map to a row (in a "master" table) plus a few other rows (in referenced tables), which are in a one-to-many relationship.

2.2   Collections

A collection serves no other purpose than to group similar objects together, either based on purpose or type (such as all configuration objects in one collection), or based on scope (such as all objects belonging to the same service).

Collections can be used without being created first, and there is no option to destroy them (except removing one-by-one all the objects that belong to it).

Therefore there are no other implications (in terms of performance or functionality) of placing an object in a collection or another, except perhaps easing operational procedures (such as removing all objects belonging to a service).

Similarities with other databases

A collection maps perfectly to a bucket in Riak, and roughly to a database in CouchDB or a collection in MongoDB (although in these two they serve other purposes).

2.3   Data

The most basic usage of an object would be to store some useful information, and have it available for later access.

The stored data can be anything, from JSON or XML to a binary file, and besides the actual data it is characterized by a content-type. Later based on this declared content-type one can decide how to interpret the data. Although there can be a single data item for an object, one could easily use multipart/mixed [MIME-multipart] to bundle together multiple data items; however it is advisable to avoid such a scenario and use either links or attachments.

Access to the data is atomic, thus consistent, and concurrent updates are permitted without any locking or conflict resolution mechanisms, the latest update overriding previous ones, thus no isolation with lost-updates being possible.

Although the data can be frequently accessed or updated without high overhead (except the usual network and disk latencies), it is advisable to cache operations by using the dedicated HTTP conditional requests [ETag] [RFC-7232].

Caution!

Because the data is stored temporarily in memory (both for caching and manipulation), it is advised to keep the amount of data small, well under a hundred kilo-bytes; any larger data should be handled as an attachment.

2.4   Indices

In addition to its data, an object can be augmented with indices which enables efficiently selecting objects on other criteria than just the object key.

An object can have multiple indices, each index being characterized by a label and a value, and it is allowed to have multiple indices with the same label. (Two identical indices, thus with the same label and value, are not allowed within the same object.)

This feature enables the following selection criteria:

  • selecting those objects which have an index with a given label and value;
  • selecting those objects which have an index with a given label, in increasing or decreasing order of the value;
  • selecting those objects which have an index with a given label, and a value in a given range (again ascending or descending);
  • deleting those objects which match one of the previously listed criteria;

The value of an index can be any valid JSON term, and it is allowed to have multiple indices with the same label but different value types. However the order between tho values of different types is unspecified, including between floats and integers. Moreover if the value is an JSON object, the order in which the attributes are specified is important, for example two objects having the same set of attribute pairs (thus matching keys and values) but given in a different order, are not technically equal (although they are logically); therefore it is advised to sort the attributes lexicographically based on the keys. In general it is advised to use either atomic values (such as strings, numbers, booleans and null) or lists (which are compared in parallel from left to right). (For a discussion on this topic see [CouchDB-collation]).

Caution!

Because index maintenance is expensive (especially in terms of I/O) it is advised to update indices less often.

Caution!

Because index values are independent of the data (and are updated in two different operations), there can be inconsistencies between what the index "promises" and what the data actually "contains". A simple solution would be to update the data and then the indices.

It must be noted that indices don't belong to a collection, and that their labels share the same global namespace, thus their scope being global. Therefore objects belonging to different collections but having the same index label can be returned.

Moreover indices don't need to be created prior to their usage, nor is there an option to destroy them (without removing any references in all objects that specify them).

Operations that apply on indices are asymmetric, in that one declares them at the object level, but can query them globally.

Note

Complementary to indices one can use links (to point to other objects) or annotations (to specify meta-data).

Similarities with other databases

The major difference between indices presented by this solution and other NoSQL (or SQL) databases is that most other solutions build their indices based on the actual data.

In CouchDB one can use views [CouchDB-views] to obtain similar functionality, meanwhile Riak has exactly the same feature [Riak-2i].

2.6   Attachments

Warning

This feature is not supported in the initial version!

Any data that logically belongs to the object, but which is either too large to be used as actual data or is rather static, can placed within an attachment. The most concrete parallel can be made with email attachments.

An attachment is created in two distinct steps (executable in either order):

  • first, the attachment is created by uploading its content, and obtaining its fingerprint; (uploading the same content twice results in the same fingerprint, and no extra storage space is used;)
  • second, a reference to the attachment (its fingerprint) is placed within the object with a given label (perhaps specifying its purpose), together with the content-type and size which serve only for informative purposes;
  • the same attachment can be referenced from multiple objects without uploading its data, provided that the fingerprint is known;

Similarly, accessing the attachment of an object is done in two steps: obtaining the attachment reference (possibly searching for a given label), then accessing the actual attachment based on its fingerprint.

Like with links, attachments are scoped under an object, only their data being globally stored.

In terms of efficiency, creating or updating attachments don't have high overhead (except the initial data upload).

Caution!

As in the case of links, one can create a reference to an non-existing attachment.

Note

In future versions this feature will reuse the artifact repository for actual attachment data storage.

Similarities with other databases

Except CouchDB which has exactly the same feature, and BLOB's in SQL databases, no other solutions present a similar feature.

2.7   Annotations

The annotations are meta-data which can be specified for objects or attachments, and are characterized by a label (unique within the same object) and any JSON term as a value.

Although annotations can be used interchangeably with either the object's data, indices or links, one should always remember the rule:

Annotations are those data which if erased don't impact the usage of the object.

In general annotations can be used to store ancillary data about the object, especially those used by operational tools. For example one can specify the creator, tool and possibly the source, ACL's or digital signatures, etc.

3   Use-cases

3.1   Service configuration

Suppose the operator has several instances of the same service type (such as an application server or database) which he would like to configure during execution. Moreover the operator would like to change the configuration and have it dynamically applied as easily as possible.

Single shared configuration

The most simple solution is to store the configuration parameters in an object created before the execution is started, preferably as a JSON term or plain text as the data, or alternatively as an attachment. Then at execution the object's reference is specified as an initialization argument to each of the instantiated services, which retrieve the data and use it to properly configure the service.

If each service continuously pools the object for updates, it can detect when the operator has changed the configuration parameters, and apply the necessary changes (possibly through a restart).

Although it might seem that pooling will involve fetching the same data again and again, possibly incurring large network overhead, such is not necessarily true if one uses HTTP conditional requests [ETag] [RFC-7232] which is rather efficient.

For example the URL might be .../objects/application-servers/data.

Multiple shared configurations

Let's suppose that the services require multiple different configuration parameters grouped in multiple "files", possibly because their syntax is different, or perhaps for better maintenance by the operator.

The solution is to create a "master" object, possibly without any data itself, and then using links to point to the other needed configuration objects.

Although it might seem that the solution is more complex, the services can easily obtain the different configuration "files" by using URL's such as:

.../objects/application-servers/data
used to get the "master"'s object data (if any);
.../objects/application-servers/links/configuration-a/0/object/data
used to get the configuration data of type a (points to [1]);
.../objects/application-servers/links/configuration-b/0/object/data
used to get the configuration data of type b (points to ...);
.../objects/application-servers/links/configuration-b/1/object/data
used to get the second configuration data of type b (in case there are multiple configuration "files" of the same type);
.../objects/application-servers-a/data [1]
actually holds the configuration of type b;

Again pooling can be applied to detect configuration changes, but because now it involves multiple objects, after an update has been detected a grace period should be waited (say 30 seconds), after which another check should be done, if no other objects updates have been detected in between, the configuration changes are applied; else a new grace period is started. (This prevents frequently restarting the service while the operator updates sequentially the configuration objects.)

Note

Technically, when accessed each of the later URL's will return an 302 moved temporarily HTTP status and the afferent redirection URL, and the HTTP client should retry the request at the new URL (most clients do this automatically). Although one would be tempted to store the redirection URL, it is advised against because the operator might choose not to change the data of the linked object, but instead swap it with a different one, therefore on the next request to the original URL, another redirection will be returned.

Single individual configuration with shared defaults configuration

Taking a step backward and getting back to the single configuration use-case, suppose now that each service instance should have its own configuration "file", possibly with another one used to specify the defaults.

The solution is to create multiple objects, one for each service instance to be executed, filled with specific configuration parameters, and another object with the defaults. Then the each of the individual objects also link to the shared one.

The URL's could be:

.../objects/application-server-1/data
used to get the configuration for the first application server;
.../objects/application-server-1/links/defaults/0/object/data
used to get the defaults for the configuration (points to [1]);
.../objects/application-server-2/data
used to get the configuration for the second application server;
.../objects/application-server-2/links/defaults/0/object/data
used to get the defaults for the configuration (points to [1]);
.../objects/application-server-defaults/data
actually holds the configuration defaults; [1]

Again the pooling (with a grace period) can be applied to detect configuration changes.

Multiple individual configurations with shared defaults

Going a step further it is easy to combine the previous two cases like this:

  • a single shared "master" object;
  • multiple shared configuration objects with the defaults;
  • the single shared "master" object links to the various shared configuration objects;
  • an individual "master" object for each service instance;
  • multiple individual configuration objects with the specific parameters, a few for each service instance;
  • each individual "master" object links to the individual configuration objects;
  • each individual "master" object links to the single shared "master" object;

The URL's could be:

  • .../objects/application-server-master/data;
  • .../objects/application-server-master/links/configuration-a/0/object/data (points to [3]); [1]
  • .../objects/application-server-master/links/configuration-b/0/object/data (points to [4]); [2]
  • .../objects/application-server-master-a/data; [3]
  • .../objects/application-server-master-a/data; [4]
  • .../objects/application-server-1/data;
  • .../objects/application-server-1/links/configuration-a/0/object/data (points to [5]);
  • .../objects/application-server-1/links/configuration-b/0/object/data (points to [6]);
  • .../objects/application-server-1/links/defaults/0/object/links/configuration-a/0/object/data (points to [1], and finally resolves to [3]);
  • .../objects/application-server-1/links/defaults/0/object/links/configuration-b/0/object/data (points to [2], and resolves to [4]);
  • .../objects/application-server-1-a/data; [5]
  • .../objects/application-server-1-a/data; [6]

3.2   Service discovery

Error

FIXME: To be revised!

Just like in the previous use-case, suppose there are several service instances of the same service type, and one needs to find the end-point of all of them.

The solution is to have an object for each running instance containing as data the end-point, and possibly other protocol details or credentials, and indexing it with a well known label, such as service-a, and any valid value, such as null.

Then obtaining all service end-points is as simple as listing all objects indexed with the well known label, and accessing their content.

Furthermore if the service becomes temporarily unavailable, it needs not to remove the object, but instead remove the index. Alternatively the value can be used as a weight or indication of availability.

As an alternative one can use a global index label, such as service-type, and as values well known tokens which identify the given service, such as service-a. In this case one should look for those objects which have the given value for the service-type index.

3.3   Service binding

Error

FIXME: To be revised!

Suppose we have two service types, each with multiple instances, and we want to pair one service instance from one type to one from the other type. As a clear example we have multiple web servers that each must connect to a given application server, the mapping being chosen by the operator.

The solution is similar to that of service discovery, but it involves the usage of links.

Each service instance will have its own configuration object, following one of the schema given in the service configuration use case, and another one for the end-point, following the schema given in the service discovery use-case (in fact only the application server needs one such end-point object). Then each object belonging to the front-end servers will have an additional link pointing to the chosen application server.

The resulting URL's are used by the services both for configuration and binding:

.../objects/application-1/data
for the configuration of the first application server; (created by the operator, and accessed only by the application server;)
.../objects/application-1-endpoint/data
for the endpoint of the first application server; (created by the application server, and accessed by those front-end servers that need to connect to it;)
.../objects/front-end-1/data
for the configuration of the first front-end server; (created by the operator, and accessed only by the front-end server;)
.../objects/front-end-1-endpoint/data
for the endpoint of the first front-end server; (created by the front-end server, and accessed by its clients;)
.../objects/front-end-1/links/application-server-endpoint/0/object/data
for the end-point of the application server designated by the operator to be used by the first front-end server, and pointing to something like .../objects/application-x-endpoint/data; (the link is created by the operator;)

Like in previous use-cases if the service continuously pools the given URL's, it can easily detect any reconfiguration done by the operator without intervention on his behalf.

4   Basic object operations

Snippet execution

One can easily execute a sequence of these snippets by following the procedure described on the dedicated page.

Creating an empty object

## create:o-0
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary '{}' \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-0

Selecting the empty object

## select:o-0
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-0 \
| _json_format

Deleting the empty object

## destroy:o-0
curl -v -s -X DELETE http://127.0.0.1:8080/v1/collections/c-1/objects/o-0

Creating an object with JSON data

Important

When using JSON as data, the data.content field must be the actual JSON expression, thus not encoded as a string.

## create:o-1
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-1 \
    <<'EOS'
{
    "data" : {
        "content-type" : "application/json;charset=utf-8",
        "content" : {
            "k-s" : "s", "k-i" : 0, "k-t" : true, "k-n" : null,
            "k-l" : [0, 1, 2],
            "k-o" : {"k-f" : false, "k-i" : 1}
        }
    }
}
EOS

Retrieving the object with JSON data

## select:o-1
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-1 \
| _json_format

Retrieving only the object's JSON data

## select:o-1:data
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-1/data \
| _json_format

Updating only the object's JSON data

Important

When using JSON as data, the request's body must be a valid JSON encoded expression.

## update:o-1:data
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-1/data \
    <<'EOS'
{
    "k-s" : "s", "k-i" : 0, "k-t" : true, "k-n" : null,
    "k-l" : [0, 1, 2],
    "k-o" : {"k-f" : false, "k-i" : 1}
}
EOS

Deleting only the object's JSON data

## destroy:o-1:data
curl -v -s -X DELETE http://127.0.0.1:8080/v1/collections/c-1/objects/o-1/data

Creating an object with text data

## create:o-2
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-2 \
    <<'EOS'
{
    "data" : {
        "content-type" : "text/plain;charset=utf-8",
        "content" : "line-1\nline-2\n"
    }
}
EOS

Retrieving only the object's text data

## select:o-2:data
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-2/data

Updating only the object's text data

## update:o-2:data
curl -v -s -X PUT -H 'content-type:text/plain;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-2/data \
    <<'EOS'
line-1
line-2
line-3
line-4
EOS

Creating (or updating) an object with binary data

## create:o-3:data
curl -v -s -X PUT -H 'content-type:application/octet-stream' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-3/data \
    < <( head -c 16 </dev/urandom )

Retrieving only the object's binary data

## select:o-3:data
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-3/data \
| hexdump

Listing all objects inside a collection

## select:c-1
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects \
| _json_format

Complete runs

_snippets_execute \
        create:o-0 select:o-0 \
        destroy:o-0 select:o-0 \
        destroy:o-0
_snippets_execute \
        create:o-1 select:o-1 \
        select:o-1:data update:o-1:data select:o-1:data \
        destroy:o-1:data select:o-1:data select:o-1
_snippets_execute \
        create:o-2 select:o-2:data \
        update:o-2:data select:o-2:data
_snippets_execute \
        create:o-3:data select:o-3:data
_snippets_execute select:c-1

5   Advanced object operations

5.1   Using indices

Creating an indexed object

## create:o-4
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-4 \
    <<'EOS'
{
    "indices" : {
        "i-i" : [0],
        "i-s" : ["a", "b"]
    }
}
EOS

Retrieving only the object's indices

## select:o-4:indices
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices \
| _json_format
## select:o-4:indices:i-1
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices/i-i \
| _json_format
## select:o-4:indices:i-s
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices/i-s \
| _json_format

Updating the object's indices

## update:o-4:indices
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices \
    <<'EOS'
{
    "i-i" : [0, 1, 2],
    "i-s" : null,
    "i-b" : [true]
}
EOS
## update:o-4:indices:i-b
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices/i-b \
    <<'EOS'
[true, false]
EOS
## select:o-4:indices:i-b
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices/i-b \
| _json_format

Deleting the object's indices

## destroy:o-4:indices:i-b
curl -v -s -X DELETE http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices/i-b
## destroy:o-4:indices
curl -v -s -X DELETE http://127.0.0.1:8080/v1/collections/c-1/objects/o-4/indices

Selecting objects with the index

## create:o-g
for i in {1..100} ; do
    curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
            http://127.0.0.1:8080/v1/collections/c-1/objects/o-g-"$i" \
        <<EOS
    {
        "indices" : {
            "i-g" : [$i]
        }
    }
EOS
done
## select:i-g:e
curl -v -s -X GET http://127.0.0.1:8080/v1/indices/i-g/select/equals/10 \
| _json_format
## select:i-g:l
curl -v -s -X GET http://127.0.0.1:8080/v1/indices/i-g/select/lesser/10 \
| _json_format
## select:i-g:g
curl -v -s -X GET http://127.0.0.1:8080/v1/indices/i-g/select/greater/90 \
| _json_format
## select:i-g:r
curl -v -s -X GET http://127.0.0.1:8080/v1/indices/i-g/select/range/10/19 \
| _json_format

Complete runs

_snippets_execute \
        create:o-4 select:o-4:indices select:o-4:indices:i-1 select:o-4:indices:i-s \
        update:o-4:indices select:o-4:indices \
        update:o-4:indices:i-b select:o-4:indices:i-b \
        destroy:o-4:indices:i-b select:o-4:indices:i-b select:o-4:indices \
        destroy:o-4:indices select:o-4:indices
_snippets_execute \
        create:o-g \
        select:i-g:e \
        select:i-g:l \
        select:i-g:g \
        select:i-g:r

5.3   Using attachments

Error

FIXME: To be written!

Warning

This feature is not supported in the initial version!

5.4   Using annotations

Error

FIXME: To be written!

5.5   Using all options at once

Creating a complex object

## create:o-c
curl -v -s -X PUT -H 'content-type:application/json;charset=utf-8' --data-binary @- \
        http://127.0.0.1:8080/v1/collections/c-1/objects/o-c \
    <<'EOS'
{
    "data" : {
        "content-type" : "application/json;charset=utf-8",
        "content" : {
            "k-s" : "s", "k-i" : 0, "k-t" : true, "k-n" : null,
            "k-l" : [0, 1, 2],
            "k-o" : {"k-f" : false, "k-i" : 1}
        }
    },
    "indices" : {
        "i-i" : [0],
        "i-s" : ["a", "b"]
    },
    "links" : {
        "l-a" : [
            {"collection" : "c-1", "object" : "o-1"},
            {"collection" : "c-1", "object" : "o-2"}
        ],
        "l-b" : [
            {"collection" : "c-1", "object" : "o-3"}
        ],
    },
    "annotations" : {
        "a-1" : "s",
        "a-2" : [0, true, null, {}],
        "a-3" : {"k" : false}
    }
}
EOS

Retrieving the complex object's various attributes

## select:o-c
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c
## select:o-c:data
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/data
## select:o-c:indices
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/indices
## select:o-c:indices:i-i
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/indices/i-i
## select:o-c:links
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/links
## select:o-c:links:l-a
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/links/l-a
## select:o-c:links:l-a:0
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/links/l-a/0
## select:o-c:links:l-a:0:object
curl -v -s -L -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/links/l-a/0/object
## select:o-c:links:l-a:0:object:data
curl -v -s -L -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/links/l-a/0/object/data
## select:o-c:attachments
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/attachments
## select:o-c:attachments:a-a
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/attachments/a-a
## select:o-c:attachments:a-a:data
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/attachments/a-a/data
## select:o-c:attachments:a-a:annotations
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/attachments/a-a/annotations
## select:o-c:annotations
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/annotations
## select:o-c:annotations:a-1
curl -v -s -X GET http://127.0.0.1:8080/v1/collections/c-1/objects/o-c/annotations/a-1

Complete run

_snippets_execute \
        create:o-c \
        select:o-c \
        select:o-c:data \
        select:o-c:indices select:o-c:indices:i-i \
        select:o-c:links select:o-c:links:l-a select:o-c:links:l-a:0 \
        select:o-c:links:l-a:0:object select:o-c:links:l-a:0:object:data \
        select:o-c:attachments select:o-c:attachments:a-a select:o-c:attachments:a-a:data \
        select:o-c:attachments:a-a:annotations \
        select:o-c:annotations select:o-c:annotations:a-1

6   Exporting and importing

Exporting

## export
curl -v -s -X GET http://127.0.0.1:8080/v1/export \
    >/tmp/mosaic-object-store--database.json

Importing

## import
curl -v -s -X POST http://127.0.0.1:8080/v1/import \
        -H 'content-type:application/json;charset=utf-8' --data-binary @- \
    </tmp/mosaic-object-store--database.json

7   Replication

The replication process has three phases: defining on the target (i.e. the server) a replication stream, which yields a token used to authenticate the replication; defining on the initiator (i.e. the client) a matching replication stream; and the actual replication which happens behind the scenes.

It must be noted that the replication is one way, namely the target (i.e. the server) continuously streams updates towards the initiator (i.e. the client). If two-way replication is desired, the same process must be followed on both sides.

Regarding conflicts, and because internally the object store exchanges "patches" which only highlight the changes, any conflicting patch is currently ignored. It is therefore highly recommended to confine updates to a certain object only to one of the two replicas. However if multiple changes happen to the same object, and multiple patches are sent, and say the first one yields a conflict, but the rest don't, only the conflicting patch will be discarded, the others being applied.

It is possible to obtain replication graphs or trees, including cycles, and the object store handles these properly.

Preparing the replication

## replication preparation
curl -v -s -X POST http://127.0.0.1:8080/v1/replication/stream-1 \
        -H 'content-type:application/json;charset=utf-8' --data-binary @- \
    <<<'{}'
{
    "token" : "4e4ee45e75526deb4ce98ed6503df07c"
}

Initiating the replication

## replication initiation
curl -v -s -X PUT http://127.0.0.2:8080/v1/replication/stream-1 \
        -H 'content-type:application/json;charset=utf-8' --data-binary @- \
    <<'EOS'
{
    "target" : "http://127.0.0.1:8080/v1",
    "token" : "4e4ee45e75526deb4ce98ed6503df07c"
}
EOS

Terminating the replication

curl -v -s -X DELETE http://127.0.0.1:8080/v1/replication/stream-1
curl -v -s -X DELETE http://127.0.0.2:8080/v1/replication/stream-1

8   Deployment

Error

FIXME: To be written!