2.2.1   Overview

In this scenario the EU (end-user) has just signed an SLA which on implementation will trigger the provisioning of a few secure web containers, that in this scenario, and in accordance with the signed SLA, are to be deployed on Rackspace's cloud infrastructure as VM's under the control of the SPECS platform.

The key aspect on which we focus in this scenario, pertaining to the subject of credential management, is how the request to create the needed VM's on Rackspace is to be authenticated against the desired Rackspace credentials which, depending on the interacion model, although not important for this scenario or the credential service, could belong to either the EU (end-user) or the SPECS platform owner.

Technically during the SLA implementation phase, there will be a component, part of the enforcement module, that will need to trigger the provisioning of the required VM's, and following that, the their proper configuration. By looking at the Rackspace API reference [pafu] we know that the implementation component in question will need to first obtain the credentials (consisting of a user name and matching password); following with a call to a Rackspace service to obtain an authentication token; only after which it can continue with the actual provisioning requests, by passing along the obtained authentication token.

2.2.2   Security concerns

Functionally there is no issue with the above workflow, except the concerns related to proper handling of the Rackspace credentials (i.e. the user name and password).

The first issues that are to be tackled are related to the conveying of those raw credentials, from either the EU (end-user) or a specialized SPECS component, to the implementation component that has to use them. This includes mutual authentication between the two parties, and the credential data exchange through a secure channel.

However once these credentials are passed over to the implementation module, all control over their usage relies solely on the correct implementation and behaviour of that delegated module, which could use those credentials for any other purpose, for as long as the credentials are valid (usually a matter of months on end).

Therefore the EU (end-user) is exposed to quite a few major risks, including improper credential usage, or even worse leakage to other parties, all of which could have, besides business disruption, also monetary losses.

2.2.3   Proposed solution

Unfortunately not much can be done to obtain proper security in absolute terms, which includes restricting the credential usage only for the purpose of VM creation and nothing else. This impossibility stems due to the way the Rackspace security model has been designed, namely once authenticated, the obtained token can be used for any purposes until it either expires (a matter of days) or it is explicitly revoked.

However SPECS, through the introduction of the credential service, could at least reduce the time frame in which the authentication token can be used for other purposes.

The solution is quite straight forward and implies running half the Rackspace authentication process by the credential service, and only the other half delegated to the implementation component.

Specifically, the credential service which holds the Rackspace credentials, will upon being requested by the implementation module, authenticate to Rackspace and obtain the needed authentication token, passing it in response to the implementation module, which in turn will use it to provision the required VM's for the secure web container deployment. However at the same time the credential service sets a timer on the obtained token, on whose triggering will revoke the authentication token, thus rendering it unusable after revocation.

Needless to say that the interactions between the credential service and the implementation module are themselves authenticated and secured, just like between any two SPECS components, possibly by the usage of the SPECS security tokens service for authentication (and possibly also for authorization), and TLS for confidentiality.

2.2.4   Discussion

Thus, like previously described, the proposed credential service can not solve the issue of leakage of the authentication token by the implementation module, but it can instead reduce the time frame during which an attack might successfully use it; nor does it solve the issue of improper usage of the token for other operations than VM creation.

As noted these limitations are not the fault of the proposed solution, namely the credential service, but that of the shortcomings of the Rackspace authentication scheme.

2.3.1   Overview

A second scenario where we highlight the potential usage of the proposed credential service, is that when the EU (end-user) has signed an SLA, which at implementation requires the provisioning of a secure storage module that will allow the EU (end-user) (or a delegated party) to securely store data on Amazon S3.

For this scenario the security module consists of a service which is handed the encrypted data to be stored on Amazon S3 on behalf of the EU (end-user). Technically this implies that the storage service will require to gain access to the EU (end-user) Amazon AWS credentials consisting of an access key (similar to an user name) and secret key (similar to a password) that are to be used each time a request is to be made to Amazon S3 on behalf of the EU (end-user).

By looking at the Amazon API reference [yixe] we observe that, unlike Rackspace's authentication scheme, the storage service needs for each individual request the actual Amazon AWS credentials, which are cryptographically combined with the request data to uniquely authenticate each individual request.

2.3.2   Security concerns

Just like in the previous scenario, the main issues are related to conveying the credential data, and the lack of control over their usage, therefore no other discussion is required, all the previous observations applying.

It must however be granted that Amazon does provide quite extensive access control policies which could limit the actions that can be carried with a certain set of credentials. However the usage of such policies are quite involving, requiring the switch to Amazon IAM, proper policy creation and enforcement, and proper credential life-cycle.

Therefore although it is practically possible to limit the allowed operations for a given set of credentials, like only being able to read and write in a given Amazon S3 bucket, this however limits only the types of operations that can be done with a potential leaked set of credentials, and not the time frame during which an attack is possible.

Moreover the type of operations, an the target resources, are quite coarse, and in some scenarios, like possibly the secure storage, there might be a need for a more fine-grained approach.

2.3.3   Proposed solution

Just like in the case of Rackspace our proposal is to delegate the actual authentication procedure to the credential service, meanwhile leaving the actual enactment of the request to the implementation module (in this case the secure storage service).

Although at a first glance there is even less that can be done by the credential service, because according to the Amazon API reference [yixe] the actual credentials are needed for each individual request, therefore once exposed to the implementation service, they can't be "taken" back. (One could revoke the actual credentials, but this is not an operation to be done time and time again.)

However on closer inspection of the mentioned API, we can observe that the credential service could have an even greater role in how the credentials are used. The solution requires the implementation service to execute the first half of the authentication procedure by computing all the required fingerprints of the request meta-data and data, and passing this along to the credential service with other information identifying the requested operation (like the region, operation, resource, etc.). Then based on this partial data uniquely identifying the each request, the credential service can complete the authentication process by cryptographically combining the partial data with the actual credentials, obtaining the authenticating signature unique to the request in question, which then can be passed back to the implementation service.

2.3.4   Discussion

As described above, the proposed solution not only completely eliminates the need for the implementation module to have access to the Amazon AWS credentials, but allows the credential service to impose further restrictions on their usage, which possibly are not provided by the generic Amazon AWS policies.

Furthermore if the implementation service does leak the obtained authenticating signature, it is useless to an attacker because they can only be used to authenticate an identical operation to the signed one (thus in the worst case the same operation is executed twice), and furthermore the authentication token has only a time-window of about 30 seconds.

3.2.1   Target service authentication schemes support

Obviously the most important requirement is that of implementing the relevant part of the authentication protocol of the chosen target service, namely the one that requires access to the credentials.

However this requirement can not be fulfilled by the credential service alone, but in cooperation with the actual client of the target service, especially since for some target services (like those of Amazon AWS) the authentication scheme requires as input meta-data uniquely characterizing the request in question.

As stated in the overview, with this requirement a generic one, not clearly indicating a list of target services. However this list will be identified as part of other SPECS deliverables and activities, and once that list of target services is finalized, this requirement will actually translate into a couple of requirements, one per target service, stating "support AWS services authentication scheme", or "support OpenNebula services authentication scheme", etc. Therefore in analyzing the achievement of this requirement, each of the resulting concrete requirements must be taken in isolation and not as a whole.

It must also be noted that not all the target services could benefit from the credential service in the same amount. Like we have described in the sections related with security concerns in each of the two possible usage scenarios, the level of sophistication, and therefore the inherent flexibility and security, of various target services varies from almost none (i.e. where the actual credentials are sent in plain with each request), to a token-based level (like in the case of Rackspace services), to a per-request signature level (like in the case of Amazon AWS services). Therefore the level of achievement of each specific requirement will vary in comparison from case to case, depending on the particularities of the target service, and as such being outside the control of the developer.

3.2.2   Access control policies to the credential usage

In addition to eliminating the need for target service clients to get access to the credential data, there should be a way to limit their access to the actual usage of such credentials.

Therefore there should be a way to impose access policies at two levels (desirably, but the second one being optional). The first level, implementable in all cases, would control if a certain client can actually use the credentials for a certain target service. Meanwhile the second level, implementable only in some cases, controls for what purposes should the client be allowed to use the credentials in question.

For example in case of Rackspace credentials only first level policies can be applied, given that once a token is generated its usage can not be controlled further (except revoked). Meanwhile for Amazon AWS credentials the policies could expand not only to the second level, namely the allowed actions, like say creating or destroying a VM, or accessing a certain Amazon S3 bucket, but even a third level controlling to which resources should the operations be applied to, like say create an Amazon S3 object, but not be able to delete or overwrite it. (Granted that in the case of Amazon S3 these policies could be directly enforced by the target services themselves, as discussed in the corresponding scenario.)

3.2.3   Multiple credentials for the same target service

Taking into account that one of the interaction models targeted by SPECS is that in which a single SPECS platform instance plays the role of a shared PaaS, by offering the same services to multiple EU (end-user), thus sharing the same infrastructure and consequently the same software components. It is therefore mandatory to be able to handle multiple credentials for the same target service, one (or perhaps multiple) credentials belonging to an individual EU (end-user) and being delegated to SPECS to act on behalf of the EU (end-user).

As such the same software component, playing the role of target service client, might concurrently interact with the same target service, but on behalf of different EU (end-user)'s, hence the requirement of being able to concurrently access different credentials of the same target service, which the credential service must be able to provide.

However it must be noted that this requirement is as much for the credential service as it is for the clients of the target services, and thus clients of the credential service itself.

3.3.1   Credential usage auditing

Due to the cloud's model of "pay as you go", any interaction with a target service most likely results in costs being incurred on behalf of the EU (end-user) owning the credentials. Moreover most of the actions are irreversible, and no recovery is possible (without possibly interacting directly with the cloud provider support team), like for example removing data from an Amazon S3 bucket or terminating a VM.

Therefore in the case of incident management an audit of the usage of the credentials is invaluable, and should contain enough information to pin-point the software component that issued the request (and possibly enough information to track the initial request that triggered the chain of actions).

Like in the case of access control policies, multiple levels of auditing would prove useful, the first one indicating when and by whom certain credentials ware used; the second level indicating which specific operations were authorized. Unfortunately, just like in the access control policies, the ability to achieve these levels depends mostly on the actual authentication scheme in question.

3.3.2   Disjoint credential data management and storage

Because no software is without faults, even the proposed credential service might be the target of (successful) attacks. Therefore in order to mitigate the amount of leaked data it would prove useful to separate the different concerns of credentials handling as follows.

One software component should store the credential data (itself or by delegation to a third component), encrypted without having access to the decryption keys. Another software component should have the decryption keys, and each time credentials are needed, requesting them from the first component that just stores them, using them and afterwards discarding them. (A certain small period of caching is foreseen as a solution to performance issues.)

4.2.1   STS via IAM roles assigned to EC2 VM profiles

The simplest, and most secure, method of leveraging Amazon STS is through the usage of Amazon IAM roles for Amazon EC2 VM's. Technically this involves defining an Amazon IAM role, then assigning it a set of permissions through Amazon IAM policies, followed by creating an Amazon EC2 VM profile bound to this Amazon IAM role, and finally by choosing the this profile when launching a VM. Once the VM has started, through a series of Amazon STS API calls, the applications running on the VM can use the resulting temporary credentials, achieving thus access delegation.

It must be noted right from the start that if one strictly (and fully) follows the cloud deployment best practices, namely having one and only one service per VM, and furthermore that each service has always the same security access requirements, then by the usage of Amazon IAM roles and Amazon STS, one can obtain a (practically) sufficiently secure solution.

There are however certain limitations to this solution, especially in the case when on the same VM one needs to run multiple services, like in the case of shared PaaS (one of the SPECS's main interaction models), where each service with its own security clearances. And because any process running on the given VM can use any of the privileges that have been granted to that VM, it implies that one can not delegate different privileges to different services.

Furthermore this solution works best when the resources, that are the target of API calls, belong to the same Amazon AWS account, therefore cross-account delegation, although not impossible, is pretty difficult to set-up. (Moreover for some services cross-account delegation is not possible.)

4.2.2   STS via delegation service

Despite the limitations of the previous technique, one of the big advantages is the out-of-the-box functionality, in that everything can be achieved only by using native Amazon AWS services and API's without requiring extra work from the EU (end-user).

However in order to solve the flexibility limitations, one could leverage Amazon STS in a software component, which when asked would create a temporary set of credentials and delegate them to the requesting service, which in turn would use them to control the targeted resources. This would allow for example to have a cluster of services that are delegated access to a large number of credentials, not necessarily part of the same Amazon AWS account.

The way such a solution would work is very similar to our proposed credential service, and therefore does not need to be discussed further.

4.2.3   Discussion

Although Amazon STS is a very powerful tool when it comes to cloud access delegation, it does have deeper disadvantages that the limitations pretested previously. In what follows we shall highlight some of these disadvantages, which are in play regardless of the variant in which Amazon STS is used, either via Amazon IAM roles or via a delegation service.

It must be noted however that in itself, and disregarding the issues presented hereafter, Amazon STS is practically sufficient, offering much better security than any other existing solution.

However perhaps the greatest disadvantage of Amazon STS is its lack of interoperability, in that it is useful only when dealing with Amazon services. Even though Amazon's API, especially the one for Amazon EC2 and Amazon S3, has been copied by other software vendors, like Eucalyptus or OpenStack, the Amazon STS service is unique to Amazon and is not available in any other compatible solution.

Putting aside the interoperability issues, and focusing solely on security aspects, there are further limitations which could negatively impact its usefulness. The first issues is related to the time frame in which temporary credentials (potentially leaked by a compromised service) can be used to inflict damage, mainly due to the fact that there is no way to revoke these temporary credentials once issued. Therefore a compromise has to be made between availability and security, because in order to increase the security one needs to decrease the time frame in which these temporary credentials are active, therefore their expiration interval, which in turn requires more frequent temporary credential creation, that in the case the delegation service is down would imply reduced availability.

The second issue is related to how specific the privileges can be for the temporary credentials (or any Amazon AWS credential for that matter). Although one can assign standard Amazon AWS policies to further restrict the access of the temporary credentials, they can go only so far as specifying actions to be taken and in some cases coarse rules to match the affected targets. It is however difficult to pin-point access to say only terminate a certain VM, or give access only to certain Amazon S3 paths that match a given regular expression.

Nonetheless, even disregarding all of these interoperability and security issues, there still remains the fact that a piece of software has access to credentials, even though temporary and tightly restricted, which can be easily leaked due to a software bug.

5.4.1   Current architecture

For a sketch of the architecture one can look at the figure below, where the involved entities and the sequence of the calls are clearly identified.

Architecture of the credential service (current version)

As seen from the previous phases, the actual operations involved, and the level of security, depends solely on the security protocol at hand.

5.4.2   Future architecture

Although functionality wise the current architecture is good enough to fulfill the practical needs of any target service client, the following figure presents an evolution which also fulfills the non-functional requirement of separating the credential management from its storage, which is mandated by the need to limit the amount of damage caused by a possible successful attack on any of the credential service components.

However in practical terms, the evolution towards this future architecture does not impact in any way existing credential service clients, as the API between the client and the current credential service will remain the same as the one between the client and the credential manager, the credential store being hidden.

Architecture of the credential service (future version)

8.1.1   Credential service API issues

First of all, we believe that the HTTP API that the clients have to use to obtain the call authenticators is both too "heavy" (especially in the case when the calls are made quite often), and opens a new set of potential security issues, in addition to exposing yet another running process to the network.

Secondly, the usage of the HTTP protocol is almost unnecessary, the reason being solely because almost any programming language (including shell scripting) provides a technique to execute such HTTP calls. However the burden of correctly implementing the handling of status codes, content negotiation, content encoding, and other technicalities is pushed to the developer of the credential service library (at least one for each programming language).

8.1.2   A simple portable solution

Therefore looking at how a similar project solves this issue, namely factotum from the Plan9 ecosystem [zoti], we propose to use a synthetic file-system mounted on each host, one that the clients shall use to delegate the credential operations. (In essence we are only replacing the "transport layer" if we might say so.)

At first sight it might seem a more complex solution than the HTTP-based one, however absolutely all programming languages offer access to the file-system, usually through a well established API|'s, like the POSIX open / read / write / close family of operations.

8.1.3   Example usage snippets

For example one could envisage the following usage of the credential manager directly from the shell, which mirrors the curl calls presented in the previous section.

$$$ ls /mos/services/credential-manager/modules

>>> aws
rackspace
open-nebula

$$$ ls /mos/services/credential-manager/modules/aws/operations

>>> query-auntenticate-v4

$$$ declare _token="$( uuidgen -r )"

$$$ mkdir "/mos/services/credential-manager/modules/aws/operations/execution/${_token}"

$$$ cat \
     >"/mos/services/credential-manager/modules/aws/operations/execution/${_token}/inputs" \
<<'EOS'
// the same input JSON like in the case of `curl` example
EOS

$$$ echo "execute" \
    >"/mos/services/credential-manager/modules/aws/operations/execution/${_token}/control"

$$$ cat \
    <"/mos/services/credential-manager/modules/aws/operations/execution/${_token}/outputs"

>>> // the same output JSON like in the case of `curl` example

$$$ rmdir "/mos/services/credential-manager/modules/aws/operations/execution/${_token}"

8.1.4   Advantages

Although looking at the previous examples, one could conclude that the process is more involved and thus the overhead is greater (there are 5 operations instead of 1), however the actual number of system calls is comparable to the HTTP-based API (if not lesser in practice). For example, per credential operation execution, in the case of the HTTP API we have one socket creation, binding and connection, at least one read and write, one socket shutdown and one close system calls, thus 7 in total. In the case of the file-system based API we have one directory creation, two open and close, one read and write, and one directory deletion, thus 8 in total. However we are removing the network overhead, and if needed one can reduce the number of system calls to only four in total (one open, write, read and then close).

But the efficiency is not the only reason, in fact it is just a by-product of the technology. Instead the main advantage is that of a much more constrained and well defined access control, because now the authentication and authorization is done by the operating system kernel (which implements the system calls), meanwhile the credential manager only having to implement the operations. (Not to mention that the service is not exposed to the network.)

Therefore the only way in which an attacker can gain access to the credentials is if the operating system itself is compromised.

8.1.5   Disadvantages

There are however drawbacks, especially since now we require one credential manager to run on each host. However this is not a major one, because mOS (the OpenSUSE-based Linux distribution that SPECS leverages) eases the execution of such specialized services.

Another potential issue is how the credential manager obtains the credential data, which gets us to the credential store (presented in the next section), that is a remote service which provides access to such credential data, which the manager caches for performance reasons. However even this interaction follows the same pattern of file-system mediated communication (as the credential store will have on each host a proxy).

Therefore in this new scheme, as opposed to the current credential service prototype, we only need to secure the interaction between two services (the credential store and credential store proxy), both of which are designed and implemented with security in mind.

8.1.6   Implementation concerns

In terms of implementation we consider that FUSE [xoce] is the best approach (9P [gihu] being the runner-up), because we have access to more information about the caller (such as process identifier, and knowing that we can easily obtain user and group information, executable, etc.)

The programming language is not important as long as there is a FUSE library available; it must however allow dynamic code loading.

8.2.1   Security concerns

The set of hard to solve security-related issues are briefly presented in what follows.

How the credential store service obtains the encryption key?

Basically in a cloud environment this is the weakest link, and no fully secure solution exists. One possibility is to convey, via the so called "user-data", the private key used to decrypt the credential data obtained from the general purpose store. Another possibility is to use a HSM that provides access to operations on such a private key; however such a solution is prohibitively expensive.

How to secure the communication between the credential store service and credential store proxy?

The obvious solution is a TLS-based communication channel with both client and server certificate-based authentication. However in such a case the problem is transformed into how to convey the required private keys to the two entities, which can be solved in a similar fashion like the previous problem.

How to enforce access policies to the credentials?

One potential solution is to encode, besides the actual credential data, also an access policy which states who is able to obtain access to that data. Such a policy can then be enforced by the credential store service, based on the identity of the credential store proxy (which is taken to be either the client public key obtained from the TLS negotiation, or a special attribute thereof.) Moreover such policies can be further conveyed, through the proxy, to the credential manager, which can further narrow down the kind of processes that can execute operations on them.

8.2.2   Implementation concerns

Although not many implementation details have been ironed out, there already exists previous work, namely secstore also from the Plan9 ecosystem [zoti].