During automatic infrastructure deployment on AWS, a common question is: what is the best way to deliver sensitive information over to EC2 instances or, more precisely applications running on them. There are numerous solutions, such as placing the information into user-data initialization script or simply SFTPing them onto the instance. Although these are perfectly viable solutions, there are well-known drawbacks with those approaches, such as size limitations on the former and the necessity to open the SSH port for the latter. There are also comprehensive solutions, such as HashiCorp Vault with Consul, that can do a lot more than just deliver credentials, but those can be an overkill for common and simple scenarios.
There is a way to solve secret management through utilization of resources provided only by AWS and a cool tool called credstash. You will find a nice guide on how to use the tool and a description on how it works if you follow the link, but the basic idea behind credstash is that it stores key/value data in DynamoDB while encrypting values with KMS (Key Management Service). As a result, only a user or resource that has read access to the DynamoDB table and permission to use that KMS Master key can access that data. The encrypted data can then be accessed through a very simple command line interface. In the most simple case the process would look like this:
On your laptop:
$ credstash put my-secret high-entropy-password
my-secret has been stored
On the EC2 instance:
$ credstash get my-secret
high-entropy-password
Boom, you transferred the password across the internet in a
totally secure fashion using nothing but AWS services. At a high
level, here is what happened in the above example. During the
put
operation:
high-entropy-password
was encrypted with
that key.my-secret
.The KMS Master key that is used by default by credstash is the
one with the name alias/credstash
, while the default
DynamoDB table name is credstash-store
. This whole key
wrapping technique is necessary, because the KMS Master key can
only encrypt up to 4KiB of data at a time.
During the get
operation the process is
inverted:
my-secret
As you probably suspect, access to the secret data can be controlled on two levels, namely through access to DynamoDB table and to KMS Master key. More on that later.
There are a number of tools that are used for automatic deployment on AWS. Terraform, being one of them, stands out as an amazing tool that allows you to describe your infrastructure as code. It works not just with AWS, but with many other providers. In this post we’ll use nothing but terraform, so if you are already familiar with it, go on reading forward; otherwise a Getting Started tutorial could be beneficial if you want to try things out while moving along.
Installing terraform is pretty straightforward, since it is written in Go you can just download a binary for your operating system from terraform downloads page.
Credstash, on the other hand, is written in Python and as such
can be installed with pip
. It does have a few
non-python dependencies that need to be installed beforehand. Here
is how you’d get it on Ubuntu:
$ sudo apt-get update
$ sudo apt-get install -y libssl-dev libffi-dev python-dev python-pip
$ sudo -H pip install --upgrade pip
$ sudo -H pip install credstash
If you’d rather not install anything globally you can use Python
environments, or even download another implementation of credstash
ported to a different language, for instance gcredstash, written in Go, which, just like terraform,
can be downloaded as a static executable and is fully compatible
with credstash
. Implementations in other languages are
listed in the README.
Naturally, the example from Introduction
will not work just out of the box, so prior to using credstash, a
database table and an encryption key must be created first. Going
through the credstash documentation will reveal that a DynamoDB
table with a default name credstash-store
can be
created by running credstash setup
, while the KMS
Master key has to be created manually:
$ credstash -t test-table setup
Creating table...
Waiting for table to be created...
Table has been created. Go read the README about how to create your KMS key
Well that’s no fun, we ought to be able to automate the whole
process. The
credstash-setup terraform module will do just that, thereby
taking care of the initial setup for us. Remember, we need to do
this only once and make sure not to run terraform
destroy
, unless you really want your secret data to be
permanently deleted.
Create a main.tf
file:
module "credstash" {
source = "github.com/fpco/fpco-terraform-aws/tf-modules/credstash-setup"
}
Then execute it in the same folder with the above file:
$ terraform get
$ terraform apply
Once applied, terraform will create a DynamoDB table with the
name credstash-store
and a KMS key with the name
alias/credstash
. After deployment is complete, you can
go ahead and start using credstash
on your local
machine.
Although, it is not strictly required, I would highly recommend using terraform’s remote state feature in order to later simplify getting the values created by this setup. We even have a terraform module that can help you with getting the s3-remote-state bucket that was created.
terraform {
backend "s3" {
encrypt = "true"
region = "us-west-1"
bucket = "remote-tfstate"
key = "credstash/terraform.tfstate"
}
}
module "credstash" {
source = "github.com/fpco/fpco-terraform-aws/tree/master/tf-modules/credstash-setup"
}
The main benefit of using remote terraform state is that credstash-related resources can be created just once and their reuse can be automated during our infrastructure deployment by all of the team members. Another more involved way would be to manually copy and paste outputs of this module into others as input variables, and that just sounds like too much work.
So far, usage of credstash was limited only to users that have
implicit access to all KMS keys and DynamoDB tables, i.e. admins,
power users, what have you. Basically, running
credstash
on an EC2 instance will result in a
permission error, but that is where it is most useful. The best way
to allow an EC2 instance access the secrets is to:
credsatsh-store
, and attach those
policies to the above mentioned role.The first two steps we can easily automate with terraform, but
the last step has to be done with aws-cli
or directly
through an API with some SDK. But wait, I said that we won’t be
using anything besides terraform, and it is so indeed,
aws-cli
is an implicit dependency, which has to be
installed despite the fact that we will not be interacting with
directly.
Let’s start with creating IAM policies first, as they can be reused as many times as we’d like.
... # also remote state, just as above
module "credstash" {
source = "github.com/fpco/fpco-terraform-aws/tree/master/tf-modules/credstash-setup"
enable_key_rotation = true
create_reader_policy = true
create_writer_policy = true
}
output "kms_key_arn" {
value = "${module.credstash.kms_key_arn}"
}
output "reader_policy_arn" {
value = "${module.credstash.reader_policy_arn}"
}
output "writer_policy_arn" {
value = "${module.credstash.writer_policy_arn}"
}
output "install_snippet" {
value = "${module.credstash.install_snippet}"
}
output "get_cmd" {
value = "${module.credstash.get_cmd}"
}
output "put_cmd" {
value = "${module.credstash.put_cmd}"
}
One of the greatest features of terraform, in my opinion, is
that it knows exactly what needs to be done in order to reach the
desired state, so if you already called terraform
apply
in the previous example, it will figure out everything
that needs to be changed and apply only those changes without
touching resources that need no modification.
When you run terraform apply
you should see
something along the lines:
Outputs:
get_cmd = /usr/local/bin/credstash -r us-east-1 -t credential-store get
install_snippet = { apt-get update;
apt-get install -y build-essential libssl-dev libffi-dev python-dev python-pip;
pip install --upgrade pip;
pip install credstash; }
kms_key_arn = arn:aws:kms:us-east-1:123456789012:key/87b3526c-8100-11e7-9de5-4bff2f10d02a
put_cmd = /usr/local/bin/credstash -r us-east-1 -t credential-store put -k alias/credstash
reader_policy_arn = arn:aws:iam::123456789012:policy/credential-store-reader
writer_policy_arn = arn:aws:iam::123456789012:policy/credential-store-writer
At this point credstash is set up and we can verify that it works. Helper snippets are targeted at Ubuntu based systems, but can be easily adapted to other operating systems.
Let’s install credstash on a local machine, store a test value and pull it out from credstash-store afterwards:
$ sudo -H bash -c "$(terraform output install_snippet)"
...
$ $(terraform output put_cmd) test-key test-value
test-key has been stored
$ $(terraform output get_cmd) test-key
test-value
We can also set a new value for the key, while auto incrementing
its version, by setting -a
flag:
$ $(terraform output put_cmd) -a test-key new-test-value2
test-key has been stored
$ $(terraform output get_cmd) test-key
new-test-value2
There are a few other useful features that credstash has, which
don’t have helper snippets like get_cmd
and
put_cmd
do, since they are less likely to be used in
automated scripts. But they can still be easily constructed using
terraform outputs. It’s worth noting that all previously stored
values are always available, unless deleted manually:
$ credstash -r us-east-1 -t credential-store get test-key -v 0000000000000000000
test-value
$ credstash -r us-east-1 -t credential-store list
test-key -- version 0000000000000000000
test-key -- version 0000000000000000001
$ credstash -r us-east-1 -t credential-store delete test-key
Deleting test-key -- version 0000000000000000000
Deleting test-key -- version 0000000000000000001
Using credstash directly is extremely simple, but setting everything up for it to work on EC2 instances can be a bit daunting, so this is what this section and the credstash-grant terraform module are about.
The simplest example that comes to mind—which is actually pretty common in practice—is deploying an EC2 instance with an nginx webserver serving a web page (or working as a reverse proxy), while protecting it with BasicAuthentication. We will use credstash to automatically retrieve credentials that we will store prior to EC2 instance deployment:
$ $(terraform output put_cmd) nginx-username admin
nginx-username has been stored
$ $(terraform output put_cmd) nginx-password foobar
nginx-password has been stored
The full example can be found in this gist, but here are the parts that are of most interest to us.
Using the credstash-grant
module will effectively
allow read access to the DynamoDB table by attaching that policy to
an IAM role and creating a KMS grant, thus allowing that IAM role
to use the KMS Master key for decryption. This grant will
automatically be revoked upon destruction, so there is no need to
worry about some dangling settings that should be cleaned up.
# lookup credstash remote state
data "terraform_remote_state" "credstash" {
backend = "s3"
config {
region = "us-west-1"
bucket = "remote-tfstate"
key = "credstash/terraform.tfstate"
}
}
module "credstash-grant" {
source = "github.com/fpco/fpco-terraform-aws/tf-modules/credstash-grant"
kms_key_arn = "${data.terraform_remote_state.credstash.kms_key_arn}"
reader_policy_arn = "${data.terraform_remote_state.credstash.reader_policy_arn}"
roles_count = 1
roles_arns = ["${aws_iam_role.credstash-role.arn}"]
roles_names = ["${aws_iam_role.credstash-role.name}"]
}
You might notice that we created a writer policy during
credstash-setup stage, but didn’t supply its ARN to the module.
This ensures that we give EC2 instance read only access to the
secret store. If ability to store secrets from within EC2 is
desired, supplying writer_policy_arn
to the module is
all that is necessary for that to work.
This is the part where credstash is called on the EC2 side:
resource "aws_instance" "webserver" {
...
associate_public_ip_address = true
iam_instance_profile = "${aws_iam_instance_profile.credstash-profile.id}"
user_data = <<USER_DATA
#!/bin/bash
${data.terraform_remote_state.credstash.install_snippet}
apt-get install -y nginx
BASIC_AUTH_USERNAME="$(${data.terraform_remote_state.credstash.get_cmd} nginx-username)"
BASIC_AUTH_PASSWORD="$(${data.terraform_remote_state.credstash.get_cmd} nginx-password)"
echo -n "$BASIC_AUTH_USERNAME:" > /etc/nginx/.htpasswd
openssl passwd -apr1 "$BASIC_AUTH_PASSWORD" >> /etc/nginx/.htpasswd
...
You are not required to use the helper snippets if you don’t
want to, but those can be very helpful in the long run, especially
if some time later you choose to customize the KMS key name,
DynamoDB table or simply try to use credstash in another AWS
region. The get_cmd
and put_cmd
snippets
encapsulate this information, so we won’t have to chase all the
places were we used credstash in order to update these values.
Applying terraform will deploy our webserver. After it gets fully initialized we can verify that it worked as expected:
$ curl -s https://$(terraform output instance_ip) | grep title
<head><title>401 Authorization Required</title></head>
$ curl -s https://admin:foobar@$(terraform output instance_ip) | grep title
<title>Welcome to nginx!</title>
In a setup where we are using credstash only on one EC2 instance we have nothing else to worry about. Nowadays though, that is not such a common scenario. You might have a database cluster running on a few instances, a webserver that is managed by an auto scaling group, a message broker running on other instances, so on and so forth. Each one of those services requires its own set of credentials, TLS certificates or what have you. In these kinds of scenarios we need to make sure that instances running a webserver will not have access to secrets that are meant only for instances running the database. To complicate it even more, we often have stage, test, dev, and prod environments, and we would ideally like to isolate those from each other as much as possible. KMS Encryption Context is a straightforward solution to this problem.
By itself, context doesn’t give any extra level of protection, but when combined with constraints, which are specified during KMS grant creation, it turns into a powerful protection and isolation concept that significantly increases overall security.
Here is how encryption contexts work in a nutshell:
Whenever you run credstash put name secret foo=bar
,
the key value pair {"foo": "bar"}
, called context,
becomes cryptographically bound to the cyphertext that is stored in
the database, and therefore the same key value pair will be
required in order to decrypt the secret. Keep in mind that this
pair does not have to be something complicated, and in fact, it
must not contain any sensitive information, as it will be visible
in CloudTrail logs.
During grant creation, which is performed for us by the
credstash-grant
module, we can supply
reader_context
and writer_context
, which
will prevent credstash from running get
and
put
commands respectively without passing exactly the
same context as extra arguments. If I create a grant with reader
context env=test service=database
, there is no way for
an instance with that IAM role to read secrets that were encrypted
with env=prod service=database
or env=test
service=webserver
contexts, or no context at all for that
matter. It has to match exactly.
When analyzing security it is important to look at such cases when the system does get compromised. In a case when an attacker acquires root access, then all bets are off and secrets can be easily extracted from memory or file storage, but if privilege escalation did not occur then one of the possible concerns could be the fact that access even to those credentials stored with specific context could be accessed long after the deployment was complete. In a simple single machine setup this situation can be alleviated by revoking the KMS grant after initialization was complete, thus preventing long term access to KMS Master key. But if an EC2 instance is deployed as part of an Auto Scaling Group, that approach will not work, as access to secrets is needed at all times since EC2 instances can come and go at anytime.
As a side note. There is a way to control access to KMS keys through an IAM policy, just as it is done with DynamoDB table, but because encryption context constraints are not available at the policy level, we resort to explicit grant creation, precisely for the reason of isolation described in this section.
Besides the obvious use case of passing credentials to EC2
instances, credstash can be a potential solution in other areas.
The practical size limit for the data being encrypted and stored
with credstash is on the order of ~100KiB, so you can store things
much larger than a short passphrases, making it perfect for storing
things like TLS certificates and SSH keys. For example we were able
to successfully supply all necessary TLS certificates used for
mutual authentication in Elastic’s Beats protocol during deployment
of Logstash, while automatically generating certificates using
certstrap
.
There is no reason to think that information stored with credstash has to be sensitive in the first place. Usual configuration files can be stored and retrieved just as well, therefore taking this heavy burden away from the initialization script. In fact, this idea could be taken up a notch, and a credstash pull command could be setup in crontab. This way an application that is running on a server can be configured to periodically reload its configuration, thus giving an ability to administrator to update the configuration dynamically without use of any other provisioning tools.
AWS Lambda also uses IAM roles, and credstash being an open source python tool could be in theory used there as well.
It is so easy to use credstash that I actually started to manage my own credentials with it.
A couple of examples for some of the use cases, plus more documentation on credstash related terraform modules, can be found in the fpco/fpco-terraform-aws repository.
The overall benefits of using credstash should be pretty obvious at this point. Sensitive information is encrypted, stored securely in DynamoDB, and is available at all times. Furthermore, we have the ability to fine tune which part of it can be accessed and by which parties through usage of IAM roles and KMS grants. There are no more worries about manually encrypting your secrets and finding the best way to move them across the wire. You are no longer limited by a tiny 14KiB size limit of user-data. There is no more need for setting up SSH connections just to pass over those couple TLS certificates. By keeping credentials in a central remote location you are less likely to forget to remove them before pushing the code to your repository, or leaving them in unencrypted form in terraform state. More importantly, it gives you a unified, programatic way to manage credentials, bringing more structure and order to your DevOps.
Subscribe to our blog via email
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.