Codify Vault Enterprise management with Terraform
Personas
The scenario described in this tutorial introduces the following personas:
admin
is the organization-level administratorstudent
is a user allowed to write data to a path in vault
Challenge
A manual system administration can become a challenge as the scale of infrastructure increases. Often, an organization must manage multiple Vault environments (development, testing, staging, production, etc.). Keeping up with the increasing management demand soon becomes a challenge without some sort of automation.
Solution
One of the pillars behind the Tao of Hashicorp is automation through codification.
HashiCorp Terraform is an infrastructure as code which enables the operation team to codify the Vault configuration tasks such as the creation of policies. Automation through codification allows operators to increase their productivity, move quicker, promote repeatable processes, and reduce human error.
This tutorial demonstrates techniques for creating Vault policies and configurations using Terraform Vault Provider.
Prerequisites
- Terraform installed
- Vault Enterprise 1.4 or later
Enterprise Only
This tutorial creates namespaces which require Vault Enterprise Standard license.
If you are running open-source Vault, see the Codify Management of Vault Using Terraform tutorial.
Scenario introduction
Vault administrators must manage multiple Vault environments. The test servers get destroyed at the end of each test cycle and a new set of servers must be provisioned for the next test cycle. To automate the Vault server configuration, you are going to use Terraform to provision the following Vault resources.
Type | Name | Description |
---|---|---|
namespace | finance | A namespace dedicated to the finance organization |
namespace | engineering | A namespace dedicated to the engineering organization |
namespace | education | A namespace dedicated to the education organization |
namespace | training | A child-namespace under education dedicated to the training team |
namespace | vault_cloud | A child-namespace under education/training dedicated to the vault_cloud team |
namespace | engineering | A child-namespace under education/training dedicated to the boundary team |
ACL Policy | admins | Sets policies for the admin team |
ACL Policy | fpe-client | Sets policies for clients to encode/decode data through transform secrets engine |
auth method | userpass | Enable and create a user, "student" with admins and fpe-client policies |
auth method | approle | Enable approle auth method in the education/training and create a test-role role |
secrets engine | kv-v2 | Enable kv-v2 secrets engine in the finance namespace |
secrets engine | transform | Enable transform secrets engine at transform |
transformation | ccn-fpe | Transformation to perform format preserving encryption (FPE) transformation on credit card numbers |
transformation template | ccn | Define the data format structure for credit card numbers |
alphabet | numerics | Set of allowed characters |
The admins
policy must be created in all namespaces: root
, finance
, and
engineering
. The expected admin tasks are the same across the namespaces.
Note
Transform secrets engine requires Vault Enterprise Advanced Data Protection (ADP) license.
The following steps are demonstrated:
Examine the Terraform files
Clone or download the demo assets from the hashicorp/learn-vault-codify GitHub repository to perform the steps described in this tutorial.
$ git clone https://github.com/hashicorp/learn-vault-codify.git
Change the working directory to
learn-vault-codify/enterprise
.$ cd learn-vault-codify/enterprise
The directory contains Terraform files to configure Vault.
$ tree . ├── auth.tf ├── main.tf ├── policies │  ├── admin-policy.hcl │  └── fpe-client-policy.hcl ├── policies.tf └── secrets.tf
Review main.tf
Open the main.tf
file in your preferred text editor to examine its content.
main.tf
#------------------------------------------------------------------------------
# The best practice is to use remote state file and encrypt it since your
# state files may contains sensitive data (secrets).
#------------------------------------------------------------------------------
# terraform {
# backend "s3" {
# bucket = "remote-terraform-state-dev"
# encrypt = true
# key = "terraform.tfstate"
# region = "us-east-1"
# }
# }
#-----------------------------------------------------------------------------------
# To configure Transform secrets engine, you need vault provider v2.12.0 or later
#-----------------------------------------------------------------------------------
terraform {
required_providers {
vault = "~> 3.7.0"
}
}
#------------------------------------------------------------------------------
# To leverage more than one namespace, define a vault provider per namespace
#------------------------------------------------------------------------------
provider "vault" {}
#------------------------------------------------------------------------------
# Create namespaces: finance, and engineering
#------------------------------------------------------------------------------
resource "vault_namespace" "finance" {
path = "finance"
}
resource "vault_namespace" "engineering" {
path = "engineering"
}
#---------------------------------------------------------------
# Create nested namespaces
# education has childnamespace, 'training'
# training has childnamespace, 'secure'
# secure has childnamespace, 'vault_cloud' and 'boundary'
#---------------------------------------------------------------
resource "vault_namespace" "education" {
path = "education"
}
# Create a childnamespace, 'training' under 'education'
resource "vault_namespace" "training" {
namespace = vault_namespace.education.path
path = "training"
}
# Create a childnamespace, 'vault_cloud' and 'boundary' under 'education/training'
resource "vault_namespace" "vault_cloud" {
namespace = vault_namespace.training.path_fq
path = "vault_cloud"
}
# Create 'education/training/boundary' namespace
resource "vault_namespace" "boundary" {
namespace = vault_namespace.training.path_fq
path = "boundary"
}
To create a education/training
namespace, use the namespace
parameter to
point the vault_namespace.education.path
parent namespace.
resource "vault_namespace" "training" {
namespace = vault_namespace.education.path
path = "training"
}
It is strongly recommended to specify the target server specific
information
using environment variables (e.g. VAULT_ADDR
, VAULT_TOKEN
); that is
what you are going to do in this tutorial.
Note
If you are not familiar with Vault Enterprise namespace, refer to the Secure Multi-Tenancy with Namespaces tutorial.
Review policies.tf
Open the policies.tf
file and examine the
vault_policy
resources. It uses the provider
parameter to specify the target namespace to
create the policies.
policies.tf
#---------------------
# Create policies
#---------------------
# Create fpe-client policy in the root namespace
resource "vault_policy" "fpe_client_policy" {
name = "fpe-client"
policy = file("policies/fpe-client-policy.hcl")
}
# Create admin policy in the root namespace
resource "vault_policy" "admin_policy" {
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the finance namespace
resource "vault_policy" "admin_policy_finance" {
namespace = vault_namespace.finance.path
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the engineering namespace
resource "vault_policy" "admin_policy_engineering" {
namespace = vault_namespace.engineering.path
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the education namespace
resource "vault_policy" "admin_policy_education" {
namespace = vault_namespace.education.path
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the 'education/training' namespace
resource "vault_policy" "admin_policy_training" {
namespace = vault_namespace.training.path_fq
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the 'education/training/vault_cloud' namespace
resource "vault_policy" "admin_policy_vault_cloud" {
namespace = vault_namespace.vault_cloud.path_fq
name = "admins"
policy = file("policies/admin-policy.hcl")
}
# Create admin policy in the 'education/training/boundary' namespace
resource "vault_policy" "admin_policy_boundary" {
namespace = vault_namespace.boundary.path_fq
name = "admins"
policy = file("policies/admin-policy.hcl")
}
Review auth.tf
Open the auth.tf
file.
- Line 4 through 6 enables
userpass
auth method. - Line 9 through 20 creates a user, "student" with
admins
andfpe-client
policies attached. The password is set to "changeme". - Line 25 through 29 enables
approle
auth method. - Line 32 through 38 creates a
test-role
role.
auth.tf
#--------------------------------
# Enable userpass auth method
#--------------------------------
resource "vault_auth_backend" "userpass" {
type = "userpass"
}
# Create a user named, "student"
resource "vault_generic_endpoint" "student" {
depends_on = [vault_auth_backend.userpass]
path = "auth/userpass/users/student"
ignore_absent_fields = true
data_json = <<EOT
{
"policies": ["fpe-client", "admins"],
"password": "changeme"
}
EOT
}
#--------------------------------------------------------------------
# Enable approle auth method in the 'education/training' namespace
#--------------------------------------------------------------------
resource "vault_auth_backend" "approle" {
depends_on = [vault_namespace.training]
namespace = vault_namespace.training.path_fq
type = "approle"
}
# Create a role named, "test-role"
resource "vault_approle_auth_backend_role" "test-role" {
depends_on = [vault_auth_backend.approle]
backend = vault_auth_backend.approle.path
namespace = vault_namespace.training.path_fq
role_name = "test-role"
token_policies = ["default", "admins"]
}
Review secrets.tf
Open the secrets.tf
file.
- Line 5 through 10 enables kv-v2 secretes engine in the
finance
namespace. - Line 19 through 53 defines a new alphabet, template, transformation, and a role.
- Line 59 through 69 tests the FPE transformation configured.
secrets.tf
#----------------------------------------------------------
# Enable secrets engines
#----------------------------------------------------------
# Enable kv-v2 secrets engine in the finance namespace
resource "vault_mount" "kv-v2" {
# depends_on = [vault_namespace.finance]
# provider = vault.finance
namespace = vault_namespace.finance.path
path = "kv-v2"
type = "kv-v2"
}
# Transform secrets engine at root
resource "vault_mount" "mount_transform" {
path = "transform"
type = "transform"
}
# Create an alphabet
resource "vault_transform_alphabet" "numerics" {
path = vault_mount.mount_transform.path
name = "numerics"
alphabet = "0123456789"
}
# Create a transformation template
resource "vault_transform_template" "ccn" {
path = vault_mount.mount_transform.path
name = "ccn"
type = "regex"
pattern = "(\\d{4})-(\\d{4})-(\\d{4})-(\\d{4})"
alphabet = vault_transform_alphabet.numerics.name
}
# Create a transformation named ccn-fpe
resource "vault_transform_transformation" "ccn-fpe" {
path = vault_mount.mount_transform.path
name = "ccn-fpe"
type = "fpe"
template = vault_transform_template.ccn.name
tweak_source = "internal"
# payments here listed by name and not reference to avoid circular dependency.
# Vault does not require dependencies like these to exist prior to creating
# other things that reference them, but they may simply not work until they do
# exist.
allowed_roles = ["payments"]
}
# Create a role named 'payments'
resource "vault_transform_role" "payments" {
path = vault_mount.mount_transform.path
name = "payments"
transformations = [vault_transform_transformation.ccn-fpe.name]
}
#-------------------------------------------------------------------
# Test the transformation
#-------------------------------------------------------------------
data "vault_transform_encode" "encoded" {
path = vault_transform_role.payments.path
role_name = "payments"
value = "1111-2222-3333-4444"
depends_on = [vault_transform_role.payments]
}
output "encoded" {
value = data.vault_transform_encode.encoded.encoded_value
}
Note
The details about the transformation, template, alphabet, and role are out of scope for this tutorial. If you are not familiar with Transform secrets engine, read the Transform Secrets Engine tutorial.
Run Terraform to configure Vault
Optional: Start a Vault server in development mode with
root
as the root token if you don't have one running already.$ vault server -dev -dev-root-token-id root
Set the client token in the
VAULT_TOKEN
environment variable.$ export VAULT_TOKEN="root"
If the token is different, be sure to set it to the correct token value that has permissions to create policies, enable secrets engines, and enable auth methods.
Set the target Vault server address in the
VAULT_ADDR
environment variable if it's not done so already.$ export VAULT_ADDR="http://127.0.0.1:8200"
If you are connecting to a remote Vault server, be sure to set the
VAULT_ADDR
value to the correct target Vault API address.Initialize Terraform to pull Vault provider plugin.
$ terraform init
This downloads the Vault plugin. When it completes, it displays a message,
Terraform has been successfully initialized!
Execute the
apply
command to configure Vault.$ terraform apply
This displays the actions to be performed by Terraform.
When prompted, enter
yes
to accept the plan and proceed with Vault configuration.Plan: 24 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes
Once completed, the output similar to the following displays.
Apply complete! Resources: 24 added, 0 changed, 0 destroyed. Outputs: encoded = "4079-1004-8601-9194"
Verify the configuration
List the existing namespaces.
$ vault namespace list Keys ---- education/ engineering/ finance/
List the nested namespaces.
$ vault namespace list -namespace=education Keys ---- training/
List namespaces under
training
.$ vault namespace list -namespace=education/training Keys ---- boundary/ vault_cloud/
Verify that policies were created.
$ vault policy list admins default fpe-client root
Verify that
admins
policy was created under thefinance
namespace.$ vault policy list -namespace=finance admins default
Similarly, verify that
admins
policy was created under theengineering
namespace.$ vault policy list -namespace=engineering admins default
Verify that
admins
policy was created under theeducation
namespace.$ vault policy list -namespace=education admins default
Verify that
admins
policy was created under theeducation/training
namespace.$ vault policy list -namespace=education/training admins default
Verify that
admins
policy was created under theeducation/training/vault_cloud
namespace.$ vault policy list -namespace=education/training/vault_cloud admins default
Verify that
admins
policy was created under theeducation/training/boundary
namespace.$ vault policy list -namespace=education/training/boundary admins default
Verify that kv-v2 secrets engine is enabled in the
finance
namespace.$ vault secrets list -namespace=finance Path Type Accessor Description ---- ---- -------- ----------- cubbyhole/ ns_cubbyhole ns_cubbyhole_f615f334 per-token private secret storage identity/ ns_identity ns_identity_1ae45907 identity store kv-v2/ kv kv_abd3188e n/a sys/ ns_system ns_system_e5b43cff system endpoints used for control, policy and debugging
Verify the transformation secrets engine configuration for credit card numbers.
List existing transformations.
$ vault list transform/transformation Keys ---- ccn-fpe
Read the
ccn-fpe
transformation details.$ vault read transform/transformation/ccn-fpe Key Value --- ----- allowed_roles [payments] templates [ccn] tweak_source internal type fpe
List existing transformation templates.
$ vault list transform/template Keys ---- builtin/creditcardnumber builtin/socialsecuritynumber ccn
Read the
ccn
transformation template definition.$ vault read transform/template/ccn Key Value --- ----- alphabet numerics decode_formats map[] encode_format n/a pattern (\d{4})-(\d{4})-(\d{4})-(\d{4}) type regex
Now, verify that you can log in with
userpass
auth method using the username, "student" and password, "changeme".$ vault login -method=userpass username=student Password (will be hidden): Key Value --- ----- token s.5MadPX3UQgBRuukSHjf9yPJd token_accessor AKvdeYEJspzN4fdOq0OcSSps token_duration 768h token_renewable true token_policies ["admins" "default" "fpe-client"] identity_policies [] policies ["admins" "default" "fpe-client"] token_meta_username student
The generated token has
admins
andfpe-client
policies attached. Now, take a look at thefpe-client
policy definition.$ cat policies/fpe-client-policy.hcl
The
fpe-client
policy permits update operation against thetransform/encode/*
andtransform/decode/*
paths.# To request data encoding using any of the roles # Specify the role name in the path to narrow down the scope path "transform/encode/*" { capabilities = [ "update" ] } # To request data decoding using any of the roles # Specify the role name in the path to narrow down the scope path "transform/decode/*" { capabilities = [ "update" ] }
The student user should be able to perform format-preserving encryption (FPE) transformation.
$ vault write transform/encode/payments value=1111-2222-3333-4444 Key Value --- ----- encoded_value 6754-4542-1216-8423
Verify that you can decode the encoded value.
$ vault write transform/decode/payments value="6754-4542-1216-8423" Key Value --- ----- decoded_value 1111-2222-3333-4444
Successfully returns the original credit card number.
Verify that
approle
auth method is enabled in theeducation/training
namespace.$ vault auth list -namespace=education/training Path Type Accessor Description ---- ---- -------- ----------- approle/ approle auth_approle_077406b6 n/a token/ ns_token auth_ns_token_99292818 token based credentials
Verify that
test-role
exists.$ vault list -namespace=education/training auth/approle/role Keys ---- test-role
Clean up
When you are done exploring, you can undo the configuration made by Terraform.
Make sure that
VAULT_TOKEN
andVAULT_ADDR
environment variables are set.$ echo $VAULT_ADDR; echo $VAULT_TOKEN
Destroy the Vault resources created by Terraform.
$ terraform destroy -auto-approve ...snip... Destroy complete! Resources: 24 destroyed.
Remove the terraform state files.
$ rm *.tfstate.*
Unset the
VAULT_TOKEN
andVAULT_ADDR
environment variables.$ unset VAULT_TOKEN VAULT_ADDR
Note
To learn more about Terraform, visit Learn Terraform.
Next steps
Treat your Terraform files like any other code and manage them through a version control system such as GitHub. You may integrate it with your favorite CI/CD tool (Jenkins, Travis CI, Circle CI, etc.), always review and test the configuration.
Travis CI example:
You can test your Terraform files against a development server that runs locally, or use a Docker image of Vault.
sudo: false
env:
- VAULT_ADDR=<target_vault_address> VAULT_TOKEN=<token>
before_install:
- scripts/install.sh
before_script:
- vault server -dev -dev-root-token-id="$VAULT_TOKEN"
script:
- terraform init
- terraform apply -auto-approve
Summary
In this guide you learned a technique for creating Vault policies and configurations using the Terraform Vault Provider. For more information, see the help and reference section.
Help and Reference
- Terraform Vault Provider documentation page
- Terraform Provider GitHub repository
- Learn Terraform
- Multi-tenancy with Namespaces
Tip
Terraform users can leverage the Vault's dynamic secrets engine to generate short-live cloud credentials when provisioning cloud resources. Inject secrets into Terraform using the Vault provider tutorial demonstrates the use of AWS secrets engine to manage AWS IAM credentials used by Terraform.