What is Vault?
HashiCorp Vault is a software application that securely stores and manages digital secrets. These secrets can be a wide range of sensitive information such as passwords, tokens, API keys, etc.
Vault solves the secret sprawl problem where secrets are scattered throughout the environment in configuration files, scripts, or databases.
Vault brings all secrets into a single place — making auditing, key rotation, or revocation easier.
Vault provides these advanced functionalities for free, and even the free version is sufficient for most use cases. Large enterprise customers can elect to buy the enterprise version of Vault for more functionalities and scalability, such as multi-region replication.
This article goes over the POC I did for a project I worked on. We’ve since automated this deployment in Kubernetes, which offers auto unsealing of the vault.
Storage For Vault Data
Vault supports many storage providers to persist its encrypted data (e.g., ZooKeeper, MySQL, Postgres, Cassandra, Swift, etc.). Still, after reviewing several of them, we use Vault’s integrated Raft storage for the backend store.
The Raft integrated provides high availability and is supported by HashiCorp.
Using Raft offers the following benefits over using other external storage:
- Integrated into Vault
- All configurations within Vault
- Supports failover and multi-cluster replication (needs Enterprise)
- Eliminates additional network requests
- Performance gains (reduces disk read/write overhead)
- Reduces cost (fewer VMs)
- Eliminates the second point of failure of external storage service
- Lowers complexity when diagnosing issues
The Vault Must Be Unsealed To Use
Vault encrypts its data with multiple keys, and multiple keys are required to unseal the vault. The vault becomes automatically sealed after reboot.
For the POC, the unsealing process will be manual.
Different keys can be held by different individuals in the organization if required.
The Architecture for the POC
We will use three VMs to run the three Docker nodes in this setup. Note that you can set up with a single node if high availability isn’t needed.
Storage for raft will come from /store/vault/raft on the VMs.
This diagram depicts what this setup will look like:
Vault Cluster Architecture
Scaling
A minimum of three vault nodes is recommended for production use as this configuration allows for one node failure.
Vault’s design funnels all traffic to a single active node (for both reads and writes), either via redirection or forwarding. Because everything must go through one active node, having additional nodes does not help with scaling and will degrade write throughput.
Vault can only be scaled vertically and not horizontally, i.e., scale vault by giving vault more CPUs and RAM instead of adding more nodes.
Steps to Deploy the Vault Cluster
The following sections will deploy the Vault Cluster. Although it’s unnecessary to use TLS, it’s highly recommended to ensure that all communication is encrypted. TLS can also ensure that the client and server can validate each other via the central authority (aka the CA).
Getting the Vault Docker Container
HashiCorp provides the Vault Docker container
Vault - Official Image | Docker HubVault is a tool for securely accessing secrets via a unified interface and tight access control.
hub.docker.com
You can copy and paste this to pull this image:
Setting the CA and TLS Certificates
This section assumes you have DNS entries for vault1, vault2, and vault3 for the vault nodes, as it will perform DNS lookups based on that to work out IP addresses.
It also assumes the valid FQDN of the host is resolvable; it uses the reverse lookup of the above names to work out the FQDN.
Generate the CA (or use another one that you’ve created)
Generate the certificates for the three vault nodes (Though if you’re only doing one host for dev, you probably don’t want the IP.2 line from below as well as skipping the loop)
This creates the CA certs and certificates for the three vault nodes.
Create the Vault Config File
On the Vault VM, prepare the directory structure and create the config file that will setup Vault:
Start the Vault Docker Container
On the docker host for the vault node, copy the CA certificate (vault-server-ca.pem) to /etc/pki/ca-trust/source/anchors then run these commands:
Copy the appropriate cert.pem and key.pem for the vault node to /store/vault/config/ on the vault node.
Repeat these commands for the second and third nodes.
Initialize Vault
Vault on the follower nodes does not need to be initialized because they will join the primary cluster and get the data replicated to them.
Follow this step to initialize the Vault secret store.
The vault will be sealed after initialization. In the sealed state, the vault secret store is encrypted and unavailable.
Unsealing of the vault needs to be done with the Vault CLI. We will use the first vault node to initialize the vault:
Vault should output something similar to this:
The keys and tokens must be protected and should never get into the wrong hands.
Five keys can be used to unseal the vault. At least three keys will be required to unseal (each key could be held by different people if desired). The root token is the key we will use to authenticate to the vault for other setup steps.
The root token should not be used after the initial setup. Other admin accounts should be created for people to use.
Unsealing the Vault
If Vault is successfully unsealed, you should see this:
`Sealed` should be `false`.
Joining the Follower Nodes to the Primary Vault Node
This section requires that the vault container has been created and started.
By this time, the primary vault/raft cluster should be operational. This instruction joins the second and third nodes to the cluster.
Log in to the second vault node (e.g., KVM guest). Then run these commands: (NOTE that primary IP should be set to 10.0.100.
Check the status of the raft cluster.
Web UI Access
You can access the WebUI at https://<one-node-ip>:8200/ui
Use the root token to login for the first time.
Conclusion
This setup touches just the core of vault. Vault has additional advanced features such as dynamic credentials. It can also be set up as a central authority (CA). It can also be used for OTP (one-time-password) login using ssh.
It’s an excellent product for managing secrets.
