My private git repos are a mess when it comes to secrets. I’ve wanted to implement vault for a while now. It will allow me to start removing the secrets from plain text files in git. Vault allows me to store everything in a central location and in the future even dynamically generate secrets.

In this one I’ll go over how I set up Vault Secrets Operator (VSO) to sync Vault secrets to Kubernetes.

The Kubernetes side of things

I had to deploy VSO and a bunch of resources to configure everything and make things talk.

1. Add the Helm repository and install VSO:

   helm repo add hashicorp https://helm.releases.hashicorp.com
   helm install --create-namespace --namespace vault-secrets-operator vault-secrets-operator hashicorp/vault-secrets-operator
   

2. Deploy VaultAuth:

The VaultAuth is what tells VSO which method and service account to use to authenticate to vault.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: vault-secrets-operator
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces: [default,vault-secrets-operator]
  kubernetes:
    role: vault-role
    serviceAccount: vault-auth

3. Create a secret containing the CA of the Kubernetes API:

   apiVersion: v1
   kind: Secret
   metadata:
     name: vault-ca-cert
     namespace: vault-secrets-operator
   type: Opaque
   data:
     ca.crt: CERT_HERE
   

4. Configure VaultConnection:

The VaultConnection tells VSO how to connect to Vault

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: vault-secrets-operator
spec:
  address: "https://192.168.20.104:8200"
  skipTLSVerify: true
  caCertSecretRef: "vault-ca-cert"

5. Deploy necessary service account, roles, and bindings:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: vault-auth
     namespace: vault-secrets-operator
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: vault-token-reviewer
   rules:
   - apiGroups: [""]
     resources: ["serviceaccounts", "serviceaccounts/token", "secrets", "pods"]
     verbs: ["create", "get", "list", "watch"]
   - apiGroups: ["authentication.k8s.io"]
     resources: ["tokenreviews"]
     verbs: ["create"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: vault-token-reviewer-binding
   subjects:
   - kind: ServiceAccount
     name: vault-auth
     namespace: vault-secrets-operator
   roleRef:
     kind: ClusterRole
     name: vault-token-reviewer
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: vault-auth-secret
     annotations:
       kubernetes.io/service-account.name: vault-auth
       kubernetes.io/service-account.namespace: default
   type: kubernetes.io/service-account-token
   immutable: true
   

6. Create the token:

I’m sure there is a better way to do this, but for now this is fine.

kubectl create token vault-auth -n default --duration=8760h

The Vault side of things

Now that Kubernetes is configured, I moved onto the Vault stuff.

1. Enable Kubernetes auth method:

   vault auth enable kubernetes
   

2. Configure the auth method:

The service account token below is what I created in step 6 above.

vault write auth/kubernetes/config \
  kubernetes_host=https://<K8S_API_URL> \
  kubernetes_ca_cert=@<path_to_ca_cert> \
  token_reviewer_jwt=<SERVICE_ACCOUNT_TOKEN>

3. Create a role:

   vault write auth/kubernetes/role/homelab-api-role \
     bound_service_account_names=vault-auth \
     bound_service_account_namespaces=default \
     policies=homelab-api-policy \
     ttl=24h
   

4. Create a policy for the role:

   path "kv/data/test1" {
     capabilities = ["read"]
   }
   

   vault policy write homelab-api-policy /path/to/policy.hcl
   

Create the Vault Secret resource

The last thing to do is to create a VaultSecret and watch it sync! Here’s an example of my Sensu API key.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-kv-sensu-api-key
  namespace: default
spec:
  vaultAuthRef: vault-secrets-operator/vault-auth
  mount: kv
  path: sensu-api-key
  destination:
    name: vault-kv-sensu-api-key
    create: true
  refreshAfter: 1h
  type: kv-v2

After deploying this resource a quick kubectl get secrets shows a new secret called vault-kv-sensu-api-key that matches what it’s Vault!

The next step for me is to setup the database engine. This will allow Vault to create and delete database credentials. I hope I’ll be able to fully remove database credentials from my Kubernetes manifests once this is implemented.