Improbable Icon
  • SpatialOS
  • Use cases
  • Community
  • Pricing
  • Documentation
  • Careers
7504fb72-black-and-white-dark-keys-792031-1500x770

Breaking into our own Vault of Secrets

19 March 2019

Keeley Erhardt is a Software Engineer at Improbable, working on SpatialOS. She previously worked as a Graduate Research Assistant at MIT, where she developed a blockchain-based framework for verifiably safe data exchange.

Like most companies, Improbable has many secrets; certificates, credentials for databases, API keys for external services, credentials for service-oriented architecture communication and more; and many, many ways of securing these secrets.

We manage each type of secret differently; however, all of our certificates are generated, secured, stored, and accessed using Hashicorp Vault. Vault is an open-source tool that provides an interface to secrets stored in persistent storage; it supports a large number of storage backends, including etcdand Google Cloud Storage (GCS), among others.

1fba0a82-chain-door-lock-164425

The Problem: locking our keys in the vault

When we started using Vault, we were using etcd. However, we recently decided to migrate our Vault storage backend from etcd to GCS, and to upgrade from Vault v0.6.5 to v0.9.1. Unfortunately, as we belatedly discovered, the open-source version of Vault does not support the migration of storage backends.

The number of secrets we had stored in Vault was relatively small, consisting only of certificates. However, Vault offers no built-in mechanism for exporting the bits of critical information that we needed to migrate. Crucially, we use Vault’s PKI secrets engine to generate dynamic X.509 certificates, and the API provides no way to export the private keys associated with these certificates.

Essentially, we needed to move our Vault, but some of the keys required to enable this migration are protected by Vault itself and cannot be revealed. After some brainstorming, we devised a cunning plan that would extract the necessary data from our Vault, unblock the migration and enable the upgrade of our Vault.

First, we needed access to the Vault unseal keys. Vault starts in a sealed state, which means that the encryption key needed to read and write from the storage backend is not yet known. To unseal the vault, we had to exchange a master key for an encryption key. So we just needed to access our master key.

To mitigate the risk of a malicious actor gaining access to the master key and using it to decrypt the entire Vault, the master key is split into multiple shares using an implementation of Shamir’s Secret Sharing technique. Only a subset of these shares is needed to reconstruct the master key. These shares are revealed when Vault is first initialised and should then be securely stored. Given that we still had access to these shares, we could reconstruct our master key.

63bca986-arches-architecture-building-60611
Copying the Vault Next, we needed a copy of the Vault storage backend holding our secrets. Vault storage backends are responsible for the durable storage of encrypted data. The Vault we wanted to migrate was using the etcd storage backend, used to persist Vault’s data in etcd. Copying the Vault storage backend required (1) exec-ing into the Kubernetes pod running the etcd storage backend, (2) snapshotting the keyspace from a running etcd member, (3) copying the snapshot from the pod, (4) restoring the snapshot to a local, temporary, 1-member etcd cluster, and (5) starting the local etcd cluster. This gave us the copy we needed.
$ # Exec into GKE node with Vault
$ kubectl exec -it $ETCD_NODE_NAME -- /bin/sh
/# etcdctl --version                                                                                  
etcdctl version: 3.3.2
API version: 2
$ # Snapshot etcd keyspace
/# ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshot.db
$ # Copy snapshot from GKE node to local machine
$ kubectl cp $ETCD_NODE_NAME:snapshot.db /tmp/etcd_backup
$ # Restore backup to a local etcd cluster
$ ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd_backup/snapshot.db \
--name m1 \
--initial-cluster m1=http://localhost:2380 \
--initial-cluster-token etcd-cluster-1 \
--initial-advertise-peer-urls http://localhost:2380
$  etcd --version                                                                                      
etcd Version: 3.3.2
Git SHA: GitNotFound
Go Version: go1.10
Go OS/Arch: darwin/amd64
$ # Start local etcd cluster
$ etcd \
--name m1 \
--listen-client-urls http://localhost:2379 \
--advertise-client-urls http://localhost:2379 \
--listen-peer-urls http://localhost:2380
7dda1065-bridge-railing-colorful-love-56877

Building more barriers to overcome

Now that we had a copy of the Vault storage backend running on a local etcd node, we had to get into it. However, the data in the backend is (of course) encrypted. We needed a mechanism for extracting and decrypting arbitrary key-value pairs from the backend, in particular, the key-value pairs that Vault restricts access to even when provided with the unseal key. Hashicorp’s Vault implementation does not permit users to export the private keys associated with CA certificates, so we needed to devise a mechanism for circumventing this restriction.

To get around the restriction, we wrote a modified Vault frontend to run on top of our encrypted etcd backend. The frontend consists of a custom Vault barrier, unsealed using the master key recreated from the initial shares mentioned above, that permits the extraction of arbitrary key-value pairs – an operation not permitted by the standard Vault API.

First, we configured a Vault frontend to run on top of use the local etcd node holding our encrypted secrets. We configured the Vault frontend to match the configuration of the old Vault instance.


import "github.com/hashicorp/vault/physical"

// The backend config must match the backend of the Vault instance from which data is being migrated.
b, err := physical.NewBackend("etcd", nil, map[string]string{
  "address":    "http://127.0.0.1:2379",
  "ha_enabled": "true",
  "etcd_api":   "v3",
})
if err != nil {
  return nil, errors.Wrapf(err, "failed to initialize new etcd backend")
}

Now that we had an unlocked barrier, we could extract any key in plain text! We first listed the paths to all of the keys stored in our Vault to find the ones we cared about.

$ ETCDCTL_API=3 etcdctl get --prefix “/” --keys-only

Next, we passed the paths to the keys we cared about to our script to individually extract each key-value pair.

var (
    fOriginVaultKeysPaths = flag.String(
     "origin_vault_keys_paths",
     "",
     "Paths to the physical locations of the storage keys in the encrypted backend to migrate",
  )
  fOriginVaultBackendName = flag.String(
     "origin_vault_backend_name",
     "",
     "Name of the backend in the origin Vault in which the data to migrate is stored",
  )
)

secrets := map[string][]byte{}
paths := strings.Split(*fOriginVaultKeysPaths, " ")
if len(paths) == 0 {
  return nil, errors.New("no paths to Vault keys specified")
}
for _, p := range paths {
  backendName := normalizeBackendName(*fOriginVaultBackendName)
  path := strings.TrimPrefix(p, "/")
  ent, err := barrier.Get(backendName + path)
  if err != nil {
     return nil, errors.Wrapf(err, "failed to get value for key at: %v", p)
  }
  if ent == nil {
     return nil, errors.New("no entry found for key")
  }
  secrets[strings.TrimPrefix(ent.Key, backendName)] = ent.Value
}

Finally, we migrated each of these extracted key-value pairs to our new GCS-backed Vault, completing the migration!

var (
  fDestinationVaultAddress = flag.String(
     "destination_vault_address",
     "",
     "The address of the Vault to migrate data to",
  )
  fDestinationVaultBackendName = flag.String(
     "destination_vault_backend_name",
     "",
     "Name of the backend in the destination Vault in which to place migrated data, e.g. secret/",
  )
)

func migrate(secrets map[string][]byte) error {
  if *fDestinationVaultAddress == "" {
     return errors.New("no destination Vault address set")
  }
  cl, err := api.NewClient(&api.Config{
     Address: *fDestinationVaultAddress,
  })
  if err != nil {
     return err
  }
  if *fDestinationVaultToken == "" {
     return errors.New("no destination Vault token set")
  }
  cl.SetToken(*fDestinationVaultToken)

  for k, v := range secrets {
     var secret map[string]interface{}
     if err := json.Unmarshal(v, &secret); err != nil {
        return err
     }
     backendName := normalizeBackendName(*fDestinationVaultBackendName)
     if _, err := cl.Logical().Write(backendName + k, secret); err != nil {
        return err
     }
     fmt.Printf("wrote key %v\n", k)
  }

  return nil
}
b12d146a-book-chain-computer-39584

Mission Accomplished

Running our modified Vault frontend enabled us to extract the private keys we needed to upgrade and move Vault. Perhaps in the future migrating between storage backends will be supported in the open-source version of Hashicorp Vault. In the meantime, we were forced to use a hacky but effective workaround – if you want to try it out, you can find the complete modified Vault frontend on our Github.

(This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, you can obtain one at http://mozilla.org/MPL/2.0/.)