How to use Azure Key Vault as your Pulumi secrets provider to keep your secrets under your control - avoid having to create a Key Vault to begin with manually, all the while keeping your sanity - trust me!

The Origin Story

Out of the box, Pulumi encrypts secrets within each stack using a Pulumi Service hosted, per-stack encryption key.

This is, to put it simply: fantastic.

I love what Pulumi have done with the encryption in their stack system - the defaults implement a strong “correct path”, a way to consistently and properly encrypt secrets for use in deployments - and it takes zero effort to put in place.

Every single user of the Pulumi Service is already doing “the right thing” in terms of secret security.

These defaults remove a bunch of problems that as developers, we’d otherwise be solving ourselves, problems like: how & when to encrypt / decrypt, how to keep data safe in transit, and where to store the secrets centrally - I’m sure that list isn’t extensive, and while I’m not a security guy - I’m already postitive I don’t want to be responsible for getting that stuff 100% correct.

Developers can safely set secret, passwords and master keys within a Pulumi stack and check the stack configuration file into source control. Yes, you heard right - check that config into source control, your CI/CD will thank you for it!

This is what a sample stack with encrypted values look like - it’s just a text file (and yeah, you can have fun trying to connect to this non-existent tenant):

secretsprovider: azurekeyvault://fgh234j21-primary-kv.vault.azure.net/keys/pulumi
encryptedkey: rVo2bEpyN05SNmtpZjN4VHlDGs3ODJPY2sxY1ZfaFc0eVJ4dG...
config:
  azure-native:location: WestUS
  soxes-infra:targetSubscriptionId: c0d3206e-7617-4bce-b8af-24cd614fd0f5
  soxes-infra:targetTenantId: bea5f427-126c-2fd3-b48a-156d8a948c97

Requirements

While the Pulumi defaults are great to get started with - the project I’m working on requires control of the encryption keys (insert manic laughter, and definitely an evil control freak outlined in silhouette by thunder & lightning effects, if you really want, throw some rain in there too - for the mood).

Thankfully Pulumi can make use of a number of encryption providers , Azure Key Vault included! (think: harps, angelic light, white knights - that kind of thing).

Back on Earth - my client has (at least) the following requirements:

  • To have full and exclusive control of the encryption key(s) used for stack secrets.
  • In the case of a (stack) security breach at Pulumi the secrets for dev / staging and production services must remain encrypted.
  • Developers must be able to access the keys to create stacks, but they should not be able to alter or delete the keys - nobody wants that type of mistake to happen. Lets make it easy to succeed, and hard to fail.
  • Key rotation is handled in the background (another blog post!).
  • The CI/CD pipeline must be able to create more vaults & keys as and if required.
  • Each project / app will have its own key vault - no re-use of vaults nor keys across projects is allowed - this reduces the scope / blast radius should a vault or key be leaked somehow.

Wow. That’s a lot of stuff - can we do it?

‘Course we can!

Solution

The get down to the solution shall we (finally, I hear you say).

It’s a two project solution in order to avoid the situation that you cannot create a pulumi stack without a key vault; but you need to create a key vault (with pulumi ideally) in order to use pulumi.

The two projects are (drum roll please):

  1. A “prep” project - which will use Pulumi’s default secrets management. This project will create a key vault, and set up the required permissions such that others can be added to an AD group in order to read from the vault and use pulumi stack init –secrets-provider=“azurekeyvault://xxxxxxxxxxx”. This project requires admin priviledges to run.
  2. The second project is for whatever you really want to keep secret - this project will reference the key vault created during the “prep” stage as well as a second AD group.

We chose to isolate our work to a single subscription; the AD account used for the “prep” project requires:

  • Subscription: Contributor
  • Subscription: Key Vault Administrator
  • Subscription: User Access Administrator

The benefits of this approach are:

  1. The first project is the only one that requires administrative privileges within Azure and because Pulumi is doing the heavy lifting we retain the benefits of automation as well as ease of future maintenance. No secrets are exposed in the stack configuration that are not otherwise additionally protected by Azure AD credentials.
  2. All of the AD resources, rights, custom roles, AD groups, infrastructure components and so on are nicely wrapped up in a Pulumi project; so the development environments are going to take advantage of versioned IaC from day one.
  3. Because the environments are modified using Pulumi - there is an audit trail and transparency that you just don’t get with the Azure Portal.
  4. Minimal documentation / training is required compared with having the first “prep” project be a traditional hand built component (in the Azure Portal). We actually tried this once, and spent a couple of hours struggling to get it to work.
  5. The second project (in our case an AKS development cluster environment) doesn’t really need to know about the tenant / subscription configuration; that is available by referencing the “prep” project.

Prep Stage

The “prep” stage does the following (more or less):

  1. Creates a Key Vault.
  2. Creates a “KV Reader” and “Developer” AD Group.
  3. Exports some outputs (this is key).

Here’s a snippet of the outputs - there’s almost enough information here to make use of the secrets provider in azure key vault!

    [Output] public Output<string> TargetSubscriptionId { get; set; }
    [Output] public Output<string> TargetTenantId { get; set; }
    [Output] public Output<string> DeveloperAdGroupId { get; set; }
    [Output] public Output<string> KeyVaultAdGroupId { get; set; }
    [Output] public Output<string> CustomRoleAksDeleterId { get; set; }

But you might say: “Hang on there Johno, those SubscriptionID / TenantID values are secrets, no?”.

Yes - they could be (especially if you called pulumi config set with the –secret flag); and they are still protected by the per stack encryption of the Pulumi Service.

Take note however; there are no passwords, or other actual secrets of any kind being stored in the stack config - even if you did get hold of the data here, using the outputs still requires authentication via another AD account or service principal; that’s the critical part that allows us to feel safe and in control at this stage.

The Second Stage

The second project uses the Azure Key Vault constructed during the “prep” stage as a secrets provider, and as I mentioned above also references the obtain information about the subscription and AD groups.

While I’m not going into detail about what the second stage builds - because its irrelevant for this post - it critically now can store secrets of any kind, safe in the knowledge that these secrets are encrypted in the Pulumi Service using a Key provided by the Azure Key Vault.

All Goal(s) achieved!

Summary

If you’d like to know specifics - don’t hesitate to get into contact - as always my contact information is in the footer.