Modern IaC solution based on Pulumi – Part 2

As I mentioned in the opening post Modern IaC solution based on Pulumi – Part 1 we decided to self-manage the Pulumi backend. In this post I will describe what we did to accomplish this and related things.

Pulumi gives you multiple options when it comes to storing a copy of the current infrastructure state. For most cases the Pulumi Service backend is ideal. Just install the Pulumi CLI and you are pretty much all set. It will manage the state for you, including state sharing, state access synchronization, secret encryption and backup.

As we did chose not to use the Pulumi Backend service we had to store the state ourselves. We could either store it on our local machines, or using cloud storage. In our case Azure Blob Storage came as a natural choice. Here is a simple example of setting this up using Azure CLI:

# Login to Azure
az login

# Set default Subscription in case you have access to more than one
az account set --subscription $SUBSCRIPTION_ID

# Create Resource Group
az group create \
  --name $RESOURCE_GROUP_NAME \
  --location $LOCATION

# Create Storage Account, adjust SKU and kind etc to fit your needs
az storage account create \
  --name $STORAGE_ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP_NAME \
  --location $LOCATION \
  --sku Standard_LRS \
  --https-only true \
  --kind StorageV2

# Create Blob Container
az storage container create $CONTAINER_NAME

Secret Management

We also had to take care of secret management ourselves. For this purpose we used Azure Key Vault. Here’s an example of creating a Key Vault using Az CLI:

# Create Key Vault
az keyvault create \
  --resource-group $RESOURCE_GROUP_NAME \
  --name $KEYVAULT_NAME \
  --location $LOCATION

The currently signed-in user that creates the key vault is automatically assigned a predefined set of permissions in an Access Policy. This includes the right to create the cryptographic key. Example:

# Create cryptographic key
az keyvault key create \
  --vault-name $KEYVAULT_NAME \
  --name $KEYVAULT_KEY_NAME

Our engineering team used a common SP (Azure Active Directory Service Principal) to log in and interact with Azure. We used it when executing code with the Pulumi CLI. As well as from within CI/CD pipelines. Here’s an example of creating an SP (or rather an application and its SP in the current Azure AD Tenant):

az ad sp create-for-rbac \
  --name $SP_NAME \
  --role contributor \
  --scopes /subscriptions/$SUBSCRIPTION_ID

Save the returned JSON in a secure place. And keep it outside your version control system. We need it to log in to Azure. The JSON should look similar to this:

{
  "appId": "00000000-0000-0000-0000-000000000000",
  "displayName": "sp",
  "name": "http://sp",
  "password": "00000000-0000-0000-0000-000000000000",
  "tenant": "00000000-0000-0000-0000-000000000000"
}

We could now create a key vault access policy with sufficient permissions to use the cryptographic key. And assign it to the SP as:

az keyvault set-policy \
  --resource-group $RESOURCE_GROUP \
  --name $KEYVAULT_NAME \
  --object-id `az ad sp show --id ${SERVICE_PRINCIPAL_APPLICATION_ID} | jq -r .objectId` \
  --key-permissions decrypt encrypt get sign unwrapKey verify wrapKey

We then removed the automatically created access policy, as it was not needed anymore:

az keyvault delete-policy \
  --resource-group $RESOURCE_GROUP 
  --name $KEYVAULT_NAME 
  --object-id `az ad signed-in-user show --query objectId --output tsv`

At this stage we were done with setting up the cloud resources needed to self-manage the Pulumi backend in Azure.

In the next post I will go through some utility scripts that we created. They were used to automate some repetitive tasks while interacting with Pulumi and Azure. We are getting closer and to closer to creating a modern IaC solution based on Pulumi, don’t miss the the next post!

Secret Management

Next post