Securing Ansible Vault with Google Cloud

When working with Ansible you will at some point have to deal with data that is of a more sensitive nature such as passwords, API- & certificate keys etc. Storing secrets in plain text is bad practice, but still quite common.

If possible the best option is to simply not store any secrets at all and instead fetch/inject these during deployment or runtime with tools such as Hashicorp’s Vault. But for smaller projects this can be too expensive, complex and time-consuming to configure. Thankfully Redhat has included a tool called Ansible Vault in the default Ansible installation. Ansible Vault can encrypt secrets inline or separate files and then automatically decrypt during playbook execution.

To perform Ansible Vault encryption/decryption-operations a vault password needs to be supplied via command prompt or file. Having to supply the vault password manually during each playbook execution will become tedious in the long run and fetching the password from a clear-text file bring about as much benefit as hiding the key on top of the safe.

What’s the better alternative? Let someone else manage the vault password for you! That someone can be a managed third-party service such as Azure Keyvault, AWS SSM, AWS Secrets Manager or Google Secret Manager.

Some of the top benefits using a cloud service provider:

• RBAC (Who has access to the vault?)
• Auditing/Traceability (Who decrypted which vault, when and where?)
• Versioning
• Enables programmatic logic (Easy to implement in existing pipelines via API and custom SDKs)

Table of contents

• A simple Ansible Vault Password client integrated with Google Secret Manager
• Prerequisites
• Environment setup and testing Ansible Vault
• Add Ansible Vault password as a Secret in Google Secret Manager
• Setup GCP Service Account for Authorization and Authentication
• Ansible Vault Password Client
• Using the client with Ansible
• Auditing

A simple Ansible Vault Password client integrated with Google Secret Manager

The Ansible Vault client in this example will be built using Python. There are of course plenty of other languages to chose from to achieve the same result. Python is however a particular good fit as it is guaranteed to be present wherever Ansible is installed.

Prerequisites

• Ansible, Python 3 and pip3 installed.
• A Google Cloud Platform account with access to IAM and Secret Manager
• gcloud CLI installed and configured (authenticated gcloud auth login and project set gcloud config set project my-gcp-project-12345)

Environment setup and testing Ansible Vault

1. First export some environment variables.

## The password Ansible Vault will use for cryptographic operations
export VAULT_PASSWORD="Winter2020"

## Tag that associates Ansible Vault with the environment our vault secrets belongs to.
#  E.g. development, production etc.
export VAULT_ID="dev"

## The Google Cloud Project ID used
export GCP_PROJECT_ID="my-gcp-project-12345"

2. Create file password that stores $VAULT_PASSWORD in plain text and vars.yml that stores some super secret data in plain text.

cat << EOF > password
$VAULT_PASSWORD
EOF

cat << "EOF" > vars.yml
credit_card_no: "4539680266571992"
favorite_color: "blue"
mothers_maiden_name: "bates"
EOF

3. Test encrypting vars.yml with Ansible vault using password from file password

ansible-vault encrypt --vault-id ${VAULT_ID}@password vars.yml

4. vars.yml should now be encrypted.

cat vars.yml
$ANSIBLE_VAULT;1.2;AES256;dev
61626463303539626332336233663033323437343365353634656132663236613936313132366533
3761363737636430633962323462386631356632353238660a643762326139663566306166626264
31303066613663373866363037383633306561353135356164653338633733343066336435353733
6430356161306266390a393634633965646331386238663466303464633863393336356439343038
32363333383831636334383561313464376539306439613063393962393535636232616636346336
62383737633364326636653931313962356239353361373661623332643764313965663837366136
31333336306538636562633935343961303835356631373064323232633165303432366632303233
61656165616233383032343539346434306565643462613566346661653331326339376137306263
3265

Add Ansible Vault password as a Secret in Google Secret Manager

Create a new Secret with identifier ansible-vault-dev in Google Secret Manager containing $VAULT_PASSWORD.

• Flag --location is set to a nearby location.
• --replication-policy=user-managed is set as replication is not required for this example.
• The --data-file-flag is set to a dash "-", which specifies that secret value will be fetched from stdout.
• Also remove file password as it is no longer going to be used.

export GCP_VAULT_SECRET_ID="ansible-vault-$VAULT_ID"

echo "${VAULT_PASSWORD}" | gcloud secrets create ${GCP_VAULT_SECRET_ID} \
--locations=europe-north1 \
--replication-policy=user-managed \
--data-file=-

rm password

Setup GCP Service Account for Authorization and Authentication

1. Create a new service account that will be named with the prefix sa- used to read the secret.

export GCP_SERVICE_ACCOUNT="sa-${GCP_VAULT_SECRET_ID}"

gcloud iam service-accounts create ${GCP_SERVICE_ACCOUNT} \
   --description="Service Account for retrieving Ansible vault password in dev env" \
   --display-name="$GCP_SERVICE_ACCOUNT"

2. With a “principle of least privilege” approach set a policy binding to only allow the service account to read from this particular secret.

gcloud secrets add-iam-policy-binding ${GCP_VAULT_SECRET_ID} \
   --member="serviceAccount:${GCP_SERVICE_ACCOUNT}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
   --role='roles/secretmanager.secretAccessor'

3. Create a credentials key for the Service Account. These credentials will be used by the Ansible Vault client script to authenticate to Google Secret Manager.

Note: Anyone with access to this key will be able to read the value of $GCP_VAULT_SECRET_ID and any resource granted by additional configured roles. Keep it safe.

gcloud iam service-accounts keys create key.json \
--iam-account $GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com

Ansible Vault Password Client

1. The Ansible Vault password client will utilize the google-cloud-secret-manager python library. pip install if not already installed.

pip3 install google-cloud-secret-manager

2. Create a new file called ansible-vault-gcp-client.py and paste in the code. To utilize the built-in Ansible function that provides the vault-id to the password script the filename has to end with ...-client.{extension}.

Replace the placeholder {GCP_PROJECT_ID} with actual GCP project ID.

#!/usr/bin/env python3
import argparse

# Replace placeholder ${GCP_PROJECT_ID} with actual project ID
project_id = "{GCP_PROJECT_ID}"
secret_id_prefix = "ansible-vault-"

def access_secret_version(project_id, secret_id, version_id):
      # Import the Secret Manager client library.
      from google.cloud import secretmanager

      # Create the Secret Manager client.
      client = secretmanager.SecretManagerServiceClient()

      # Build the resource name of the secret version.
      name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

      # Access the secret version.
      response = client.access_secret_version(request={"name": name})

      # Print the secret payload.
      # WARNING: Do not print the secret in a production environment - this
      # snippet is showing how to access the secret material.
      payload = response.payload.data.decode('UTF-8')
      print(format(payload))

# Build command line parser
parser = argparse.ArgumentParser()

# Accept argument flag for Vault ID from Ansible
parser.add_argument("--vault-id")
args = parser.parse_args()

if args.vault_id:
      vault_id = args.vault_id

access_secret_version(project_id, secret_id_prefix + vault_id, "latest")

The code is more or less derived from Google’s sample code provided in their python-secret-manager repository. And a friendly disclaimer that the code here is only an example and should be viewed as NOT PRODUCTION-READY.

3. Export path to Service Account key.json file to environment. Google Cloud SDK will automatically use these credentials when this environment variable is present.

export GOOGLE_APPLICATION_CREDENTIALS="$PWD/key.json"

4. Make ansible-vault-gcp-client.py script executable.

chmod +x ansible-vault-gcp-client.py

Using the client with Ansible

1. To see the script in action try to decrypt file vars.yml which was previously encrypted with Ansible Vault.

ansible-vault decrypt --vault-id dev@ansible-vault-gcp-client.py vars.yml

Should return "Decryption successful" and the content should now be readable with cat vars.yml:

credit_card_no: '4539680266571992'
favorite_color: 'blue'
mothers_maiden_name: 'bates'

Success!

2. To use the client together with a playbook we’ll modify vars.yml to follow best practices by referencing the secrets which will be placed in a new file, vault.yml.

cp vars.yml vault.yml

## Remove secret value from vars.yml
## and replace with vault variable references
sed -i -r 's/^(.*)\:(.*)/\1: "{{ vault_\1 }}"/' vars.yml

## update variables with vault_ prefix in vault.yml
sed -i -e 's/^/vault_/' vault.yml

When done cat vars.yml should produce this:

credit_card_no: '{{ vault_credit_card_no }}'
favorite_color: '{{ vault_favorite_color }}'
mothers_maiden_name: '{{ vault_mothers_maiden_name }}'

and cat vault.yml:

vault_credit_card_no: '4539680266571992'
vault_favorite_color: 'blue'
vault_mothers_maiden_name: 'bates'

3. Encrypt vault.yml with our script.

ansible-vault encrypt --vault-id ${VAULT_ID}@ansible-vault-gcp-client.py vault.yml

4. Let’s now create an Ansible Playbook, print-secrets.yml, with the only task of including and printing out secrets that has been encrypted with Ansible Vault

cat << EOF > print-secrets.yml
- name: Print secrets from Ansible Vault
hosts: localhost
gather_facts: false

tasks:
- include_vars:
      file: "{{ item }}"
      loop:
      - vars.yml
      - vault.yml
- debug:
      msg:
         - "Credit Card number is: {{credit_card_no}}"
         - "Favorite color is: {{ favorite_color }}"
         - "Mother's maiden name is:  {{ mothers_maiden_name }}"
EOF

5. Now execute the playbook:

ansible-playbook --vault-id ${VAULT_ID}@ansible-vault-gcp-client.py print-secrets.yml

The result should now print out the secrets:

PLAY [Print secrets from Ansible Vault] ***********************************************

TASK [include_vars] *******************************************************************
ok: [localhost] => (item=vars.yml)
ok: [localhost] => (item=vault.yml)

TASK [debug] **************************************************************************
ok: [localhost] => {
      "msg": [
         "Credit Card number is: 4539680266571992",
         "Favorite color is: blue",
         "Mother's maiden name is:  bates"
      ]
}

PLAY RECAP ****************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Auditing

Head on over to the metrics dashboard for Secret Manager API. There it should now display at least a couple of requests towards the method google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion. Requests towards this method are performed every time an encrypt/decrypt action with Ansible Vault and the script ansible-vault-gcp-client.py is executed.

If we’d like we could narrow it down even further by using the Credentials Filter and only selecting the SA-ansible-vault-dev credential, which is the service account previously created with permission to read the Ansible Vault password.

For more granular auditing there’s also the option to enable Audit Logs for Secret Manager API. Every read or write operation will then be indexed and possible to query via Cloud Logging.