Connecting Dynaconf to Hashicorp Vault

Using Dynaconf for settings and secrets in Python projects is my favorite approach. Connecting it to Hashicorp vault as the backend is a bit tricky though since the documentation for this is very sparse.

The documentation here gives a working example, but does not explain how you set up the same with a config.py file which is the dynaconf standard. Here is a working example.

First here is the config.py you will need:

from dynaconf import Dynaconf

settings = Dynaconf(
    environment=True,
    load_dotenv=True,
    settings_files=["settings.toml", ".secrets.toml"],
)

And here is how you use it from your main script:

from config import settings

print(settings.PASSWORD)
print(settings.from_env("prod").PASSWORD)
print(settings.USERNAME)

You will need a .env file with contents like this:

# .env
VAULT_ENABLED_FOR_DYNACONF=true

#VAULT_URL_FOR_DYNACONF="http://localhost:8200"
# Specify the secrets engine for kv, default is 1
VAULT_KV_VERSION_FOR_DYNACONF=1
VAULT_MOUNT_POINT_FOR_DYNACONF="cubbyhole"
VAULT_PATH_FOR_DYNACONF="dynaconf"
ENV_FOR_DYNACONF="prod"

#VAULT_ROOT_TOKEN_FOR_DYNACONF = "REDACTED"
VAULT_TOKEN_FOR_DYNACONF="REDACTED"

Using these settings dynaconf will look for PASSWORD and USERNAME in the secret in the engine called cubbyhole under the path dynaconf. Under dynaconf you can have different environements including a default one which will be picked if there is no match in the other envs. In our .env vi pick the prod environment.

Note: If you also want to use settings file e.g. settings.toml and .secrets.toml at the same time you will need to make these use environments as well e.g.

[prod]
USERNAME = "Hans"
PASSWORD = "Prod override"

[default]
USERNAME = "default user"
PASSWORD = "default password"
TEST = "This is a test key"

The loading order is given

CORE_LOADERS_FOR_DYNACONF and LOADERS_FOR_DYNACONF which you can inspect in your settings object when running your script. This in effect means that first settings and secret files are loaded, then the vault is loaded and lastly environment variables are loaded. In case of overlap the last loaded value overrides and previous values. In case you need to merge secrets look into this here.

Note that you should decide the correct auth method for your use case towards the Vault. In this case we are using tokens, but these expire. Username password could be a useful alternative for cases expiring tokens will be a problem.