Secrets Management

The problem: secrets end up everywhere except where they should be#

A microservices system multiplies the number of secrets in play — a database password per service, an API key for every third-party integration, a signing key for every JWT issuer, credentials for every message broker — and multiplies the number of places those secrets can leak. Three patterns cause the overwhelming majority of real incidents:

• Secrets committed to source control. A database password hardcoded into application.yml or appsettings.json "just for now" gets committed, and even if it's removed in a later commit, it remains in the repository's history forever unless that history is rewritten and force-pushed everywhere — something most teams never actually do. Automated bots continuously scan public GitHub repositories and package registries specifically for patterns that look like cloud provider keys or database connection strings, and can find and attempt to use a leaked credential within minutes of a push, long before a human notices.
• Secrets in plain environment variables. An env var is readable by anything with access to the process's environment — /proc/<pid>/environ on Linux, a crash dump, a process listing on some platforms, or any logging/monitoring agent configured to capture the environment for debugging. It also gets inherited by every child process a service spawns, whether or not that child needed it.
• Secrets in plain config files on disk. A .env file or a config file with real credentials, sitting unencrypted on a filesystem, is one misconfigured backup, one overly broad file permission, or one compromised host away from being exposed — and unlike a committed secret, there's often no audit trail at all for who read it.

The fix isn't "be more careful" — it's removing the secret from all of those places entirely and fetching it, at startup or on demand, from a system built specifically to store and hand out secrets securely.

Secrets retrieval in each stack#

Spring Boot — Spring Cloud Vault#

Reading a secret from HashiCorp Vault at startup, authenticating via a Kubernetes service account rather than a static credential:

# application.yml
spring:
  cloud:
    vault:
      host: vault.internal
      port: 8200
      scheme: https
      authentication: KUBERNETES
      kubernetes:
        role: inventory-service
        service-account-token-file: /var/run/secrets/kubernetes.io/serviceaccount/token
  config:
    import: "vault://secret/inventory-service"

@Component
public class DatabaseConfig {

    @Value("${db.password}")
    private String dbPassword; // resolved from Vault at startup — never appears in application.yml, an env var, or git

    // ... used to configure the DataSource bean
}

The KUBERNETES authentication method means inventory-service never holds a static Vault token at all — it proves its identity using the service account token Kubernetes already injects into the pod, and Vault exchanges that for a short-lived Vault token behind the scenes.

Where a full Vault deployment isn't in place, the simpler pattern is a Kubernetes Secret mounted as a file, which many teams use as a stepping stone toward Vault:

# Deployment spec — mounts a Kubernetes Secret as a read-only file, not an env var
volumes:
  - name: db-credentials
    secret:
      secretName: inventory-db-credentials
containers:
  - name: inventory-service
    volumeMounts:
      - name: db-credentials
        mountPath: /etc/secrets
        readOnly: true

String dbPassword = Files.readString(Path.of("/etc/secrets/db-password")).trim();

ASP.NET Core — Azure Key Vault#

var builder = WebApplication.CreateBuilder(args);

builder.Configuration.AddAzureKeyVault(
    new Uri($"https://{builder.Configuration["KeyVaultName"]}.vault.azure.net/"),
    new DefaultAzureCredential());

var app = builder.Build();

// A secret named "InventoryDb--Password" in Key Vault becomes available at configuration
// path "InventoryDb:Password" — Key Vault uses "--" where .NET configuration uses ":"
var dbPassword = app.Configuration["InventoryDb:Password"];

DefaultAzureCredential is what avoids a static credential to reach Key Vault in the first place — running in Azure, it transparently uses the resource's managed identity; running locally, it falls back to the developer's own Azure CLI/Visual Studio login. Once wired in, secrets from Key Vault appear in IConfiguration exactly like any appsettings.json value, so the rest of the application doesn't need to know or care where a given setting actually came from.

Go — HashiCorp Vault's official client, and the Kubernetes file-mount pattern#

config := vault.DefaultConfig()
config.Address = "https://vault.internal:8200"

client, err := vault.NewClient(config)
if err != nil {
    log.Fatal(err)
}
client.SetToken(os.Getenv("VAULT_TOKEN")) // a short-lived token injected by the platform, not a hardcoded one

secret, err := client.Logical().Read("secret/data/inventory-service")
if err != nil {
    log.Fatal(err)
}

data := secret.Data["data"].(map[string]interface{}) // KV v2 nests the actual values under "data"
dbPassword := data["db_password"].(string)

The simpler pattern — reading a secret Kubernetes has already mounted as a file, with no Vault client or network call involved at all:

passwordBytes, err := os.ReadFile("/etc/secrets/db-password")
if err != nil {
    log.Fatal("could not read db password secret: ", err)
}
dbPassword := strings.TrimSpace(string(passwordBytes))

Both patterns share the same underlying principle as the Spring Boot and ASP.NET Core examples above: the secret's actual value is fetched at runtime from something designed to hold it, and it is never present in the service's own source, image, or committed configuration.

Rotation: why static secrets are risky, and what replaces them#

A static, long-lived secret — a database password set once and left unchanged for years — is a liability that grows over time rather than shrinking: every engineer who ever had access to it, every log line that might have captured it, every backup that ever included it, remains a way for it to have leaked, indefinitely, with no way to know if it already has. Rotating it manually is itself risky (every service using it must be updated in a coordinated window, or the old and new values must both work during a transition), which is exactly why it's so often deferred and simply never happens.

The modern approach is short-lived, dynamically generated secrets: instead of one password shared by every instance of a service forever, Vault's database secrets engine (as one concrete example) generates a unique database username and password for each request, with a lease that expires automatically after a set duration — an hour, a day. A leaked credential is only useful until its lease expires, rotation happens continuously and automatically as a side effect of normal operation, and no human ever needs to schedule a rotation window at all. The trade-off is that a client now needs to renew or re-fetch its lease before expiry rather than treating a credential as permanent — a small amount of extra client logic in exchange for removing an entire class of long-term exposure.

Never logging secrets, even accidentally#

The most common way a secret ends up in a log isn't a deliberate log.info(password) — it's a framework serializing an entire object, exception, or request/response body that happens to contain one. A DataSource configuration object logged for debugging, an exception whose message embeds a full connection string, or a request-logging filter that dumps every header including Authorization are all realistic ways a secret ends up sitting in a log aggregator, readable by anyone with log access and often retained far longer than the secret's own rotation period. Treat any object that might carry a secret field as something to explicitly exclude from toString()/serialization, mask sensitive header values in request-logging middleware by name (Authorization, X-Api-Key, Cookie), and review what an exception's message actually contains before letting it flow into logs unmodified.

Common mistakes#

• Believing a secret is safe once a commit that added it is reverted — the original commit, and the secret in it, still exist in the repository's history and remain fetchable by anyone with clone access unless the history itself is rewritten.
• Storing secrets in plain environment variables and considering that "secure enough" — they're still inherited by child processes, visible in crash dumps and some process listings, and offer no audit trail for who read them.
• Hardcoding a real secret as a fallback/default value "just for local development" — this value ships in the same source and image as production code, and a missing environment variable in production silently falls back to it instead of failing loudly.
• Treating a static secret as permanent and never revisiting it after it's first set — especially after an incident, or after an engineer with access to it leaves, a secret that's never rotated stays exactly as exposed as it ever was.
• Logging a full config, request, or exception object without checking what it actually serializes to — a secret can end up in a log aggregator this way without a single explicit line of code that looks like it's logging anything sensitive.