Managing Secrets

Managing Secrets

Secure storage, rotation, and delivery of secrets (API keys, database credentials, TLS certificates) for applications and infrastructure.

When to Use This Skill

Use when:

• Storing API keys, database credentials, or encryption keys
• Implementing secret rotation (manual or automatic)
• Syncing secrets from external stores to Kubernetes
• Setting up dynamic secrets (database, cloud providers)
• Scanning code for leaked secrets
• Implementing zero-knowledge patterns
• Meeting compliance requirements (SOC 2, ISO 27001, PCI DSS)

Quick Decision Frameworks

Framework 1: Choosing a Secret Store

Decision Tree:

• Kubernetes? → External Secrets Operator (ESO) with chosen backend
• Single cloud? → Cloud-native (AWS/GCP/Azure)
• Multi-cloud/on-prem? → HashiCorp Vault
• GitOps? → SOPS or Sealed Secrets

Framework 2: Static vs. Dynamic Secrets

Framework 3: Kubernetes Secret Delivery

HashiCorp Vault Fundamentals

Core Components

• Secrets Engines: KV v2 (static), Database (dynamic), AWS, PKI, SSH
• Auth Methods: Kubernetes, JWT/OIDC, AppRole, LDAP
• Policies: HCL-based access control (least privilege)
• Leases: TTL for secrets, auto-renewal, auto-revocation

Static Secrets (KV v2)

# Create secret
vault kv put secret/myapp/config api_key=sk_live_EXAMPLE

# Read secret
vault kv get secret/myapp/config

# List versions
vault kv metadata get secret/myapp/config

Dynamic Database Credentials

# Configure PostgreSQL
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb"

# Create role
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\"..." \
  default_ttl="1h"

# Generate credentials
vault read database/creds/app-role

For detailed Vault architecture, see references/vault-architecture.md.

Kubernetes Integration

External Secrets Operator (ESO)

Syncs secrets from 30+ providers to Kubernetes Secrets.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          role: "app-role"

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/database/config

Vault Secrets Operator (VSO)

Kubernetes-native Vault integration with automatic lease renewal.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    name: dynamic-db-creds

For ESO vs CSI vs VSO comparison, see references/kubernetes-integration.md.

Secret Rotation Patterns

Pattern 1: Versioned Static Secrets (Blue/Green)

1. Create new secret version in Vault
2. Update staging environment
3. Monitor for errors (24-48 hours)
4. Gradual production rollout (10% → 50% → 100%)
5. Revoke old secret (after 7 days)

Pattern 2: Dynamic Database Credentials

Vault auto-generates credentials with short TTL:

• App fetches credentials from Vault
• Vault automatically renews lease (at 67% of TTL)
• On expiration, Vault revokes access
• On renewal failure, app requests new credentials

Pattern 3: TLS Certificate Rotation

Using cert-manager + Vault PKI:

• cert-manager requests certificate from Vault
• Automatically renews before expiration (default: 67% of duration)
• Updates Kubernetes Secret on renewal
• Optional pod restart (via Reloader)

For detailed rotation workflows, see references/rotation-patterns.md.

Multi-Language Integration

Python (hvac)

import hvac

client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes(role='app-role', jwt=jwt)

# Fetch dynamic credentials
response = client.secrets.database.generate_credentials(name='postgres-role')
username = response['data']['username']
password = response['data']['password']

Go (Vault API)

import vault "github.com/hashicorp/vault/api"

client, _ := vault.NewClient(vault.DefaultConfig())
k8sAuth, _ := auth.NewKubernetesAuth("app-role")
client.Auth().Login(context.Background(), k8sAuth)

secret, _ := client.Logical().Read("database/creds/postgres-role")

TypeScript (node-vault)

import vault from 'node-vault';

const client = vault({ endpoint: 'https://vault.example.com' });
await client.kubernetesLogin({ role: 'app-role', jwt });

const response = await client.read('database/creds/postgres-role');

For complete examples, see examples/dynamic-db-credentials/.

Secret Scanning

Pre-Commit Hooks (Gitleaks)

# Install Gitleaks
brew install gitleaks

# Run on staged files
gitleaks protect --staged --verbose

Pre-commit hook prevents secrets from being committed. For setup, see examples/secret-scanning/pre-commit.

CI/CD Integration

# GitHub Actions
- name: Run Gitleaks
  uses: gitleaks/gitleaks-action@v2

Remediation Workflow

When a secret is leaked:

1. Rotate immediately (within 1 hour)
2. Revoke at provider
3. Remove from Git history (BFG Repo-Cleaner)
4. Force push (notify team)
5. Audit access (who had access during leak window)
6. Document incident

For detailed remediation, see references/secret-scanning.md.

Zero-Knowledge Patterns

Client-Side Encryption (E2EE)

User password → PBKDF2 → encryption key → encrypt secret → send to server

Server stores only encrypted blobs (cannot decrypt).

Shamir's Secret Sharing

Split secret into N shares, require M to reconstruct (e.g., 3 of 5).

# Initialize Vault with Shamir shares
vault operator init -key-shares=5 -key-threshold=3

# Unseal requires 3 of 5 key shares
vault operator unseal <KEY_1>
vault operator unseal <KEY_2>
vault operator unseal <KEY_3>

For implementations, see references/zero-knowledge.md.

Library Recommendations (2025)