Deployment Gotchas

Deployment Gotchas

Not a deployment guide — these are the 7 things that break every first Phoenix deploy. Every rule maps to a real production incident pattern.

RULES — Follow these with no exceptions

1. Use runtime.exs for secrets and URLs — config.exs/prod.exs are compiled into the release and cannot read env vars at boot
2. Run migrations via release commands (bin/migrate) — mix is not available in production releases
3. Set PHX_HOST and PHX_SERVER=true — without these, URL generation breaks and the server won't start
4. Run mix assets.deploy before building the release — forgetting this means no CSS/JS in production
5. Never hardcode secrets — use System.fetch_env!/1 in runtime.exs (the ! crashes on boot if missing, which is what you want)
6. Split health checks into liveness and readiness — liveness returns 200 without touching the DB; only readiness queries the database
7. Use config :logger, level: :info in production — :debug logs query parameters including user data

1. runtime.exs vs config.exs

The incident: App deploys fine but uses the wrong database URL. DATABASE_URL was set correctly in the environment, but the release ignores it.

Why: config.exs and prod.exs are evaluated at compile time and baked into the release. runtime.exs is evaluated at boot time and can read environment variables.

Bad:

# config/prod.exs — compiled into release, cannot read env vars at boot
config :my_app, MyApp.Repo,
  # Evaluated at BUILD time — captures the build machine's env, not the
  # runtime env. Silently wrong in a release; use runtime.exs instead.
  url: System.get_env("DATABASE_URL")

Good:

# config/runtime.exs — evaluated at boot, reads env vars correctly
if config_env() == :prod do
  database_url = System.fetch_env!("DATABASE_URL")

  config :my_app, MyApp.Repo,
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
end

Rule of thumb: If the value comes from the environment, it goes in runtime.exs. If it's a static setting, it goes in config.exs.

2. Release Migrations

The incident: Deploy succeeds but the app crashes on boot because new columns don't exist. Developer tries mix ecto.migrate on the server — mix: command not found.

Why: Production releases don't include Mix or the Elixir compiler. Migrations must be run via release commands.

Bad:

# mix is not available in production releases
ssh prod-server "cd /app && mix ecto.migrate"

Good:

# lib/my_app/release.ex
defmodule MyApp.Release do
  @app :my_app

  def migrate do
    load_app()

    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    load_app()
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos do
    Application.fetch_env!(@app, :ecto_repos)
  end

  defp load_app do
    Application.ensure_all_started(:ssl)
    Application.load(@app)
  end
end

# Run migrations in production
bin/my_app eval "MyApp.Release.migrate()"

# Or via rel/overlays if configured
bin/migrate

3. PHX_HOST and PHX_SERVER

The incident: Deploy succeeds, health check passes, but all URLs in emails and redirects point to localhost:4000. Or worse — the server doesn't start at all.

Why: Without PHX_SERVER=true, the Phoenix endpoint doesn't start its HTTP listener. Without PHX_HOST, URL helpers generate localhost URLs.

Bad:

# config/runtime.exs — missing host and server config
config :my_app, MyAppWeb.Endpoint,
  url: [host: "localhost"],  # Wrong in production!
  http: [port: 4000]
  # Server doesn't start without server: true

Good:

# config/runtime.exs
if config_env() == :prod do
  host = System.fetch_env!("PHX_HOST")
  port = String.to_integer(System.get_env("PORT") || "4000")

  config :my_app, MyAppWeb.Endpoint,
    url: [host: host, port: 443, scheme: "https"],
    http: [ip: {0, 0, 0, 0}, port: port],
    server: true  # Or set PHX_SERVER=true env var
end

4. Asset Deployment

The incident: App deploys, pages load, but CSS/JS are missing. The page is unstyled raw HTML.

Why: Assets must be compiled and digested before the release is built. The release bundles priv/static — if assets aren't there at build time, they won't be in the release.

Bad:

# Dockerfile — builds release without compiling assets
RUN mix release

Good:

# Dockerfile — correct order
RUN mix assets.deploy
RUN mix release

# Manual build order
mix deps.get --only prod
MIX_ENV=prod mix compile
MIX_ENV=prod mix assets.deploy  # Must come before release
MIX_ENV=prod mix release

What mix assets.deploy does:

1. Runs tailwind and esbuild to compile CSS/JS
2. Runs phx.digest to fingerprint files for cache busting
3. Generates cache_manifest.json for the endpoint to serve

5. Never Hardcode Secrets

The incident: Secret key leaks into git history via config/prod.exs. Rotating it requires a new release.

Why: Secrets in compiled config are baked into the release binary and visible in version control.

Bad:

# config/prod.exs — secret in source code
config :my_app, MyAppWeb.Endpoint,
  secret_key_base: "actual_secret_key_here_in_git_history"

Good:

# config/runtime.exs — read from environment, crash if missing
if config_env() == :prod do
  secret_key_base = System.fetch_env!("SECRET_KEY_BASE")

  config :my_app, MyAppWeb.Endpoint,
    secret_key_base: secret_key_base
end

Why fetch_env! (with bang): If the secret is missing, the app crashes immediately on boot with a clear error. Plain System.get_env/1 returns nil when missing and fails later with a confusing error.

# Generate a secret
mix phx.gen.secret

# Set in environment (never in source)
export SECRET_KEY_BASE="generated_secret_here"