Should you test configuration files?

Test configuration files with one test that loads each environment's file through the real loader and validates it against a settings model. A missing key in the production file then fails CI instead of failing for customers.

Should I write unit tests that assert configuration values?

Unit tests that assert ordinary configuration values, such as timeouts and URLs, repeat the file and fail on every intended change, so skip them. Assert values only where a wrong value costs money or exposes data, such as tax rates or debug mode.

Where is the boundary between configuration and code?

Configuration that runs is code: a factory, a dependency injection module, or a template with conditionals gets the same decision as other code. Static values get a load-and-validate test, and wiring code gets a startup test that builds the object graph once.

Should I test environment variables?

Validate environment variables at application startup instead of in a CI test, and make startup fail when a required variable is missing or malformed. A CI run cannot see production values, because the deploy platform injects them at runtime.

Should I test configuration?

Verdict

Yes

Write one test that loads the configuration file of every environment through your real loader and validates it, and do not assert individual configuration values.

Why

Test configuration minimally: one test that every environment's file loads and validates, none for individual values. Blast radius is users, because a missing key in the production file breaks a feature customers use. Change frequency is regularly, since a service typically changes a key about once a month. Detectability is same-day: many loaders fall back to a default for a missing key, so the application starts and error reports arrive within hours. Reversibility is trivial, because a revert and a redeploy leave no wrong data behind, and Test cost is moderate, since the test needs a settings model and placeholder secrets.

When the decision changes
When	Decision	Why
The configuration holds prices, tax rates or payment provider settings	Test mandatory: validate every configured price and tax rate against an allowed range in CI	Blast radius rises to money and Reversibility to costly, because wrong charges need refunds
The configuration switches debug mode, TLS verification or allowed CORS origins	Test mandatory: assert that production configuration disables debug mode, verifies TLS and allows no wildcard origin	Blast radius rises to safety-or-legal, Detectability to never and Reversibility to impossible, because leaked data raises no error
The application validates configuration at startup, and the deploy keeps the old version until health checks pass	Do not add a configuration test; keep the startup validation	Detectability rises to immediately, because a bad value stops the deploy in front of the developer who made the change
The configuration only sets up local development, such as a Docker Compose file	Do not test local development configuration; fix it when a developer's setup breaks	Blast radius falls to internal, because no customer runs the local setup
Non-developers change values at runtime in a feature flag service	Test flag changes differently: roll out in stages and alert on the error rate	Change frequency rises to constantly and Test cost to prohibitive, because no CI run sees those values
Templates with conditionals render the configuration, such as a Helm chart	Test the rendered output for each environment in CI	Detectability moves to eventually and Test cost falls to trivial, because a production-only branch fails nowhere else and rendering needs no cluster

What breaks if you don't test

A developer adds a key to the development and staging files and forgets the production file. CI and staging pass, because their files have the key. In production the loader substitutes a default, the application starts, and the feature behind the key fails for every customer until someone compares the files.

What you lose if you over-test

A test that asserts timeout_seconds == 30 repeats the configuration file. An intended timeout change fails the test, and the fix is to copy the new number into it. Every configuration change becomes two edits, and reviewers learn to update the test without reading it. Such a test also misses a key absent from the production file, because it reads the test file.

How to test

Write one unit test, parametrized over environments with pytest parametrize. For each environment, load the real file through the application's loader and validate it against a settings model with no defaults for required keys. Make the model reject unknown keys, for example with extra="forbid" in pydantic, so a misspelled key fails. Use placeholder secrets, because the real ones live in the deploy platform. Add a value assertion only after a wrong value causes an incident.

Procedure and references

When the answer changes

Some keys control money or security, such as tax rates or TLS verification, and need value assertions.
The configuration gains logic: templates with conditionals or a factory that picks implementations. Test the configuration's rendered output, for example with helm template.
The application refuses to start on invalid configuration and the deploy rolls back on a failed health check, so the startup check replaces the test.

Real incident + Code example

The password reset email that went to localhost

On a SaaS product I worked on, a developer added smtp_host to the development and staging YAML files and missed the production file. Our loader defaulted to localhost, so production started without an error and every password reset email failed. Support tickets told us fourteen hours after the deploy. The test we added loads every environment file through a settings model that has no default for smtp_host:

from pathlib import Path

import pytest
import yaml

from app.settings import Settings  # pydantic model, extra="forbid"

CONFIG_DIR = Path(__file__).parent.parent / "config"
ENVIRONMENTS = sorted(p.stem for p in CONFIG_DIR.glob("*.yaml"))


@pytest.mark.parametrize("env", ENVIRONMENTS)
def test_environment_config_is_valid(env):
    raw = yaml.safe_load((CONFIG_DIR / f"{env}.yaml").read_text())
    Settings.model_validate(raw)

The test asserts no values. It fails on a missing key, a wrong type, or an unknown key such as smtp_hots. A new environment file adds a test case without an edit.

FAQ

Should you test configuration files?: Test configuration files with one test that loads each environment's file through the real loader and validates it against a settings model. A missing key in the production file then fails CI instead of failing for customers.
Should I write unit tests that assert configuration values?: Unit tests that assert ordinary configuration values, such as timeouts and URLs, repeat the file and fail on every intended change, so skip them. Assert values only where a wrong value costs money or exposes data, such as tax rates or debug mode.
Where is the boundary between configuration and code?: Configuration that runs is code: a factory, a dependency injection module, or a template with conditionals gets the same decision as other code. Static values get a load-and-validate test, and wiring code gets a startup test that builds the object graph once.
Should I test environment variables?: Validate environment variables at application startup instead of in a CI test, and make startup fail when a required variable is missing or malformed. A CI run cannot see production values, because the deploy platform injects them at runtime.