Should I run tests on every Ruby version I support?

Yes, run the suite on every Ruby minor version that required_ruby_version in your gemspec allows, one CI matrix job per version.

Should I test for unsupported Django versions?

No, do not test a package against Django versions it does not declare. Declare the supported range in its dependencies, so the installer picks a compatible release or reports the conflict.

Is it enough to test only the oldest and newest supported version?

Testing only the oldest and newest supported version is enough when each extra version costs real time, such as a 40-minute suite. New syntax fails on the oldest version and removed APIs fail on the newest, so the two ends catch most version breaks.

Should my application run its tests on more than one Node version?

No, an application needs its tests only on the Node version production runs, pinned in the Dockerfile and in .nvmrc. Run the suite on the next version in the upgrade branch before switching production.

Should I test against every language version?

Answer

Yes, test a package against every language version it declares as supported, with one CI matrix job per minor version from the oldest to the newest.

Verdict on the code under testYes

Why

Blast radiususers
Change frequencyregularly
Detectabilitysame-day
Reversibilitywith-effort
Test costtrivial

Yes, test against every language version your package declares as supported; rule R11 gives Test. The typical case is a library published to PyPI, npm or RubyGems with a supported range in its metadata, developed on the newest version. Blast radius is users, because the package's users get the error. Change frequency is regularly, because each monthly change can bring in a call the oldest version lacks. Detectability is same-day: a call the old version lacks raises an error there, and a user reports it soon after the release. Reversibility is with-effort, because PyPI does not let you re-upload a file under the same name, so you yank the release and publish a fix. Test cost is trivial: the suite exists, and a CI matrix takes one line.

When the decision changes
When	Decision	Why
An application runs only in a Docker image pinned to one language version, in CI, on developer machines and in production	Do not test other language versions; run the suite in the pinned image, and add the next version on the branch that upgrades it	Detectability moves to immediately and Reversibility to trivial, because the version changes only on the upgrade branch, whose suite runs in the new image and fails before merge
The package is a payment SDK that merchants call at checkout on their own Node versions	Test mandatory: every supported Node version is a required check before each release, with a test for each charge path	Blast radius rises to money and Reversibility to costly, because a checkout that fails on one Node version loses orders for good
The declared range still includes Python 3.8, past its end of life, and its job needs its own container and old pins for every test dependency	Do not test Python 3.8; raise requires-python in the next release, so pip on Python 3.8 installs the last release that supported it	Test cost rises to heavy, because the old image and pins break with each dependency update, while Detectability stays same-day
The full suite takes 40 minutes on one version, and the declared range spans five minor versions	Test minimally: run the oldest and the newest supported version on every pull request	Test cost rises to moderate, because each extra version adds a 40-minute run to every push
The package parses timestamps with datetime.fromisoformat, which accepts a trailing Z only from Python 3.11	Test on every supported version, with a timestamp that ends in Z in the test data	Detectability moves to eventually, because only inputs with a Z fail on Python 3.10; the decision holds, since the matrix already runs that version

What breaks if you don't test

A contributor uses a function from a newer version, CI passes on the newest, and users on the oldest declared version get an exception after the upgrade. The newest version breaks the other way: Python 3.12 removed distutils under PEP 632, so a package that still imported it failed on 3.12 while its CI on 3.11 stayed green.

What you lose if you over-test

Five Python versions on three operating systems against two Django versions make 30 jobs per push, and any flaky one blocks the merge. Patch versions add nothing, because a Python maintenance release accepts only bug and security fixes, as the devguide states.

How to test

Declare the range in one place: requires-python in pyproject.toml, engines in package.json, required_ruby_version in the gemspec.
Run one CI job per minor version in that range, with the latest patch, as a matrix strategy with fail-fast: false. Run the same set locally with tox.
On the JVM, compile once on the newest JDK with --release, and run the tests on each supported JDK through Gradle toolchains.
Change the declared range and the matrix in the same commit.

Procedure and references

When the answer changes

The code runs only on one version you control.
Money moves through the package.
A version in the range needs its own container to run.

Real incident + Code example

The removeprefix call that reached Python 3.8

On a Python library I maintained, the metadata said python_requires=">=3.8", and CI ran only on 3.11. A contributor replaced a slice with str.removeprefix, which PEP 616 added in Python 3.9. The evening after the release, a user on 3.8 reported AttributeError: 'str' object has no attribute 'removeprefix'. We yanked the release and published a patch with the slice back. A matrix went in the next day; with today's lower bound it reads:

# pyproject.toml: requires-python = ">=3.10"
name: tests
on: [pull_request, push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false   # a red 3.10 must not cancel 3.14
      matrix:
        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install -e ".[test]"
      - run: pytest

Sources

Should I test in every browser?Test it differently
Should tests run on every branch?Yes
Should I test Dockerfiles?Yes
Should integration tests run in CI?Yes
Should I test the production build?Yes

FAQ

Should I run tests on every Ruby version I support?: Yes, run the suite on every Ruby minor version that required_ruby_version in your gemspec allows, one CI matrix job per version.
Should I test for unsupported Django versions?: No, do not test a package against Django versions it does not declare. Declare the supported range in its dependencies, so the installer picks a compatible release or reports the conflict.
Is it enough to test only the oldest and newest supported version?: Testing only the oldest and newest supported version is enough when each extra version costs real time, such as a 40-minute suite. New syntax fails on the oldest version and removed APIs fail on the newest, so the two ends catch most version breaks.
Should my application run its tests on more than one Node version?: No, an application needs its tests only on the Node version production runs, pinned in the Dockerfile and in .nvmrc. Run the suite on the next version in the upgrade branch before switching production.