Should I test that?

Data and state

How to test data and state: production data, migrations, caching, race conditions, dates and time zones.

Most asked

  1. Should I test with an in-memory database?Code under test: Yes
  2. Should I test with production data?Test it differently
  3. Should I test SQL queries?Yes
  4. Should I use random data in tests?No
  5. Should I test for race conditions?Yes

Yes5

  • Yes, give each transformation step of a recurring data pipeline one test that runs it on a few hand-built input rows, chosen so that a wrong join or filter changes the result, and compares the output with rows you worked out by hand.

  • Yes, test each race condition the code must survive, such as two requests creating the same record, with a test against the real database that runs both checks before either write; do not add random stress tests to hunt for races nobody has named.

  • Yes, test every SQL query that filters, joins or aggregates with an integration test against the database engine production runs, using rows the query must return, rows it must leave out and rows with NULL in the compared columns.

  • Test each stored procedure that holds business rules with a database test on the production engine, started in a container, using rows that each branch and filter of the procedure must treat differently.

  • Should I test with an in-memory database?

    Yes, test code that runs SQL, on the database engine production runs: use an in-memory database only when production runs the same engine, such as SQLite in a mobile app, and otherwise start PostgreSQL or MySQL in a container.

    Code under test: Yes

No1

  • No, do not fill example tests with random values; give each test fixed values, one from each class of input that the code treats in its own way, so a failure repeats on the next run.

Test it differently2

  • Should I test data quality?Test it differently

    Test data quality differently: after each load, run checks on the share of empty values, the row count and the time of the newest row, and alert when a check fails.

  • Test it differently: keep production data out of the test suite, and find the record shapes your hand-written fixtures miss with aggregate queries on a read replica and an alert on records your code rejects in production.