Verdict
Yes
Yes, test a vibe-coded app that customers use with end-to-end tests of its main flows, run on every push and after every prompt session against expected results you write yourself, because a prompt session can rewrite a file the prompt did not name and make the app save bookings with a plausible wrong date that no error shows and no customer reports for weeks.
Why
- Blast radiususers
- Change frequencyconstantly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costmoderate
Yes, test a vibe-coded app once customers use it, such as a class booking app built by prompting an assistant and accepting changes unread. Blast radius is users, because customers see broken screens and wrong bookings. Change frequency is constantly: every prompt session rewrites code, often in files the prompt did not name. Detectability is eventually, because I click through only the feature I asked for, and a rewritten file can store bookings with a plausible wrong date that no error shows and no customer reports for weeks. Reversibility is with-effort, since wrong records need a repair script, and Test cost is moderate, about an hour per flow, so rule R11 gives Test.
| When | Decision | Why |
|---|---|---|
| The vibe-coded app runs only on your own machine, such as a tool that tags your own notes | Do not test the app; check its output when you use it | Blast radius falls to none: a failure affects nobody but you |
| The vibe-coded app is a demo with fake data that you click through after every prompt before you show it to your co-founders | Do not test the demo; click through every screen before each showing | Blast radius falls to internal, Detectability moves to immediately and Reversibility to trivial: you see each break at once, and fake data needs no repair |
| The vibe-coded app charges cards with live payment keys | Test mandatory: test the charged amount, a repeated request and a refund before the first real payment | Blast radius rises to money and Reversibility to costly: a double charge ends in refunds |
| Customers sign in to a vibe-coded app that stores their personal records, and the assistant wrote the access rules, such as Supabase row-level security policies | Test mandatory: test that a signed-out request and a second account read none of one customer's records | Blast radius rises to safety-or-legal and Detectability to never: an open table returns its rows to anyone and raises no error |
| The vibe-coded app is a static page that stores nothing, and visitors report a broken page by email | Test minimally: one end-to-end test that loads the page and follows its main link | Detectability moves to same-day and Reversibility to trivial: visitors report the break that day, and a redeploy fixes it |
| You stop prompting, and the booking flow of the vibe-coded app changes a few times a year | Test minimally: one end-to-end test of the booking flow against seeded records | Change frequency falls to rarely: a flow that changes a few times a year gets few chances to break |
What breaks if you don't test
The prompt "add a waitlist to full classes" can also change the query that lists bookings, because the assistant refactors whatever it reads. The new feature works when you try it, and the old one breaks where you did not look. Customers find it first: a cancellation does not free the seat, and the support email arrives a week later. Nobody read the diff, so you debug code you have never seen.
What you lose if you over-test
An agent asked for "full test coverage" writes hundreds of unit tests in minutes, each asserting what the generated code already returns. The next prompt rewrites the code and the tests to match, so the suite stays green and checks nothing you decided. Unit tests pinned to a structure that changes weekly also fail on refactors that break no flow.
How to test
Test the flows customers use, end to end:
- Write each flow as steps and expected results before you prompt: book a class, cancel, see the booking list.
- Turn each flow into a Playwright test. The agent may type the code; you check every expected value against your list.
- Tell the agent to run the suite before it finishes each task, and run it in CI on every push.
- When a customer reports a bug, add a test that reproduces it before you prompt for the fix.
Procedure and references
When the answer changes
- The app takes payments or stores records that only their owner may read.
- Nobody but you uses the app.
- You stop prompting and the code settles.
Real incident + Code example
170 apps with open tables
In March 2025, Matt Palmer scanned 1,645 apps built with Lovable, a vibe-coding tool that stores data in Supabase, and found 170 whose tables returned data such as emails, payment details or API keys to requests from outside (his disclosure). The cause was missing or wrong row-level security policies. A signed-in author sees the data they expect, so only a request made as someone else shows the leak:
import { test, expect } from "@playwright/test";
import { createClient } from "@supabase/supabase-js";
const url = process.env.SUPABASE_URL!;
const anonKey = process.env.SUPABASE_ANON_KEY!;
// The test project holds one booking for a@test.local.
test("a signed-out request reads no bookings", async () => {
const { data } = await createClient(url, anonKey).from("bookings").select("*");
expect(data ?? []).toHaveLength(0);
});
test("customer B reads none of customer A's bookings", async () => {
const b = createClient(url, anonKey);
await b.auth.signInWithPassword({ email: "b@test.local", password: process.env.B_PASSWORD! });
const { data } = await b.from("bookings").select("*").eq("email", "a@test.local");
expect(data ?? []).toHaveLength(0);
});
Related questions
FAQ
- Do vibe-coded apps need tests?
Yes, a vibe-coded app that customers use needs end-to-end tests of its main flows. Each prompt can change code the prompt did not name, and nobody reads the diff to catch it.
- Is clicking through the app enough testing for a vibe-coded app?
No, clicking through a vibe-coded app checks only the screens you open, while the assistant also edits flows you do not open. A suite that runs every flow after each prompt session catches those breaks.
- Can the coding agent write the tests for my vibe-coded app?
Yes, a coding agent can type the tests if you give it the steps and expected results. Tests the agent derives from its own code assert its own mistakes.
- What should I test first in a vibe-coded app?
Test the access rules first when a vibe-coded app stores customer records: request the data signed out and as a second account. Then test payments, if any, and then the main daily flow.