Skip to content
Scalpels home

Sign up now for $10 in credit on a new Laravel Cloud account.

Idea

Sort the test suite into gate, nightly, and delete

· Ed Grosvenor

This week is CI Week in the lab. Every idea, experiment, and stream we publish this week is about continuous integration, and this is the first of them.

Building applications with agents produces a lot of tests. Some of them tell us the application works. Many of them are something else. They are the agent's own scaffolding, written to organize its work and confirm that the step it just took did what it claimed. Both kinds land in the same directory, run in the same job, and cost the same minutes on every pull request.

Developers argue about which kind is worth having. There are people who think the only test worth writing is a unit test against a single class, and people who think the only one that counts is a browser driving the real application. We do not want to pick a side. We already have the tests. Agents wrote them, and there are a lot of them. The useful question is which of them to keep running and how often.

What a test has to earn

A test exists to make an application easy to change with confidence. That gives it two jobs. It should fail when behaviour a user can see breaks. It should also stay green through a refactor that rearranges the code without changing what it does. A test you have to edit every time the application changes is doing the opposite of its job. It charges for the change instead of checking it.

For the tests that do pass that bar, risk decides how often they run. Money, security, data loss, and tenant isolation are the places where a failure is not something to look at in the morning.

Three buckets

The first bucket runs on every pull request and blocks the merge. A failure here means the application is broken in a way somebody would notice and nothing should ship. We expect this to be the small bucket.

The second runs occasionally, nightly to start. A failure here means something changed and deserves a closer look, but the application is probably fine. This bucket only works if a red run opens an issue by itself. A nightly job that nobody is told about is a job that goes red in March and is discovered in June.

The third gets deleted. The clearest example is a test that asserts the exact words on a page when what it should assert is that the right template rendered with the right data. Reword the page and the test fails, though nothing a user can do has changed. The rest of this bucket has the same shape. It holds tests that read the application's own source, tests that script which methods a mock receives in which order, and assertions on exact row counts that have to be edited whenever a seeder grows.

No regex can decide this

The patterns are easy to find and easy to misread. assertDontSee($token) on a page that must not leak a credential is a security test, and one of the better ones we have. assertSee('Welcome back') is a copy assertion. To a grep they are the same family of call. Anything that classifies a suite by pattern alone will delete the first one.

So we want four signals, of which exactly one is decisive.

Static patterns are the cheap pass. They look for copy strings, source reads, reflection, mock choreography, and exact counts. They nominate candidates and settle nothing.

Churn coupling is also cheap and comes out of git. How often is a test file edited in the same commit as the application code it covers, and when it is, does the diff only ever change an expected string or a number? A test that gets adjusted every time somebody touches the feature is telling us what it really asserts.

Mutation testing is expensive and decisive. Break the behaviour on purpose and see which tests notice. A test has no unique kills when every mutant it catches is caught by something else as well. A test like that can be removed for free, and that is the proof we want behind a deletion rather than an argument about it. A test holding unique kills in code that moves money or checks authorization is a gate test by definition.

The fourth signal is per-test CI history, and we do not collect it. We want each test's timing and outcome over months, and for every failure, whether somebody then fixed the application or edited the test. That distinction is the difference between a catch and a false alarm. It is also the one piece of evidence that cannot be reconstructed after the fact. Instrumenting it is the first thing we have to build.

How it would apply

Pest groups carry the decision. The gate group runs on pull requests, the whole suite runs nightly, and a red nightly opens an issue automatically. Deletions go in batches a human can actually read, grouped by directory and by the reason they were nominated, each one backed by its mutation result.

Then there is the write side, which is the part that decides whether any of this holds. The same loop that produced four thousand tests will produce four thousand more. If nothing changes about how it writes them, we will be doing this again in two months. So the tooling has to come with instructions for the agents doing the building.

Our own suite is the subject

We measured this application's suite on main today. There are 4,005 test cases in 358 files. Of those files, 352 sit under Feature and 6 under Unit, so almost everything we have is a feature test. Around 950 of the assertions ask what a rendered page contains. Of those, 557 use plain assertSee or assertDontSee. There are 1,406 calls to toContain. Twenty test files read the application's own source code. There are no browser tests at all.

Those numbers are the reason we think this is worth tooling rather than an afternoon of judgement. We are starting with one area of the application where the risk tiers are obvious, and we will publish the experiment when it has run.

What we want to know

How much of the suite lands in each bucket? Our guess is that the gate is small and the delete pile is larger than anyone would like to admit, but a guess is all it is.

How much CI time does the gate actually save, measured in wall-clock on a real pull request rather than in test counts?

Do the cheap signals agree with mutation testing often enough to be trusted on their own? Mutation runs are slow and expensive. If static patterns and churn coupling predict the mutation verdict most of the time, this becomes something you can run on any repository in an afternoon. If they do not, every suite needs the expensive pass, and the tooling is worth much less.

And can it carry to an application we did not write? We know our own suite's habits because we watched them form. A client's suite, written by other people and other agents over years, is the real test of whether any of this generalizes. That is where we want to end up. We want tooling we run on our own applications and then take to the projects we are brought into. We also want a Scalpels guide for the teams who would rather do it themselves.

Support the Lab

The Lab runs on our supporters

Every idea here gets tried for real: deployed, broken, fixed and written up. Right now that is one person doing the work between everything else, and supporters are what buy the time. As support grows, it will also pay freelancers to run experiments of their own, so more of these ideas get tested, sooner.

Our supporters

  • Chill Vibes

    A curated collection of fun and interesting experiments and projects built with AI to entertain and educate.

    Supporter Visit
    Visit Chill Vibes
  • Fawnfox Fables

    Create and publish your own children's books. Stories built around the people you love.

    Supporter Visit
    Visit Fawnfox Fables
  • Sputnik Intelligence

    Spain

    We transcribe the podcasts and read the newsletters, and make them searchable. Set up alerts, or hand it all to your AI agent over MCP or API.

    Supporter Visit
    Visit Sputnik Intelligence
  • Team of Me

    Free resources for indie makers and people thinking about striking out on their own.

    Supporter Visit
    Visit Team of Me
  • Wicked Viral

    Earth

    Connecting builders with promoters who can help them take their products to market. Get your product listed today!

    Supporter Visit
    Visit Wicked Viral