Experiment
The suite was green and two test files had stopped reporting
· Ed Grosvenor
It is CI Week in the lab, and this one came out of the week's own work rather than a plan. While measuring our suite for the triage idea we noticed that the serial run and the parallel run did not agree on how many tests we have. Two numbers that should have matched did not, so we stopped to find out why.
The gap was 196 test cases out of about six thousand. Both runs exited 0, no file was missing from the repository, and git log showed the relevant tests being written and passing months earlier.
What was actually wrong
Two test files in one feature directory were contributing no test cases of their own. Three other test files in that directory had require_once __DIR__.'/…Test.php' at the top, eight lines in total. They were left over from before we moved those files' shared helpers into tests/Pest.php and tests/Helpers. The lines did nothing useful and nobody had looked at them since.
The 67 test cases in those two files were still executing. They were being credited to whichever file had required them, which is why the totals looked sane and why the suite stayed green. Run on their own, the two files pass with 67 cases and 662 assertions. Run as part of the suite, they reported nothing. Neither CI lane had any record that they existed, and nothing in six thousand lines of output said so.
Under --parallel it goes wrong in the other direction. Each worker loads only its own file, so each requiring file pulls the pair in again and the same cases run two or three times per run. Those duplicate executions are the 196, and they are the only reason we saw anything at all.
Finding it from outside the suite
The suite cannot report this, because from the suite's point of view nothing has gone wrong. So we asked a different question. Which *Test.php files on disk appear in neither lane's JUnit log?
./vendor/bin/pest --parallel --exclude-group=postgres --log-junit build/junit/parallel.xml
./vendor/bin/pest --group=postgres --fail-on-skipped --log-junit build/junit/postgres.xml
The parallel lane recorded cases for 404 distinct files and the Postgres lane for 15. There are 421 test files on disk. Fifteen of the parallel lane's absences are the Postgres group, and all fifteen turn up in the other log. The remaining two were the answer, and the whole investigation after that was explaining why those two and not any others.
The answer is file ordering. A requiring file only shadows a file that loads after it, and all three requirers sorted ahead of the pair alphabetically. Two other files required by one of the three sorted earlier, got loaded as themselves first, and were fine. Change nothing but a filename, and this bug appears or disappears.
Whose bug it is
Pest's test-suite loader, in both halves.
Pest\Support\Backtrace::testFile() decides which file a test() call belongs to by walking the backtrace until it reaches the loader and keeping the outermost frame below it. A test() call inside a required file therefore resolves to the requiring file, and its TestCaseFactory is stored under that key.
When the loader then reaches the required file itself, include_once is a no-op because the file is already in get_included_files(). TestRepository::makeIfNeeded() returns early because nothing is keyed by that filename, no new class is declared, and TestSuiteLoader::load() falls through to a method that hands back a ReflectionClass(IgnorableTestCase::class). That is the whole failure. Pest silently returns an ignorable case where PHPUnit's own loader would throw ClassCannotBeFoundException.
We reproduced it on a bare install of Pest 4.7.8 with two files and no framework:
<?php
// tests/AlphaTest.php
require_once __DIR__.'/BetaTest.php';
test('alpha one', function (): void {
expect(true)->toBeTrue();
});
<?php
// tests/BetaTest.php
test('beta one', function (): void {
expect(true)->toBeTrue();
});
test('beta two', function (): void {
expect(true)->toBeTrue();
});
PASS Tests\AlphaTest
✓ beta one
✓ beta two
✓ alpha one
Tests: 3 passed (3 assertions)
Every file attribute in the JUnit log reads tests/AlphaTest.php. Add a second file that also requires BetaTest.php, and the serial run reports 4 cases while --parallel reports 6. That is our 196 in miniature. Plain class-based PHPUnit 12.5.33, in a project with no Pest installed, attributes both files correctly. Class-based tests inside Pest keep their own file too. It is the functional API only.
There is one more consequence we found after filing, and it does lose coverage outright. A pest()->group('x') call in the required file lands on the requiring file's factory, so it tags that file's own tests as well. With pest()->group('beta') in BetaTest.php, pest --exclude-group=beta reports No tests found. alpha one is gone too, and it was never in the group. Our own Postgres lane is selected exactly that way.
Nobody had reported this upstream, so we filed it as pestphp/pest#1984. Supporting require_once between test files may well not be worth doing. A file Pest was told to load that produces no test case and declares no class could just say so.
The fix and the guard
The fix was deleting eight lines across three files. That came to 11 deletions, and we moved nothing. The serial and parallel counts now agree, the two files report under their own names, and the lane does 196 fewer case executions per run.
That took an afternoon and proves nothing about the next one. Our hard gate could not see this class of problem at all. The signal we noticed by accident was two numbers that should have matched and did not, and a script could have checked them the whole time.
So bin/assert-test-files-ran.php collects every *Test.php under the roots you give it, collects every <testcase file=…> from the JUnit logs you give it, and fails naming each file that appears in no log. It is framework-free and takes every path as an argument, so it works in any PHP repository that can write a JUnit log per lane. It also refuses to pass when it has nothing to say. A missing log, a log with no test cases in it, roots that match no file, or an allowlist entry naming a file that is not on disk each count as a failure rather than an empty diff.
<?php
/*
| Fails when a test file on disk had no test case execute in any of the JUnit
| logs it is given. CI splits the suite into lanes, and a file can stop being
| collected without anything going red: Pest's test-suite loader include_once's
| each file, so a file another test file has already required registers no test
| cases of its own and is quietly swapped for an ignorable case. The suite stays
| green and the file's coverage is gone.
|
| Deliberately framework-free and path-agnostic so it can be dropped into any
| PHP repository that can produce a JUnit log per lane:
|
| php bin/assert-test-files-ran.php \
| --junit=build/junit/parallel.xml --junit=build/junit/postgres.xml \
| --tests=tests --tests=packages/forge-client/tests
|
| Options:
| --junit=PATH A JUnit log. Repeat it once per lane; a file only has to
| appear in one of them. Required.
| --tests=PATH A directory to scan for test files. Repeat as needed.
| Defaults to "tests".
| --suffix=NAME Filename suffix that marks a test file. Default "Test.php".
| --root=PATH The directory the JUnit paths are relative to. Defaults to
| the working directory.
| --allow=PATH A test file that is allowed to run in no lane, relative to
| the root. Repeat as needed. Every entry needs a reason in
| the pull request that adds it.
|
| A file counts as having run when a <testcase> carries it, whatever the
| outcome; a file whose every case is skipped still passes here. Keep the lane
| that can skip under --fail-on-skipped, which is what proves it ran.
*/
$options = [
'junit' => [],
'tests' => [],
'allow' => [],
'suffix' => 'Test.php',
'root' => getcwd(),
];
foreach (array_slice($argv, 1) as $argument) {
if (! preg_match('/^--(junit|tests|allow|suffix|root)=(.+)$/', $argument, $matches)) {
failWith("Unrecognised argument [{$argument}].");
}
if (is_array($options[$matches[1]])) {
$options[$matches[1]][] = $matches[2];
continue;
}
$options[$matches[1]] = $matches[2];
}
$root = realpath($options['root']);
if ($root === false) {
failWith("The root [{$options['root']}] does not exist.");
}
if ($options['junit'] === []) {
failWith('Pass at least one --junit=PATH.');
}
if ($options['tests'] === []) {
$options['tests'] = ['tests'];
}
$onDisk = [];
foreach ($options['tests'] as $directory) {
$path = absolutePath($root, $directory);
if (! is_dir($path)) {
failWith("The test directory [{$directory}] does not exist.");
}
$iterator = new RecursiveIteratorIterator(
new RecursiveDirectoryIterator($path, FilesystemIterator::SKIP_DOTS),
);
foreach ($iterator as $file) {
if (! $file->isFile() || ! str_ends_with($file->getFilename(), $options['suffix'])) {
continue;
}
$onDisk[relativePath($root, (string) $file->getRealPath())] = true;
}
}
if ($onDisk === []) {
failWith(sprintf(
'Found no file ending in [%s] under [%s]; the paths this guard was given cannot be right.',
$options['suffix'],
implode(', ', $options['tests']),
));
}
$ran = [];
foreach ($options['junit'] as $log) {
$path = absolutePath($root, $log);
if (! is_file($path)) {
failWith("The JUnit log [{$log}] does not exist. The lane that writes it did not get as far as writing it.");
}
$document = new DOMDocument;
if (@$document->load($path) === false || $document->documentElement === null) {
failWith("The JUnit log [{$log}] is not readable XML.");
}
$cases = 0;
foreach ((new DOMXPath($document))->query('//testcase[@file]') as $case) {
$cases++;
// Pest writes "<file>::<test name>" here, PHPUnit writes the bare path.
$file = explode('::', $case->getAttribute('file'))[0];
$ran[relativePath($root, $file)] = true;
}
if ($cases === 0) {
failWith("The JUnit log [{$log}] records no test case at all.");
}
}
$missing = array_values(array_diff(
array_keys($onDisk),
array_keys($ran),
$options['allow'],
));
$unused = array_values(array_diff($options['allow'], array_keys($onDisk)));
if ($unused !== []) {
failWith(sprintf(
"These --allow entries do not name a test file on disk:\n %s",
implode("\n ", $unused),
));
}
if ($missing !== []) {
sort($missing);
failWith(sprintf(
"No test case ran for %d of %d test files:\n %s\n\n%s",
count($missing),
count($onDisk),
implode("\n ", $missing),
"A test file whose cases run in no lane is dead coverage. The usual reason is\n"
."that another test file requires it, which leaves Pest nothing to collect\n"
.'for it; put the shared helpers in the shared test helpers instead.',
));
}
printf(
"Every one of the %d test files ran at least one case across %d lane(s).\n",
count($onDisk),
count($options['junit']),
);
exit(0);
function absolutePath(string $root, string $path): string
{
return str_starts_with($path, DIRECTORY_SEPARATOR) ? $path : $root.DIRECTORY_SEPARATOR.$path;
}
function relativePath(string $root, string $path): string
{
$path = str_starts_with($path, DIRECTORY_SEPARATOR) ? $path : $root.DIRECTORY_SEPARATOR.$path;
$real = realpath($path);
$path = $real === false ? $path : $real;
if (str_starts_with($path, $root.DIRECTORY_SEPARATOR)) {
$path = substr($path, strlen($root) + 1);
}
return str_replace(DIRECTORY_SEPARATOR, '/', $path);
}
function failWith(string $message): never
{
fwrite(STDERR, $message.PHP_EOL);
exit(1);
}
We added --log-junit to both of our lanes, and a step after them runs the script over both logs and all three test roots:
- name: Prove every test file ran in one of the lanes
run: |
php bin/assert-test-files-ran.php \
--junit=build/junit/parallel.xml \
--junit=build/junit/postgres.xml \
--tests=tests \
--tests=packages/laravel-cloud-client/tests \
--tests=packages/forge-client/tests
To adopt it, add --log-junit to each lane you already run, drop the script in, and point it at your test directories. Pest creates the log directory itself, the logs go somewhere gitignored, and the allowlist stays empty until a file earns an entry with a reason in the pull request that adds it. When we put the eight lines back on a branch, the suite passes 6,123 tests, and the guard says:
No test case ran for 2 of 422 test files:
tests/Feature/ManagedAuth/TransitionAcknowledgeTest.php
tests/Feature/ManagedAuth/TransitionStageTest.php
What we do not know yet
How common this is. Our suite is largely agent-written, and a require_once between two test files is exactly the kind of thing an agent adds to make a helper available before someone moves that helper somewhere shared. We have one instance in one repository, and we found it by accident. We have no idea whether the suites in our other applications have any.
Whether a count comparison is worth gating on as well. Serial total against the union of the lanes should match, and ours was off by 196 for months. It is a cheaper check than the JUnit diff, but it needs a serial run of the whole suite, which is the thing the lanes exist to avoid.