Guide
Cut Laravel Nightwatch query volume without losing the trace
· Ed Grosvenor
Last updated
One request is not one Nightwatch event. It is a request event plus every query, cache operation, outgoing request, log and other event inside it. A job that runs one query for each item in a loop can therefore produce thousands of query events while producing only one job attempt.
That matters because Nightwatch treats each request, exception, query, log and other observation as an individual event. Plans include an event allowance, and additional usage is priced by event. Sampling requests helps, but a single sampled-in request or job can still carry a very large query trace.
The first fix is usually the loop that produces the queries.
Find the noise before hiding it
Start in Nightwatch's Queries panel. The query list shows each SQL shape, its connection, total calls, average duration and p95 duration. Open an individual execution to see its parent request or job and the rest of that trace.
Look for a query whose call count is much larger than its parent count. Then check the code around the parent for an N+1, a missing eager load, repeated existence checks or writes that could be batched. Fixing that removes work from the application and from Nightwatch. A filter only removes the telemetry.
Consider a vulnerability-matching job that emitted about six million queries a day because it queried once per candidate inside a loop. The right order was:
- replace the N+1 with a bulk query;
- confirm that the job still produced the right matches;
- decide whether the smaller set of routine matching queries was worth retaining;
- exclude those remaining queries by their application source file while keeping the job attempt.
Do not start at step four. Hiding the queries from an N+1 leaves the application doing the same work.
Reject individual queries by source file
Register query filters during application boot. A callback returns true to reject a query and false to keep it:
use Laravel\Nightwatch\Facades\Nightwatch;
use Laravel\Nightwatch\Records\Query;
public function boot(): void
{
Nightwatch::rejectQueries(
static fn (Query $query): bool =>
$query->file === 'app/Services/MatchEngine.php',
);
}
rejectQueries() is available from Nightwatch v1.9.0. Its Query record gives the callback these useful fields:
| Field | Match on | Watch out for |
|---|---|---|
$query->file |
The application file that originated the query | Check the value Nightwatch actually reports before deploying the rule. Indirect queries may point at a model, listener or another service. |
$query->line |
A particular call site | Line numbers move during ordinary edits. Use them to narrow a file rule, not as the first choice. |
$query->sql |
SQL shape or table name | Bindings are absent, and quoting differs between MySQL and PostgreSQL. |
$query->connection |
Laravel connection name | This is the configured connection name, not necessarily the database driver. |
$query->duration |
Query duration in microseconds | A duration filter hides fast queries now, but those queries can become slow later. |
Prefer file over SQL text when one application service owns the noise. It survives SQL formatting and database-dialect changes. Run the workload in a non-production environment first and inspect the query's reported file; do not assume it is the file you had in mind.
Add a connection condition when the same service legitimately uses more than one database:
Nightwatch::rejectQueries(
static fn (Query $query): bool =>
$query->connection === 'analytics'
&& $query->file === 'app/Services/MatchEngine.php',
);
What rejection changes
A rejected query is not sent to Nightwatch, and it does not increment the parent request, command or job attempt's query count. The parent remains, but its queries value counts only the retained queries, not every query the application ran.
Rejection is precise, and it still costs something. Nightwatch takes the query backtrace and builds the Query record before it calls your filter. If the application still executes millions of queries, it still pays that per-query client-side work even though it saves event allowance, transport and storage.
This combination keeps a matching job at 100% while dropping only its queries:
use Illuminate\Contracts\Queue\ShouldQueue;
use Laravel\Nightwatch\Facades\Nightwatch;
final class MatchVulnerabilities implements ShouldQueue
{
public function handle(MatchEngine $matcher): void
{
Nightwatch::sample(); // Keep this job attempt at 100%.
$matcher->run(); // Queries from MatchEngine.php are rejected by the boot filter.
}
}
You retain the job's status, duration, memory use, exceptions and other allowed child events. You lose its rejected query details and those queries from the job's count.
Ignore a whole region when callback cost is too high
If all telemetry inside one database-heavy call is expendable, pause Nightwatch around that call:
$matches = Nightwatch::ignore(
fn () => $matcher->run(),
);
This is cheaper than rejectQueries(): Nightwatch sees that capture is paused before it takes each query backtrace or constructs each query record. It is also much broader. Inside the callback, Nightwatch drops:
- queries;
- cache events;
- outgoing requests;
- mail;
- notifications;
- queued-job events.
Logs and exceptions are still captured. A job dispatched inside the callback also carries a sampled-out context, so its later attempt is absent. Keep dispatches outside the ignored block when you still need those jobs in Nightwatch.
Nightwatch::ignore() restores the previous state even when the callback throws. If a callback is impractical, Nightwatch::pause() and Nightwatch::resume() expose the same pause, but guard them with finally so a thrown exception cannot leave a long-lived worker paused:
Nightwatch::pause();
try {
$matcher->run();
} finally {
Nightwatch::resume();
}
Combine filtering with execution sampling
Query exclusion decides which child events disappear from a trace. Sampling decides which whole executions become traces at all. Use both.
Nightwatch exposes global rates from 0.0 to 1.0 for its primary execution contexts:
NIGHTWATCH_REQUEST_SAMPLE_RATE=0.1
NIGHTWATCH_COMMAND_SAMPLE_RATE=1.0
NIGHTWATCH_EXCEPTION_SAMPLE_RATE=1.0
NIGHTWATCH_SCHEDULED_TASK_SAMPLE_RATE=1.0
The same values live under sampling.requests, sampling.commands, sampling.exceptions and sampling.scheduled_tasks in config/nightwatch.php. Nightwatch's sampling guide recommends reducing request sampling first.
Individual executions can also make their own sampling decision:
- use route sampling middleware for requests;
- call
Nightwatch::sample($rate)orNightwatch::dontSample()when code can make the decision; - attach
Laravel\Nightwatch\Console\Sample::never()to a scheduled task that should never be traced; - call
Nightwatch::sample()at the start of a specific job when it must be retained regardless of the parent execution's decision.
Jobs otherwise inherit the sampling decision of the request, command, scheduled task or job that dispatched them. There is no global job sample-rate setting.
There is one exception. If an execution was sampled out and then reports an exception, the exception sample rate gets another decision. With the default exception rate of 1.0, the exception can turn collection back on so Nightwatch can send the failure and its context. dontSample() is therefore not an unconditional promise that a failing execution will stay invisible.
Whole-execution sampling and query rejection stack cleanly. You can sample 10% of ordinary requests, retain one important matching job at 100%, and reject that job's routine queries by source file. The executions you care about remain visible without every child query consuming an event.
Turn off all query telemetry when that is honest
NIGHTWATCH_IGNORE_QUERIES=true disables query capture application-wide. It returns before the backtrace and record construction, so it is the cheapest option when you need no query visibility at all.
Other event types remain available, but every parent query count becomes zero. Use it for an application whose database is observed elsewhere, during an emergency volume spike, or when query telemetry has no operational value. Do not use it merely because one workload is noisy; a source-file filter is safer for that.
Query-only downsampling is possible, but the counts lie
Nightwatch's query documentation is explicit: there is no native query sample rate. A random rejection callback can approximate one:
Nightwatch::rejectQueries(function (Query $query): bool {
if ($query->file !== 'app/Services/MatchEngine.php') {
return false;
}
return random_int(1, 100) !== 1; // Retain about 1%.
});
That callback runs once per query execution. Neither the retained query nor its parent carries the 1% weight, so Nightwatch cannot multiply the counts back up. The UI shows an unlabelled undercount, the parent's query count is also the retained count, and rare slow queries may disappear from the sample.
Use this only when approximate shapes and durations are useful but truthful counts are not. If the queries have no value, reject all of them. If their counts matter, keep them or sample the whole parent execution instead.
Two tempting hooks do not solve this
IngestingEvents is a batch gate rather than a record filter. Nightwatch v1.29 added this event for rate limiting before transmission, and v1.30 added eventCount(). A listener can return false, but that cancels the entire pending ingest. It cannot return a modified record list. A noisy query usually shares its batch with its parent and useful sibling events, so this is the wrong layer for selective exclusion.
Reordering Laravel's QueryExecuted listeners is brittle. In theory, an earlier listener can return false and stop Nightwatch's listener from running. In practice, Laravel has no listener-priority API here, package-provider order decides who is first, and returning false also blocks every later query listener. Do not rewrite or reorder the event listener list to avoid a supported Nightwatch API.
Choose the smallest loss
| Goal | Mechanism | What you lose |
|---|---|---|
| Remove one workload's queries, keep its parent | Nightwatch::rejectQueries() by file |
Rejected query records and an accurate parent query count; per-query backtrace cost remains. |
| Remove all observed activity inside one code region cheaply | Nightwatch::ignore() |
Queries, cache, outgoing request, mail, notification and queued-job events in that block. |
| Remove a whole normal execution | Nightwatch::dontSample() or an execution sample rate |
The parent and all child visibility; an exception can sample it back in. |
| Never capture any query in the application | NIGHTWATCH_IGNORE_QUERIES=true |
All query records and all parent query counts. |
| Keep only a random fraction of selected queries | Random rejectQueries() callback |
Truthful counts, explicit sample weighting and some rare outliers. |
| Put a hard ceiling on total ingestion | IngestingEvents rate-limit listener |
Whole batches after the ceiling, including unrelated events. |
The same controls work with Hone
Hone is an open-source, self-hosted, Nightwatch-compatible receiver. It receives exactly what the Nightwatch client decides to send, so rejectQueries(), ignore(), whole-execution sampling and NIGHTWATCH_IGNORE_QUERIES apply in the same way. Hone does not add another sampling layer of its own.
This guide was verified against the latest Nightwatch release, v1.30.2, including a real queued-job test that retained the job attempt while rejecting a query by source file. rejectQueries() requires v1.9.0 or newer; IngestingEvents requires v1.29.0, and its eventCount() helper requires v1.30.0.
Changelog
- Published.