Skip to content
Scalpels home

Scalpels is officially launching on October 31, 2026. Special early adopter pricing is available now.

← All documentation

Documentation

Assay

Troubleshooting

Updated

Most problems show up in one of two places: the Pipeline Health table on /assay/dashboard, and the response Assay gives the client's ShipEnvelope job in the reporting application.

Nothing arrives from an application

Work through these in order:

  1. The client is inert. If ASSAY_URL or ASSAY_TOKEN is missing in the reporting application, the client sends nothing and raises no error. Check both are set in the environment the application actually runs in, and that its config cache was rebuilt after you set them.
  2. laravel/ai is missing. Without laravel/ai 1.x installed, the client registers no listeners. Calls made with another AI library are not captured.
  3. No queue worker. Telemetry ships from queued ShipEnvelope jobs. Run a worker for ASSAY_QUEUE, or for the default queue if it is unset.
  4. Wrong URL. ASSAY_URL must be the full ingest endpoint, ending in /ingest.
  5. The application is new to Assay. It appears in Pipeline Health after its first batch has been processed by Assay's own queue.

If the application appears in Pipeline Health but a rising transport drop count sits next to it, the client is failing to deliver and giving up after its retry window (24 hours by default).

Assay rejects a batch

Response Meaning What to do
401 Unauthorized. The token is not a live assay.ingest installation credential. Check ASSAY_TOKEN is the secret for an installation credential issued under Settings → Installation credentials, and that it has not been revoked or rotated. MCP credentials cannot submit telemetry.
413 with payload_too_large, too_many_records or too_many_sources The batch is over one of Assay's ingest limits. The client splits the batch and retries automatically. A single record that is still too large is dropped and counted as a transport drop. Keep ASSAY_BATCH_SIZE at or below Assay's ASSAY_INGEST_MAX_RECORDS (500 by default).
422 Envelope vN is newer than this Assay server (max v1). Upgrade your Assay server. The client speaks a newer telemetry format than your Assay deployment. Update your Assay deployment to a release that supports it. GET /capabilities on your deployment lists the format versions it accepts.
422 Envelope JSON is malformed. or Envelope is invalid. The body did not match the format. Make sure the client package and Assay are both current releases.

202 Accepted. means Assay queued the batch; it is written to the database when Assay's queue processes it.

Records are being dropped by the filter

A rising hook drop count means your PayloadFilter returned null or threw for Assay payloads. Both drop the record rather than send it unfiltered. Check that the filter returns payloads for other products unchanged and returns a valid OutboundPayload for the Assay payloads it keeps. See Capturing content.

Telemetry with a subject is refused

Assay will not process subject-bearing telemetry without a valid erasure key, because it cannot check the subject against past erasures. Scalpels writes ASSAY_ERASURE_KEY at install. If it has been removed, changed to something shorter than 32 bytes, or is not valid base64 after its base64: prefix, restore the original value. Do not generate a new one: a new key orphans every earlier erasure.

Runs show as incomplete

A run that has not sent its final record 30 minutes after Assay first received it is marked incomplete (or failed_or_lost for non-agent operations). Common causes are a request or worker that died mid-run, or a client batch that was dropped. Pipeline Health counts incomplete agent runs per application.

Content is missing for a run

  • The application is in usage mode, or the run was not sampled in full mode and did not fail.
  • The run was a standalone non-agent operation, which failure capture does not cover.
  • The failure buffer overflowed: the run shows failure_capture as truncated.
  • The run content passed its retention period (30 days by default).
  • Content sent for a record that never arrived is held for the same 30-minute window, then dropped. Pipeline Health counts runs whose content is incomplete.

A screen returns 401 or 403

  • 401 means you are not signed in. Sign in at /bfc/login, or through Scalpels.
  • 403 on /assay/datasets, /assay/runs/... or an export means you do not have content access. Ask an Owner, or an Admin with content access, to grant it at /assay/access.
  • 403 on /assay/access itself means you are a Member, or an Admin whose content access has been denied. Only Owners and content-capable Admins manage overrides.

A run cannot be added to a dataset

Assay refuses runs whose captured content is incomplete, and runs whose subject has since been erased. Choose another run.

An export download fails

  • 410 Gone: the request is more than 15 minutes old. Request the export again.
  • 403: the download was opened by a different person or credential than the one that requested it, or the requester has lost content access since. Export URLs cannot be shared.

An MCP tool is missing or refused

  • A tool that is not listed needs an ability the credential lacks. Content tools need assay.content; usage tools need assay.usage.
  • A write or destructive tool called on /mcp is refused. Flagging, labelling, dataset changes and delete_subject are only served at /mcp/destructive.
  • delete_subject makes no changes on its first call. Call it again with the returned confirmation in confirm.

Reapplying erasures after a restore stops early

assay:erasures:reapply prints a --cursor= value when there are more entries than --limit, or when an entry failed. Run it again with that cursor. The erasure journal could not be read. means the command could not reach the bucket the journal lives in; run it inside the Assay environment, where the bucket is attached.