Curation and datasets

Updated

Curation turns production runs into datasets you keep. All of it needs content access: Owners always have it, Admins have it by default, and Members only when an Owner or Admin grants it. See Access, retention and erasure.

The screens are deliberately plain. You find run IDs on the dashboard (the run_id column of Top Runs, or a run tree) and paste them into the curation forms.

Flag a run

Open /assay/runs/{run}/curate with the run's ID. The form has:

Labels, free text. The form takes one label; through the JSON route or the MCP flag_run tool a run can carry up to 25 labels of up to 100 characters each.
Rating: pass, fail, or 1 to 5, or not rated.
Note: up to 10,000 characters.

Saving replaces the run's labels, rating and note with what you submitted. Flags are copied into a dataset item when the run is added, so set them first.

Create a dataset

Open /assay/datasets. Enter a name and choose a retention:

Operator default: items expire 365 days after they are added, unless your installation's default has been changed (see Configuration).
Custom days: any number from 1 to 36,500.
No expiry: items stay until they are removed or erased.

The list on the same page shows every dataset with its item count. Select one to open it.

Add runs

On a dataset's page, paste a run ID and choose Add immutable snapshot. Assay copies the run as it is now, including its instructions, input messages, tool schemas, observed output and your labels, rating and note, into a new dataset item. Later changes to the run or its flags do not change the item.

Assay refuses to add:

a run whose captured content is incomplete;
a run whose subject has since been erased.

Add only runs captured in full mode if you need their content. A usage-only run has no instructions, messages or output to copy.

Export a dataset

On a dataset's page, choose Request JSONL export. Assay records the request and returns its download URL, /assay/exports/{export}. Open that URL to download assay-dataset-v1.jsonl.

The download is rendered from current data each time; Assay never stores export files. Three conditions apply:

the download must be made within 15 minutes of the request;
it must be made by the same person or credential that requested it;
that person must still have content access when they download.

The URL is not a shareable link. Anyone else who opens it is refused. Because exports are rendered live, a subject erased after the request is already missing from the file.

The export format

The file is newline-delimited JSON (application/x-ndjson), one item per line. Every line has schema set to assay.dataset-item.v1, an item_id, and these fields from the snapshot:

Field	Contents
`agent_class`, `provider`, `requested_model`	What ran
`instructions`, `input_messages`, `tool_schemas`	The inputs, with messages in order
`observed_output`	`text`, `structured` and `tool_calls`
`labels`, `rating`, `note`	Your flags at the time the item was added
`source_client_version`, `deploy`, `capture`, `sampled`, `hook_applied`	Where the run came from and how it was captured
`replay_fidelity`	`complete`, or `partial` when the client reported that some replay inputs were omitted

replay_fidelity describes the data, not a feature: Assay does not replay runs itself. The format is defined in docs/dataset-export-v1.md in the Assay repository.

How long items last

An item's retention starts when it is added, not when the run happened, so a dataset item can outlive the run content it came from. While an item is live, Assay keeps the usage metadata of its source run and that run's ancestors so a later erasure can still find them. It does not keep their messages or content beyond normal content retention. The /assay/datasets page explains this above the create form.