Skip to content
Scalpels home

Sign up now for $10 in credit on a new Laravel Cloud account.

Idea

Clank Overflow would let everyone's agents compare notes

· Ed Grosvenor

Developers used to learn what a tool really does from each other. You hit something strange, searched for it, and found a Stack Overflow answer from someone who hit it two years earlier. Then you added a comment saying it still happens in the new version.

Our agents do most of that discovering now, and they do it alone. When one of ours finds that a Laravel Cloud API field documented in the OpenAPI spec doesn't come back from the live API, it writes that down in its memory. The next session on that project benefits, and nobody else does. Somewhere, someone else's agent is spending the same afternoon on the same field. Its notes are on someone else's machine.

We've done a fair amount of thinking and experimenting with how agents inside one organization can talk to each other and hand off work. We have never looked at the other half. That is unrelated agents, working for unrelated developers, sharing what they've learned about the tools everyone uses. We've been calling it Clank Overflow.

We're already writing the content

Before deciding whether this was worth building, we wanted to know how much of it already existed. So we had agents go through our own notes from the last couple of months. That covered project logs, research write-ups, agent memories, and the playbooks we keep for ourselves. They were looking for anything about a public tool that was surprising, undocumented, version-specific, or different from what the docs say.

They found about 270 candidates. After merging the duplicates, about 180 were distinct. None of them were written to be shared. They're just what agents leave behind while working.

  • Requests to Laravel Cloud's API sent with Accept: application/json come back 401 Unauthenticated, which looks exactly like a bad token. The API wants application/vnd.api+json.
  • For a while, Laravel Cloud's OpenAPI spec documented a schemas relationship on database clusters after the live API had renamed it to databases. Code written against the spec got null and looked correct.
  • Laravel Cloud's network is IPv6-only. A server bound to 0.0.0.0 fails its health checks while its own logs look clean. Bind to [::].
  • cloud tinker doesn't exist in Laravel Cloud's CLI before v0.6.0. An agent following a runbook written for a newer CLI just gets "Too many arguments."
  • Cloud injects APP_URL into an environment, set to your verified custom domain, but not APP_ENV. If you never set APP_ENV, your staging environment thinks it's production.
  • Laravel Forge retired the legacy /api/v1 on August 31. Its API tokens are signed JWTs well over a thousand characters long, so a max:500 validation rule rejects real tokens.
  • In Laravel itself, php artisan serve passes only a fixed list of environment variables through to its worker, so an exported DB_DATABASE is silently replaced by the one in .env. On SQLite, lockForUpdate() compiles to nothing, so a race-condition test passes whether your lock is there or not.
  • Run in the background without </dev/null, OpenAI's codex exec waits for more input forever. That looks exactly like a slow review.
  • ffmpeg 9 rejects -vsync. It's -fps_mode now. And ffmpeg -h encoder=<anything> exits 0 for encoders that don't exist.

About half of what we found is about Laravel Cloud. That says more about where we spend our time than about Cloud. It's a young, fast-moving platform, and fast-moving platforms are exactly where everyone's agents keep rediscovering the same things. The rest is spread across Laravel packages, GitHub and its CLI, Cloudflare, NativePHP, Homebrew, Packagist, YouTube's APIs, ffmpeg, and the agent tools themselves.

Two patterns stood out

The most common kind of finding, by a wide margin, was a tool that reports success and didn't do the thing:

  • A CLI command reports success while carrying a non-zero exit code.
  • An update command exits 0 and leaves the setting unchanged.
  • An API PATCH returns 200 and ignores the write.
  • A "watch the checks" command exits 0 on a failing check.
  • A mutation-testing filter matches nothing and reports a pass.

Agents trust exit codes and status fields far more than a suspicious human would, so this is where they lose the most time.

Our own notes also contradict each other. In several places, an early note says one thing and a later note says the opposite:

  • One note says a managed queue's visibility timeout tops out at 90 seconds. A later one found 180 seconds on a real queue.
  • One note says a CLI flag silently does nothing. A later one says the write always worked and only the read-back was broken, and that it was fixed two versions later.
  • An early experiment concluded that scale-to-zero didn't work. A later one found that it worked fine. The measurement just couldn't see it.

In every case, the later note was the one someone checked more carefully. That's the strongest argument for this idea we found, and it shapes the design. Whatever Clank Overflow stores has to carry when it was last confirmed, and by whom.

What it would look like

The unit is a claim

Stack Overflow is built around questions, and that's the wrong shape for agents. An agent session is short. A question asked at two in the afternoon gets answered after the session that needed it is over, and the next session won't remember asking.

The useful unit is a claim about a public thing, with evidence:

  • The subject: a package at a version, an API endpoint at a spec revision, or a CLI at a release.
  • What was observed, and what the documentation says instead, if anything.
  • Evidence, such as the shape of a request and response, or a line of source at a tag.
  • A probe that lets someone else check it.
  • When it was first seen, and when it was last confirmed.

That makes Clank Overflow look less like a forum and more like a security advisory feed, but for behavior instead of vulnerabilities. Reading from it should work like composer audit. At the start of a session, your agent reads your lockfile and the services your project talks to, and gets the known surprises for the exact versions you're on. It finds out before it hits them, instead of halfway through a wasted hour.

Writing to it should cost almost nothing, because agents are already writing these notes. Our memory files are full of entries like "this CLI version lacks that command". Those are already claims. Contributing is mostly a matter of taking the notes about public tools, stripping them of anything about the project they came from, and sending them on.

Probes are what make it trustworthy

Upvotes can be faked and confident prose can be wrong, as our own notes show. A claim that nine unrelated developers' agents reproduced in the last two weeks is much harder to fake.

Every claim should come with a probe, a way for another agent to check it. Our notes suggest there are three kinds, and the safest kind is also the most common:

  1. Read the source or the spec at a pinned version. Examples: "Line 214 of this file at this tag passes only these variables through", or "this field is absent from the published spec." About a third of our findings can be checked this way. It's deterministic, needs no credentials, and is safe to run anywhere.
  2. Make a read-only call against the agent's own account. For example, echo request headers from a test app, or read an environment's metrics and see what comes back.
  3. Make a write against throwaway resources. Create a scratch environment, flip a setting, read it back, tear it down. It's slower, sometimes costs money, and should always be opt-in.

A probe can't be a script from a stranger that your agent runs blindly. That's the most obvious way to turn this into a supply-chain attack. Probes should be declarative instead: a request template, a command from a known-safe list, or a file and line at a tag. The agent running one should be able to see exactly what it will do before it does it.

Probes also catch the thing nobody bothers to report, which is a fix. When a vendor fixes a bug, the next confirmation fails, and the claim can be marked resolved, with the version that fixed it. "This was fixed in 0.5.2" is as useful as the original report.

The reputation belongs to a person

Identity is a GitHub account. Agents do the work, but the reputation belongs to the human whose agent did it. That human is the one who approved the report, and the one who answers for it if it's wrong.

The leaderboard should reward the things that make the commons better:

  • Original reports that hold up.
  • Confirmations from unrelated accounts.
  • Refutations.
  • Adding a probe to a claim that didn't have one.
  • Marking a claim superseded when the facts change.

It should not reward restating the documentation. Some of our own "findings" turned out to be documented behavior we just hadn't read, so every new claim needs a novelty check against the official docs.

Nothing leaves your machine unchecked

The biggest risk to an individual developer is leaking something. An agent working inside a client project knows client names, internal hostnames, resource IDs, and API keys. Left to post freely, it will eventually post some of them.

The structure of a claim does most of the protecting. A claim must be about a public, versioned thing. That's why free-text fields are where trouble gets in, and why they should be kept small.

A claim is checked by a script, by a classifier, and by a person before it is published.

  • Deterministic checks for secret-shaped strings, hostnames that don't belong to the subject's vendor, resource IDs, and cost figures. These were exactly the things that made a handful of our own notes risky to share.
  • A cheap classifier that scores how likely the claim is to contain project or client information. We already use TypeSafe's Jev on this site to moderate the names people submit as Community supporters, and it fits this job too. We'd ask it two separate questions. One is whether the claim contains project or client information. The other is whether it contains a secret. That way each one can have its own threshold.
  • A human approval, at least to start. The report is drafted and queued locally, and its owner approves it before it goes anywhere. As someone's reports keep holding up, that can loosen.

Reading is the other half of the risk

Your agent would be reading text written by strangers' agents, which is exactly how prompt injection works. Everything in Clank Overflow is data, never instructions. Claims describe behavior. They never tell your agent to install a package or run a command, and the structure shouldn't leave room for them to try.

Anything that looks like a security vulnerability doesn't belong in a public commons at all. A webhook that accepts unsigned requests under some configuration, or a command that prints secrets into a log, should go to the vendor's disclosure process first.

Vendors might want this too

Vendors spend real money trying to collect feedback like a stream of independently confirmed reports. Each one might say "your published spec and your live API disagree here" and link to the upstream issue where one exists. Several of our findings already point at open issues and pull requests in the tools they're about. If vendors could subscribe to the claims about their own products, the commons would have a reason to exist beyond goodwill.

What we expect to be hard

The first thing is getting agents to look at the right moment. Reading at the start of a session is easy to arrange. The more valuable moment is the surprise in the middle, when a call returns something the agent didn't expect. Whether an agent thinks to check before it starts guessing is the part we're least sure about.

Keeping the leaderboard honest is the next. Reputation attracts gaming. Confirmations only mean something if the accounts confirming are really independent, and a GitHub account is cheap.

Claims about fast-moving tools go stale. A claim nobody has confirmed in a month should fade, but a claim nobody has probed because nobody uses that tool anymore should probably just disappear.

Then there is who runs the probes. Write probes cost money and leave resources behind. Someone has to pay for that, and someone has to make sure the cleanup happens.

What success looks like

We'd start with a small version, which is an MCP server with three tools.

  • lookup, which takes the subjects in your project and returns what's known about them.
  • report, which drafts a claim and queues it for your approval.
  • confirm, which records whether your agent reproduced a claim or not.

We'd seed it with the findings we already have, mostly about Laravel Cloud, Forge, and the Laravel ecosystem, and invite a handful of developers to point their agents at it.

What we want to know is whether a fresh agent session that checks Clank Overflow first avoids a detour that a session without it takes. If it does, even sometimes, the notes our agents are already writing were worth sharing all along.

Support the Lab

The Lab runs on our supporters

Every idea here gets tried for real: deployed, broken, fixed and written up. Right now that is one person doing the work between everything else, and supporters are what buy the time. As support grows, it will also pay freelancers to run experiments of their own, so more of these ideas get tested, sooner.

Our supporters

  • Chill Vibes

    A curated collection of fun and interesting experiments and projects built with AI to entertain and educate.

    Supporter Visit
    Visit Chill Vibes
  • Fawnfox Fables

    Create and publish your own children's books. Stories built around the people you love.

    Supporter Visit
    Visit Fawnfox Fables
  • Sputnik Intelligence

    Spain

    We transcribe the podcasts and read the newsletters, and make them searchable. Set up alerts, or hand it all to your AI agent over MCP or API.

    Supporter Visit
    Visit Sputnik Intelligence
  • Team of Me

    Free resources for indie makers and people thinking about striking out on their own.

    Supporter Visit
    Visit Team of Me
  • Wicked Viral

    Earth

    Connecting builders with promoters who can help them take their products to market. Get your product listed today!

    Supporter Visit
    Visit Wicked Viral