Skip to content
Scalpels home

Sign up now for $10 in credit on a new Laravel Cloud account.

Idea

How far can we push Hone's ingest?

· Ed Grosvenor

Hone is our self-hosted receiver for Nightwatch telemetry. The apps we run post their records to it, it keeps them, and we query them later. It has been one PHP application on Laravel Cloud since we built it, and it has stopped keeping up.

The way we found out is the part that bothers us. Consumer apps were timing out on the post and dropping batches, and for a while nothing said so out loud. We filed it as hone #106. We weren't the only ones to notice. Peter Van Dijck, who runs Hone for Sputnik's search, counted eight timeouts against the client's 500 ms budget in a single day. The clients that did count their losses know they threw away about 149,000 records across five apps. A batch lost to a timeout never reaches a counter at all, so that number is a floor.

The obvious move is to buy a bigger instance. We are not going to start there. Hone is a Built for Cloud app, which means anything we do to it has to be something a customer's own Cloud account can run, and "pay for more compute" is the answer we want last rather than first.

What is left on the stack we already have

Hone runs on PHP-FPM. We have never turned on Octane, which keeps the application booted in memory between requests, even though we have watched it buy real headroom in client apps. The credential lookup that authenticates every post runs against Postgres on every request, and so does a write that stamps the token as used. The queue worker takes its messages one at a time. None of that is exotic, and all of it is ordinary PHP on the generally available runtime.

So the first question is just how much of this is a capacity problem at all. If the box is starved, Octane should show it, because a request that holds a worker for a quarter as long is a request that leaves room for the next one. If Octane buys enough, we stop there and nothing else changes for anyone installing Hone.

There is a constraint on all of it that we are not willing to trade. Nightwatch's whole selling point is that it does not measurably slow the app it watches, and our client has to match that. Whatever we do on the server, the client is the last thing we touch.

What Rust changes

We have early access to the Rust runtime on Laravel Cloud, the same access that let us put Campfire's Rust port on it a couple of weeks ago. The long-term shape we keep coming back to is a compiled service that owns POST /ingest and nothing else. It would sit next to the PHP app, and the queue jobs, the rollups, and the MCP surface would stay exactly where they are.

We do not know whether Cloud will let us do that. Two targets out of one repository might mean a second application or a second environment. On Cloud, one application can have several environments, and each one deploys separately. We found last time that Cloud's detection of what kind of app a repository holds is fussy about where it looks. Two applications sharing one database is the thing the whole shape depends on, and we have never attached one schema to two environments. A compiled service also cannot use the framework's auth, so somebody has to write credential resolution a second time, including the revocation-guarded write that makes a token's use a security check rather than a statistic. And the queue sits between the two halves, so we need to know whether a non-PHP application on Cloud can produce to a managed queue or consume from one.

Then there is the awkward part. Rust is early access for our organisation only, so a customer installing Hone today cannot have any of this. Our own Hone can go first, and theirs stays on PHP until the runtime is generally available. That makes the install recipe, and the manifest that drives it, the real deliverable rather than the binary.

What we want out of it

An order of work, ranked by what it costs to ship, with numbers behind each step rather than guesses. The timeouts stopping is the bar. Knowing which of the two routes is worth building, and what we would be asking Laravel for if we picked the second one, is the thing we actually get to keep.

Experiments on this idea

  1. Experiment

    Scaling Hone's ingest with Octane and Rust

    Octane cut the median ingest request from 85 ms to 22 ms on the same 512 MB instance, with no capacity added, and the multi-second tail went away. A throwaway Rust service on a second Cloud target handled about six times the throughput of PHP on Octane at the same size. It also hit one hard platform limit. Cloud refuses managed queues to non-PHP apps.

    Read the experiment

Support the Lab

The Lab runs on our supporters

Every idea here gets tried for real: deployed, broken, fixed and written up. Right now that is one person doing the work between everything else, and supporters are what buy the time. As support grows, it will also pay freelancers to run experiments of their own, so more of these ideas get tested, sooner.

Our supporters

  • Chill Vibes

    A curated collection of fun and interesting experiments and projects built with AI to entertain and educate.

    Supporter Visit
    Visit Chill Vibes
  • Fawnfox Fables

    Create and publish your own children's books. Stories built around the people you love.

    Supporter Visit
    Visit Fawnfox Fables
  • Sputnik Intelligence

    Spain

    We transcribe the podcasts and read the newsletters, and make them searchable. Set up alerts, or hand it all to your AI agent over MCP or API.

    Supporter Visit
    Visit Sputnik Intelligence
  • Team of Me

    Free resources for indie makers and people thinking about striking out on their own.

    Supporter Visit
    Visit Team of Me
  • Wicked Viral

    Earth

    Connecting builders with promoters who can help them take their products to market. Get your product listed today!

    Supporter Visit
    Visit Wicked Viral