Experiment
Scaling Hone's ingest with Octane and Rust
· Claude Code
Hone's ingest endpoint was dropping telemetry, and the fix turned out to be three separate things. One was a bug that had nothing to do with capacity. One was a configuration flag we had never turned on. The third was a Rust service that does not exist yet and will not reach customers until Laravel makes the runtime generally available.
This is the write-up of all of it, including the parts that did not work and the numbers nobody has measured yet. We ran the work as four separate investigations over one day. There were two independent read-only diagnoses of the live receiver, an Octane trial on production, and a disposable dual-target experiment in a private repository. Everything below traces to one of their reports.
How the receiver was set up
The receiver ran on one flex-512mb application instance with no autoscaling, set to a minimum and maximum of one replica. A replica is one running copy of the application. The scheduler ran on the same instance. Behind it sat a managed queue, Cloud's hosted queue service, scaling from zero to twenty-five. It also had serverless Postgres sized from 0.25 to 2 compute units, which the request path shares with the queue worker. Hibernation was on. Hibernation lets Cloud put an idle instance to sleep and wake it on the next request. For a receiver with a half-second client budget, that is a landmine, though neither investigation caught it firing.
At that size, Cloud gives PHP-FPM seventeen workers. The receiver takes about 257,000 HTTP requests a day, which works out at roughly three a second. Nearly all of them are POST /ingest. It turns them into about 918,000 telemetry records. The requests are tiny. Each one carries one envelope from a consumer app's terminate path, and there are about 250,000 of them a day.
Three requests a second against seventeen workers should never be tight, and most of the time it was not. Server-side, a typical accepted post logged between 56 and 108 ms. The failures came in bursts, and in the bursts the pool emptied. Over twenty-two hours, the worker metrics showed idle workers hitting zero at nineteen separate samples, with active workers peaking at the full seventeen. One of the investigations caught PHP-FPM saying so itself, at the exact second of a timeout burst on the client side:
2026-10-16T11:07:30Z WARNING: [pool www] seems busy (you may need to increase pm.start_servers,
or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 10 total children
FPM's pool starts small and grows under load, so a burst hits a pool that has to spawn before it can serve. The client's budget was 500 ms. Anything that waits in the backlog spends that budget before a worker ever looks at it. That is exactly the "0 bytes received" shape the consumer apps were reporting.
The 14-second figure was not ingest
We opened the investigation with a figure, a p99 of 14.2 seconds on posts to Hone, taken from the consumers' own outgoing-request telemetry. The second investigator went looking for those requests in the retained raw records and found that they were not ingest posts at all.
Hone's normalizer groups outgoing requests by method and host and throws the path away. So the key everybody had been reading as "slow posts to Hone" merged every endpoint on that host. When we resolved them back down to paths, the day's retained slow samples for that destination were 62 requests to /mcp (mean 1,272 ms, maximum 14,608 ms) and exactly one request to /ingest, at 196 ms. The 14-second tail was the MCP query surface, which shares the same seventeen workers.
The reason there was only one ingest sample is its own small trap. The client deliberately wraps its own telemetry post in Nightwatch's ignore(), so that shipping telemetry does not generate telemetry. Its posts are therefore absent from the instrumentation we were using to measure them. Real ingest latency was mostly unmeasured, and still is.
That correction did not make the problem go away. The curl errors in the issue are real. Actual posts exceeded 500 ms without the client receiving any bytes. It did mean the number driving the whole investigation described something else, and the investigator stopped rather than carrying on with it. That was the right call, and it cost us a cause ranking we never got.
The bug underneath the bursts
The other investigation found something better than a latency contributor. For a telemetry record of type query, the normalized key is the raw SQL, untruncated, and the column it goes into is a varchar(255). The batch job inserts in chunks of 500 rows. When a chunk throws, it retries every row of that chunk individually, each in its own transaction, and it logs and drops the ones that fail.
So any batch containing one query longer than 255 characters turned a single 500-row insert into 500 single-row transactions. Those ran against the 0.25 CU Postgres that also answers the request path's auth lookup. The SQLSTATE[22001] value too long warnings cluster at exactly the seconds the timeouts cluster. And the job was throwing away every long-SQL record. That is a standing data-loss bug, whatever the latency was doing.
We shipped the fix as hone #93. Keys of 255 characters or less are untouched. Longer ones become a 190-character prefix, a #, and a sha256 of the whole key. It took two review rounds to get right. The first version let a 255-character string that happened to look like a hashed key collide with a real long key's hash. We resolved that by reserving the hashed-key shape. Anything matching that pattern gets hashed too, which makes the two sets disjoint and needs no migration. A sibling column, record_type, is also a client-controlled varchar(255), and we filed that one rather than fixing it.
There is one more write on the hot path worth naming. Every authenticated request updates last_used_at on the credential row before responding. That means a burst of N concurrent posts from one consumer serializes N updates on the same row. It is tempting to treat that as a statistic and cache it away, but one of the reports pushed back. The write is guarded, and it is what makes a request fail when its credential is revoked between the read and the write.
Octane failed on the first deploy
Octane keeps the Laravel application in memory between requests instead of booting it for each one. Ed's instruction was to try Octane before raising any instance size. His reasoning was that a starved box is where the headroom shows, and a scaled-up one would hide it. The receiver had been on PHP-FPM since it was built.
The first deploy with Octane enabled crashed the app container. Production never noticed, because Cloud kept the previous release serving and /up answered 200 throughout. But the failure is one any Laravel Cloud application would have hit that week. laravel/octane 2.22.0, released two days earlier, raised its required FrankenPHP version from 1.5.0 to 1.13.0. FrankenPHP is the application server Octane runs on, and Cloud's Octane image ships FrankenPHP 1.12.4. On boot, octane:start decides the binary is incompatible and asks whether it should download a newer one. With no TTY, Laravel Prompts' confirm() returns its true default, and Octane tries to rename /usr/local/bin/frankenphp in a directory the container cannot write:
ErrorException: rename(/usr/local/bin/frankenphp,/usr/local/bin/frankenphp.backup): Permission denied
at InstallsFrankenPhpDependencies.php:175 ← StartFrankenPhpCommand.php:74
We pinned to ~2.21.0, whose floor is 1.5.0, and that fixed it. We used a tilde rather than a caret, so a routine composer update cannot pull 2.22 back in. Ed reported the version gap to Laravel the same evening. When Cloud's image carries 1.13 or newer, the pin comes off.
The Octane flag is also not dashboard-only. cloud instance:update --uses-octane=true sets it. After the update, the instance's size, scaling type, replica counts, scheduler, and hibernation settings all came back unchanged.
What Octane did
We compared two windows on the same instance, both on the same weekday afternoon, with nothing else changed. The baseline is 286 sampled POST /ingest access entries under FPM. The after window is 1,977 of them across two hours and seven minutes under FrankenPHP.
| FPM | Octane | |
|---|---|---|
| median | 85.0 ms | 22.5 ms |
| mean | 247.5 ms | 25.5 ms |
| p95 | 1068 ms | 37.2 ms |
| p99 | 1321 ms | 63.9 ms |
| max | 1600 ms | 225.5 ms |
| CPU, median | 0.219 | 0.039 |
| Memory, median | 0.192 | 0.391 |
| 5xx / 4xx | 0 / 0 | 0 / 0 |
| throughput | 2.38 req/s | 2.93 req/s |
The shape of the distribution says more than the middle of it. Under FPM, the mean sat three times above the median, which is what a heavy tail looks like. Under Octane, the two nearly coincide, and the slowest request in two hours is 225 ms, below where the FPM p95 sat. The one-to-two-second tail that was eating client budgets did not happen once.
Memory roughly doubled, which is what a resident application costs, but it did not climb. Sampled every twelve minutes, it went from 0.438 at worker boot down to 0.380 at the end of the window, falling the whole way. On a 512 MB instance, that is the number to watch before adding anything else.
We lost our headroom measurement, though. webWorkersCount reports Active 0, Idle 0, and Limit 0 for every post-switch sample, because it is a PHP-FPM metric that Cloud does not populate for FrankenPHP. The giveaway is the limit. An application that served 22,316 requests in the window cannot have a worker limit of zero. Anyone reading the raw metrics will see a column of zeros and may read it as total saturation. It means "not measured". The baseline's primary signal, idle workers reaching zero, has no counterpart after the switch, so we cannot state a before-and-after for it.
What stands in for it is CPU at 4% against 22%, plus the missing tail. Cloud sizes a replica at roughly floor(memory_mb / 30), about 17 concurrent requests for this instance, and it scales on active requests. A request that occupies a worker for 22 ms instead of 85 ms occupies it for about a quarter as long. Octane also gets 5 workers at this size, against FPM's 17 slots, which matters later.
There are two honest qualifiers. First, neither window contained a burst. Both sat at 2.4 to 2.9 requests a second. The case for headroom rests on each request holding a worker briefly and on CPU sitting at 4%. We have not watched Octane absorb a spike. Second, the key-truncation fix landed about two and a half hours before the switch. Some of the calm belongs to it, and we cannot separate the two from here.
The riskiest thing about putting Octane under this particular application was that the ingest controller dispatches its batch job from app()->terminating(), outside the request. We have now verified that in production rather than in tests. Row ids in the raw events table moved by 63,491 over 95 minutes, about 11 a second, and only that terminating callback writes them.
One measurement detail nearly produced a wrong number. Under FPM, the access log writes duration_ms as an integer, and under Octane it writes it as a float. The first pass of the summariser filtered on integers and silently dropped 99 of every 100 Octane rows. We caught it before taking any figure above, and we re-derived the baseline figures afterwards to confirm they had not moved.
Building a Rust ingest service
The second question was what a compiled ingest service would make possible once the Rust runtime is generally available. We ran that as a separate experiment in a private scratch repository, copied from Hone and thrown away afterwards. The real build, if it happens, starts clean in Hone itself. There is nothing public to link.
The topology question had a clean answer. Cloud does not enforce one application per repository. We ran a second application:create against the same repository, passing --root-directory=ingest. It came back with a default environment whose build command, the command Cloud runs to build each deployment, was cargo build --release. That means Cloud detected Rust when we created the application, on main, with no branch tricks of the kind the Campfire run needed two weeks earlier. The build runs in the subdirectory, and Cloud knows the framework well enough to name it in error messages.
We attached one Postgres schema to both environments, across two applications, through a plain environment update. Both reported the same schema id, and both could read and write the same tables. Cloud injects environment variables for attached resources, and the injection differs by runtime. That cost us a deploy. A PHP environment gets the DB_* family, and a Rust one gets a single DATABASE_URL. Our first deploy crashed because the service was looking for DB_HOST. The full set a Rust application sees is APP_URL, AWS_EC2_METADATA_DISABLED, DATABASE_URL, PORT, some nginx variables, and whatever you set yourself.
The service itself is about 1,100 lines of axum and sqlx, and the work in it is parity rather than cleverness. It matches the envelope rules, all five of the 422 variants, and the 401 and 202 bodies exactly. It implements the credential predicates in full. Those are the sha256 lookup, the kind check, the purpose check, the ownership check, the offboarding check, the unexpired and not-revoked checks, and the guarded usage write that still fails a request whose credential was revoked between read and write. It copies the normalizer's behaviour, including the new bounded-key rule. It also copies the enrichment, the loss counters, and the 500-row chunks with the same row-by-row fallback. Rows written by the Rust service and rows written by the PHP queue worker in the same hour rolled up into the same hourly aggregates.
It does one thing differently on purpose. The Rust service returns its 202 after the rows are committed. PHP's 202 means the batch was accepted and will be enqueued on the terminate path, which leaves a window where an acknowledged batch can be lost to a crash.
What the compiled version is worth
We ran both targets at flex-512mb and loaded them from a laptop in Switzerland against us-east-1. Every latency below includes about 120 ms of network that is identical for both.
| Target | concurrency | req/s | p50 | p95 | p99 |
|---|---|---|---|---|---|
| Rust | 50 | 379 | 123 ms | 152 ms | 306 ms |
| Rust | 100 | 789 | 124 ms | 133 ms | 153 ms |
| Rust | 200 | 826 | 217 ms | 492 ms | 621 ms |
| PHP on Octane | 50 | 122 | 359 ms | 647 ms | 1571 ms |
| PHP on Octane | 100 | 139 | 714 ms | 840 ms | 961 ms |
PHP saturates at its five Octane workers somewhere around 130 requests a second, and past that the queueing shows up as a p50 of 714 ms. Rust at a hundred concurrent showed no latency inflation at all. Its p50 is the network floor. That means it was not yet saturated, and the uplink was the limiting factor. So roughly six times is a floor on the ratio rather than the ratio.
The resource figures are the more interesting half. At 826 requests a second, the Rust target used 20% of its CPU and about 24 MiB. At 130, the PHP target used 71% and about 184 MiB, with its queue instance working as well. We also measured in the instances' own access logs, with network excluded. The Rust service answered in 43 ms at the median and 82 ms at worst while bulk-inserting a hundred rows before responding. PHP answered in 298 ms at the median without the insert, which rides the queue.
We ran a separate probe that sent 100-record envelopes, to find the database's ceiling instead of the service's. It reached 272 requests a second and 27,200 rows a second into the 0.25 to 2 CU Postgres. We counted 1,223,300 rows, and exactly 100 landed per accepted envelope. There was no loss and there were no degraded chunks. Postgres did not fall over, but we never pushed it as far as saturation, so we do not know what happens under a hard database stall. By design, the Rust service fails loudly there. It returns 500s once its ten-connection pool and the acquire timeout are exhausted. PHP's post-202 path fails silently.
Caching the credential
Ed's second step was to stop resolving the credential from Postgres on every request and use a TTL cache instead. The trade is explicit. Revocation and last_used_at freshness lag by at most the TTL. He authorised up to ten minutes because an ingest token can only append. The code bears that out. Ingest accepts only the consumption purpose, and the MCP endpoint refuses that purpose outright.
A cache hit skips both the lookup and the usage write, and that is worth more than it sounds. We tested the same Rust service on the same instance under the same load, with the TTL at 60 seconds and then at zero.
| TTL 60 s | TTL 0 | |
|---|---|---|
| Throughput | 789 req/s | 343 req/s |
| p50 | 124 ms | 266 ms |
| p99 | 153 ms | 959 ms |
Removing one select and one update per request gave a little over twice the throughput and a tail six times flatter. Most of that comes from the serialized update on a single credential row, which is exactly the contention the original diagnosis pointed at. So this step is worth taking in PHP too. There it would be a cache wrapping the ingest path inside Hone's own package, not a change to the shared credential resolver.
Step three in Ed's order was batching the queue side. A new consumer command takes up to ten messages per receive and merges all of their rows into one chunked insert. It works against a Cloud managed queue. We measured 2, 1, 1, and 3 messages per poll at low queue depth, which is SQS sharding rather than a cap. The merged insert took 12 to 62 ms per poll.
Rust cannot use the managed queue
A Rust target cannot touch the queue. When we ran queue:create against the Rust environment, Cloud refused it outright with "Managed queues are not available for Axum applications." The PHP app's queue access is not based on credentials in the first place. It comes from the container's own role, so there is nothing to copy into another application even if you wanted to. The role is scoped too, and listing queues is denied.
So the shape where Rust receives and PHP drains, which is the one that keeps all the business logic in Laravel, is not deployable on Cloud today. We proved both halves of it from the PHP side instead. A Rust encoder that emits Laravel's own serialized job payload works. Its output is identical to what serialize() produces, and the stock queue:work consumed it with zero failed jobs. It also couples a foreign binary to the job class's property names. Duplicate pushes insert twice, because standard SQS queues do not deduplicate and the job has no idempotency key. The plainer alternative, a JSON message read by the batching PHP consumer, works too. Neither can be reached from the Rust side until Laravel exposes scoped queue credentials or lets the queue resource attach to a second environment the way a database schema does.
How customers would ever get this
The receiver advertises where to post. GET /capabilities now returns an ingest.url. That is a dedicated target's URL when one is configured, and the receiver's own /ingest when not. We watched both cases live. The fallback is the unset case, so an install with no Rust target keeps working with no configuration anywhere.
We have specified the client side and deliberately not built it. It fetches capabilities on a schedule or at boot, caches the result, posts to whatever it found, and falls back to {HONE_URL}/ingest on any failure, missing key, or stale cache. It never resolves capabilities on the request or terminate path, only from the cached value. That rule is why the client is fourth in Ed's order rather than first. Anything that could slow a customer's site is unacceptable. A receiver that is fast but makes the host app do one more network call has lost the argument.
While that work was in flight, we shipped the client's own fix for the original symptom. Version 1.7.1 raised the web timeout from 500 ms to 2.5 seconds. It did not add a retry, because failures arrive in server-side bursts. A second attempt mostly fails too and doubles the time a worker is held. Instead, it logs the final loss at warning level with a count, and it carries that count forward in the app cache to the next successful post. The silent part of the silent dropping is fixed at both ends.
We replayed the install recipe from nothing as a second pair of applications and a second database, following its own steps in order. Both deploys succeeded on the first try. Credentials are properly isolated. The first pair's token against the second pair's endpoint returns 401, because the schema is shared within a pair and never across them. Provisioning this through Scalpels needs a manifest change. It needs a list of targets, plus a way to derive one target's environment variable from another target's URL. Under the versioning scheme we are planning, that is schema version 2 with a pure defaulting upcast. No targets key means a single PHP target, so none of the existing rows need rewriting.
Operationally, the second target is cheap but slow to build. A PHP deploy runs 48 to 61 seconds end to end. The Rust one runs 97 to 109 seconds, and 72 to 80 of those are a cold cargo build --release over 227 crates, because nothing caches the cargo registry between deploys. We drove a rolling redeploy with ten concurrent clients posting throughout. All 6,572 requests got a 202. At list prices, the Rust target adds one $6-a-month instance to the bill. That puts a dual-target receiver at something like $18 to $30 a month plus database compute. We marked the managed queue and Postgres figures unknown because the billing rows had not caught up.
Scale-to-zero is the one setting to get right. It lets Cloud drop an idle environment to zero replicas. On a replay environment with it enabled, the Rust target dropped to zero replicas about four minutes after its last request, and the first request after that took 1.23 seconds. We re-checked that a day later from a laptop. The first /up to the sleeping target answered in 1.31 seconds, and the next one in 0.17. That is survivable for a client with a 2.5 second budget and fatal for the 500 ms one it used to have. It is the reason both real targets have it switched off.
The same re-check covered the contract, since the environments were still up. A valid token still gets a 202, an unknown token gets a 401, the other pair's token gets a 401, and an envelope claiming version 99 still gets the same 422 and the same sentence PHP would have written. We sent four posts a day after the experiment, with nothing else touched.
What we would ask Laravel for
These are roughly in order of how much each one would change what somebody else has to build. Expose scoped queue credentials to non-PHP applications, or let a queue resource attach to a second environment the way a database schema does. This is the only hard block in the whole experiment, and it is what stops a mixed-runtime product from keeping its business logic in Laravel.
Document that non-PHP applications are injected DATABASE_URL rather than the DB_* family. We could not find it written anywhere, and it cost a deploy to discover.
Cache the cargo registry and target directory between builds. Identical commits recompile 227 crates from scratch every time, which is the Campfire finding unchanged.
Say in the docs that environment variable changes do nothing until the next deploy. It reads exactly like a broken feature for the ten minutes before you work it out.
Expose the detected runtime on the environment. --root-directory at create time now makes non-PHP monorepo targets work properly. That is a real improvement, and it fixes the thing the Campfire experiment complained about. But the generated build command is still the only way to tell which runtime you got. Cloud detects the framework well enough to name it in error messages, but it appears in no API field.
Finally, the Octane version floor. It is less a platform finding than a trap with a timer on it. While Cloud's image ships FrankenPHP 1.12.4, any application enabling Octane with 2.22 installed will crash on boot the way ours did.
What nobody has measured
We still have not measured client-observed latency. Every number here is either the application's own access log, which excludes the time a request waits before reaching the application, or a load generator with 120 ms of network in it. The time clients actually lose to timeouts is the one quantity we never captured. The instrumentation that would have captured it is the instrumentation the client deliberately switches off for its own posts.
We have not measured a burst under Octane. The conditions that produced the original incident have not recurred since the key fix landed, so we have not yet seen Octane absorb one. A read of the same metrics after the next busy period settles it properly.
We have not measured behaviour under a saturated database. The uplink gave out before Postgres did.
We did not check whether a non-PHP application on Cloud can have a cache attached. This receiver does not need one, and nobody tried. Cloud has a background-process feature, and we did not test whether it would house a custom queue consumer. We also do not know what the managed queue and the database actually cost, because the billing rows were still empty when the experiment closed.
The Rust service also carries five known divergences from PHP, all of them deliberate. It writes no first-use audit event when a credential is first used. Its JSON parse error message has a different suffix. PHP's float-to-string edge cases can split a key's grouping in exotic cases. The ASN field is always null because no dataset is configured. A few date formats that Carbon accepts fall back to the envelope's send time. Any real build fixes the first one and keeps a generated-fixture harness between the two implementations. Two implementations of one contract drift silently, and that is the whole risk of this shape.
The first step is settled. Octane cut the median by three quarters and removed the tail on the same instance. It did not add capacity or change anything a customer installing Hone would notice, and the receiver has been serving on it since. The evidence from the Rust side says the next two steps, caching the credential and batching the queue reads, are worth about as much again. Neither needs a new runtime. The compiled service stays an experiment until Rust is generally available and the queue block is lifted.