Skip to content
Scalpels home

Sign up now for $10 in credit on a new Laravel Cloud account.

Experiment

We built a central log sink on Laravel Cloud

· Ed Grosvenor

The storage works. We put Parquet files on the object storage bucket attached to a Laravel Cloud environment, the DuckLake catalog in Cloud Postgres, and an MCP server on top. An agent can ask questions about five million log lines and get answers in between a quarter of a second and 2.7 seconds. Five million rows of Parquet is 280 MB, about 56 MB per million lines. All of it ran on the smallest instance Cloud offers, Flex at 512 MiB, for about $25 a month.

The code is public at artisan-build/ducklake-logs-experiment. It is also all that's left. We deleted the Cloud app, the Postgres cluster, and the bucket on 12 October. A coding agent did most of the building from written briefs, and we checked its numbers against the live app.

What we asked

We asked What happened
Can PHP on Cloud drive DuckDB at all? Yes, the second way. Cloud's PHP 8.5.11 has no ffi, so the FFI client was out. The CLI binary, downloaded during the build, works
Where does an accepted batch wait? A Postgres staging table. Ingest commits the batch and returns 202 with a median of 234 ms. A scheduled command writes the Parquet a minute later
Does compaction keep the small-files problem in check? It runs and it keeps up, but merging inside app-and-day partitions cannot produce large files. Coarsening the partitions is what mattered
Are agent queries fast enough? Not until we changed the layout, and then only with three small Postgres tables alongside
Can the driver be invisible? Yes. Against a dead sink, the host app answered in 180 ms, and handle() cost 0.13 ms
What does all of this cost? About $25 a month, and almost none of it is the Parquet

Phase one: everything except the searches

We got further in the first hour and a quarter than we expected. Ingest authenticates a per-app bearer token, accepts 1 to 500 rows, keys them on (app, idempotency_key), and commits to a Postgres staging table before it answers. We sent eight deliberately tiny 25-line batches, and they came back 202 with a p50 of 234 ms. A duplicate idempotency key returns the original receipt with a 200. A scheduled command runs every minute. It claims staged batches with FOR UPDATE SKIP LOCKED and writes them into DuckLake. Then it nulls the payload and keeps the receipt.

Eight separate flushes wrote eight Parquet files, and ducklake_merge_adjacent_files turned them into one. We ran two Cloud commands on separate compute, and each flushed a 100-line batch at the same time. The lake ended up with 200 rows and 200 distinct event IDs. We installed the driver in a local Laravel app as a path package and logged one line. All five MCP tools found it within a minute.

The driver stays out of the way by never doing network I/O in handle(). It buffers normalized records in memory and registers one callback that runs when the application terminates, with a 0.5-second connect timeout and a 2-second total timeout. We pointed it at an unreachable address. Three live host responses were all HTTP 200, and handle() itself took 0.914 ms cold and 0.13 ms warm. Redaction is a rebindable filter, as the idea page described. In a live probe, we rebound it and shipped [REDACTED BY CONSUMER]. The string it was hiding never appeared in the outbound body or in the MCP result.

The MCP server is laravel/mcp with five fixed tools and no raw-SQL tool. Every response starts with UNTRUSTED LOG CONTENT. That is a label on the output and nothing more. The prompt-injection risk the idea page described is still open, and we still have no answer to it.

Running DuckDB without FFI

The DuckDB FFI client needs ext-ffi and a platform-matched C library. Cloud's PHP runtime has proc_open but not ffi. So the working path is DuckDB 1.5.6's CLI binary plus the ducklake, postgres, and httpfs extensions. The build step fetches all of them, and the app invokes them through Symfony Process at request time. The build step runs during each deployment, before the new version serves traffic. Downloads have to land somewhere under the app root, so they go in var/duckdb-extensions. Nothing is fetched while the app is serving. Starting a fresh process costs 15 to 19 ms.

Bucket credentials do not arrive as AWS_* variables. They come through Laravel's resolved s3 disk configuration. Each CLI process turns that into a temporary DuckDB secret passed on stdin. Nothing is written to a credentials file. Setting DuckLake's DATA_INLINING_ROW_LIMIT to 0 keeps log rows in Parquet instead of letting small writes collect in the catalog.

The first searches were slow

Then we loaded 500,000 synthetic rows across 40 apps and ran the five query shapes an agent would actually use. One-app errors took 4.2 seconds. A literal text search across all apps over 24 hours took 14.7. The same search as a regex took 13.6. A 7-day count by level, app, and hour crossed the 19-second client deadline and failed. A full count(*) took about 72 seconds. A 5M-row insert ran for 8 minutes 26 seconds and rolled back.

The agent recommended parking the query plane. It wanted to keep DuckLake as cold storage and put a hot index tier in front of it for everything interactive. That is a reasonable design, but the agent was choosing it before anyone knew what made up the 14 seconds. Two hundred and eighty-one Parquet files averaging 100 KB, on a shared-CPU 512 MiB instance, with no time breakdown, doesn't tell you whether the problem is the architecture or the arrangement. So we sent it back with one instruction. Profile the request before changing anything about it.

The agent broke down the 14-second search using DuckDB's HTTP logging and DuckLake's metadata logger.

Where the time went Cross-app literal search, 24h
Starting the DuckDB process 19 ms
Loading extensions and settings 68 ms
Attaching the catalog 124 ms
Running the query 15,790 ms
Catalog metadata calls 14
Ranged S3 GETs 132, totalling 15,341 ms
Bytes transferred 2.0 MiB

Ninety-seven percent of the query was spent waiting on object-store requests that moved two megabytes between them. The 7-day aggregate had the same problem at a larger scale, with 572 GETs, 60.6 seconds of HTTP, and 3.4 MiB of data. The one-app query was faster for exactly one reason. Pruning to a single app took it from 132 requests to 16.

So the partitioning was the problem. Partitioning by app and day across 40 apps produces a file per app per day, each around 100 KB. A query touching more than one app has to open all of them. Process startup and catalog attach are the two things a persistent DuckDB service would have fixed. Together they accounted for about a tenth of a second.

We ran the identical query from Zurich against the same catalog and the same objects. It made the same 132 GETs and moved the same 2.0 MiB, but it took 29.5 seconds and spent 2.78 seconds attaching the catalog. The layout decides how many requests a query makes. Distance decides what each one costs. Both are worth knowing, and the second one means keeping the catalog in the same region as the compute.

Partitioning by day and sorting by app

The fix was to give the object store less to do. We rewrote the table partitioned by day only, sorted by (app_slug, occurred_at), with a 32 MB target file size. That turned 286 active files into seven.

Query Before After GETs before GETs after
Cross-app literal, 24h 15.79 s 1.65 s 132 14
Counts by app, level, and hour, 7d 62.10 s 3.21 s 572 14

One change to the layout took one to two orders of magnitude off the request count. We never resized the instance.

Two queries were still too slow at five million rows

At 500,000 rows, the day-only layout looked like the whole answer. At 5,000,502 rows, three of the five shapes passed. Those were text search, regex, and the 7-day aggregate. Two shapes failed the 19-second deadline. The last 50 errors for one app over 30 days took 19.75 seconds, and the ±5 lines of context around a single event took 19.02.

Both failures come from asking a daily file to answer a question it isn't sorted for. A day's file contains every app, so a 30-day single-app query opens 30 files and needs useful row-group pruning inside each one. Context lookup starts from a random event UUID, which is a column with no order at all, so it searches everything.

The obvious next move was finer row groups, so we tried 10,000-row sorted groups. It made things worse. Text and regex searches over 24 hours went back up to 11 to 13 seconds, because smaller groups mean more ranged reads. The 30-day error query still timed out. We threw it away and rewrote the table one more time into coarse daily files, one per day across the 31-day corpus.

Three small Postgres tables fixed those two shapes. The same flush that writes the lake keeps them up to date. One holds hourly counts per app and level, 85,680 groups of them. One maps an event ID to an app and a timestamp, so a context lookup reads one app's slice of one day instead of hunting a UUID through the whole lake. The third holds the error rows themselves, 49,542 of them, about 1% of the corpus. That is what makes a long-range question about one app's failures cheap. Parquet still holds every line and still serves all the text and regex work.

MCP query Rows Cold Warm p50 Served by
Last 50 errors, one app, 30d 50 1,018 ms 262 ms Postgres error subset
Literal text, 24h, all apps 100 2,199 ms 2,225 ms Daily Parquet
Regex, 24h, all apps 100 2,692 ms 2,168 ms Daily Parquet
Counts by level, app, and hour, 7d 20,179 466 ms 393 ms Postgres hourly rollup
Context, ±5 lines 11 1,229 ms 1,251 ms Postgres locator, then one Parquet slice

A final live ingest proved that the real code path keeps the derived tables up to date, not just the one-off rebuild. We sent one error line in through HTTP, and the lake, the locator, the error index, and the hourly counts all moved by one.

512 MiB does set limits. DuckDB is capped at 384 MB inside it, so a 5M-row sorted insert rolls back. Forty-five inserts of 100,000 rows each took about a minute. Sorted compaction with five or more outputs in one run also goes over the cap. Three outputs run reliably.

What it costs

Of roughly $25.40 a month, the Parquet costs half a cent. Five million rows of active Parquet is 279,875,087 bytes in 32 files. At $0.02 per GB-month, that is $0.0056. Postgres storage after the query-plane rebuild is 629,448,704 bytes, about 32 cents. The Flex instance is capped at $6.

The remaining $19.08 is a serverless Postgres cluster at 0.25 CU. It never sleeps, because a once-a-minute poll of an empty staging table keeps waking it up. An event-driven flush, or just a slower cadence, would let it suspend between batches. That is the cheapest change available here, and it has nothing to do with storage or queries.

What we didn't prove

We asked for 50 million rows and reached 5 million. The locator is one Postgres row per log line, and it makes up most of that 629 MB. So it is the first thing to benchmark at 50M. A bucketed Parquet locator or a key-value store may be cheaper than a table.

The lake commit and the derived-table updates happen in sequence across two systems. A crash in the gap between them can replay rows or lose a counter. Production needs an idempotent outbox or manifest. Snapshot expiry and physical file deletion are defined, but we have not scheduled or proven them. We have not run concurrent queries against this at all. Every latency on this page is one question at a time. The hourly rollup returns whole hourly buckets, so a caller that needs exact partial-hour boundaries has to reconcile the first and last hour itself.

What comes next

The storage question was the one we couldn't answer by reading. Parquet on a bucket with a SQL catalog will serve an organization's logs to an agent, as long as you treat object requests as the scarce resource and lay the files out accordingly. The same discipline should transfer. Hone collects telemetry into Postgres and has the same kind of problem ahead of it. So does anything else that takes a lot of small writes and reads them back weeks later.

So we are building it under a name, from what we learned, instead of extending a throwaway. Our measurements stop at five million rows, and the next round has to start there. It needs to cover 50 million rows, the locator's real cost, and a lot of agents asking at once.

The idea behind this experiment

  1. Idea

    A central log sink for many Laravel apps

    We know someone running about forty Laravel apps who would like to talk to all of their logs at once. Our Postgres approach to telemetry is the wrong shape for raw log volume. Before designing anything, we want to know whether the storage works. The plan is Parquet files on the environment's bucket, the catalog in Cloud Postgres, and ingest in PHP. Nothing has run yet.

    Read the idea

Support the Lab

The Lab runs on our supporters

Every idea here gets tried for real: deployed, broken, fixed and written up. Right now that is one person doing the work between everything else, and supporters are what buy the time. As support grows, it will also pay freelancers to run experiments of their own, so more of these ideas get tested, sooner.

Our supporters

  • Chill Vibes

    A curated collection of fun and interesting experiments and projects built with AI to entertain and educate.

    Supporter Visit
    Visit Chill Vibes
  • Fawnfox Fables

    Create and publish your own children's books. Stories built around the people you love.

    Supporter Visit
    Visit Fawnfox Fables
  • Sputnik Intelligence

    Spain

    We transcribe the podcasts and read the newsletters, and make them searchable. Set up alerts, or hand it all to your AI agent over MCP or API.

    Supporter Visit
    Visit Sputnik Intelligence
  • Team of Me

    Free resources for indie makers and people thinking about striking out on their own.

    Supporter Visit
    Visit Team of Me
  • Wicked Viral

    Earth

    Connecting builders with promoters who can help them take their products to market. Get your product listed today!

    Supporter Visit
    Visit Wicked Viral