react-server675fbba4
react-servercommit9af9b44a72bf

feat: fork static export (#415)

The static exporter has been rebuilt around a streaming path source and an opt-in multi-process render coordinator. Before this change, exporting a large site materialised the full path list into an array, then ran every render in Promise.all on a single thread. At ~24k pages of Shiki-rendered HTML the resident set climbed into the multi-gigabyte range even though only a handful of pages were live at any instant, and runs would OOM long before they finished. Sites with non-trivial server-component work were also bottlenecked on a single RSC thread regardless of available CPU.

The new pipeline splits that work in two. A shared streaming layer (buildPathStream, pMapStream, fanout) consumes path sources lazily — options.exportPaths and configRoot.export can now be async generators or any AsyncIterable, and the renderer pulls one path per free worker, so peak memory is O(concurrency × chunkSize) regardless of total path count. Layered on top of that, a forked coordinator can spread the render across N child processes, each running its own RSC main thread and SSR worker thread, dispatching paths over IPC. Every artifact (HTML, gz/br sidecars, .postponed.json, .prerender-cache.json) is written to disk inside the child, so output bytes never cross the IPC boundary; the coordinator only ferries small log entries back. The two modes produce an identical artifact set — postpone and prerender-cache work in multi-process mode because each child uses the same setupStaticRender + emitAllArtifacts path the single-process exporter does.

The number of render workers is exposed as a new --export-concurrency flag (and an exportConcurrency field in react-server.config.mjs), defaulting to a CPU-scaled value between 2 and 4. Setting it to 1 collapses to the single-process in-line exporter — cheapest for small sites and useful for debugging. Documentation has been added in both English and Japanese covering the flag itself and the documented "level up" pattern for config.export as an async function* for cases where the path list is too large to materialise (CMS pagination, large database queries, tens of thousands of file-router slugs).

A handful of related bugs surfaced and were fixed alongside the rewrite. The backpressure PR had moved prerenderInit after the static handlers in lib/start/create-server.mjs, which broke the PPR resume flow because lib/handlers/static.mjs mutates PrerenderStorage when it serves a .postponed.json sidecar; the middleware is moved back ahead of the static handlers. The static-worker child's fatal() was racing process.exit(1) against an unflushed IPC send, so worker crashes appeared to the parent as bare exit codes with no underlying error — fixed by waiting on the send callback before exiting. The renderer worker URL in static-runtime.mjs was incorrect (./render-stream.mjs resolved into lib/build/); it now resolves correctly into lib/start/. The Postponed control signal that React's prerender machinery throws at dynamic boundaries was being printed as a red error stack by the static exporter's logger proxy; it's now filtered by the REACT_SERVER_POSTPONED digest. Per-path render failures are once again printed in red on stderr above the summary line — the streaming refactor had silently dropped the error log while keeping the count. The compression default is restored to the documented false (a previous attempt to fix "compressed sizes not shown" in multi-process had inadvertently flipped the default on); the real fix lives in the IPC plumbing instead. And the prerender-disabled check in emitHtml now gates the entire postpone/prerender-cache machinery (the callback wiring, the cache Set allocation, the sidecar emission) on a single computed prerenderEnabled, derived from both the per-path and config-level prerender state — previously a per-path prerender: false flowed into render but still installed the onPostponed callback.

Tests cover the new pipeline at three layers. Unit tests exercise each streaming primitive in isolation — pMapStream for bounded concurrency and lazy pulls, toPathStream for input-shape normalisation, buildPathStream for the generator/array branch split (including a passthrough-then-append pattern that mirrors the docs and a fail-fast error-propagation case), validatedPathStream for descriptor validation, and fanout for backpressured one-chunk-in-flight delivery. A second describe asserts the load-bearing structural invariant — pulled - completed <= concurrency — at a modest N where the assertion is N-independent but cheap. The end-to-end at-scale coverage lives in a new fixture spec under test-build-start: test/fixtures/static-export-many/ is a two-file fixture (a single-component entry plus a react-server.config.mjs whose export is an async function* yielding STATIC_EXPORT_MANY_COUNT paths, defaulting to 100k), and the spec runs the full build + serve round-trip and spot-checks four sample paths. The harness's vitestSetup.mjs now derives options.export from a serialisable initialConfig.export flag so a spec can opt into static export without touching internal build options; the actual generator function still lives on disk because functions can't cross the JSON boundary into the build worker.

Author
Viktor Lázár <lazarv1982@gmail.com>
Date
Commit
9af9b44a72bf8274ca1d455db2ba9e19d484f11d
24 files changed+2605 -720