feat: fork static export (#415)
The static exporter has been rebuilt around a streaming path source and
an opt-in multi-process render coordinator. Before this change,
exporting a large site materialised the full path list into an array,
then ran every render in Promise.all on a single thread. At ~24k pages
of Shiki-rendered HTML the resident set climbed into the multi-gigabyte
range even though only a handful of pages were live at any instant, and
runs would OOM long before they finished. Sites with non-trivial
server-component work were also bottlenecked on a single RSC thread
regardless of available CPU.
The new pipeline splits that work in two. A shared streaming layer
(buildPathStream, pMapStream, fanout) consumes path sources lazily
— options.exportPaths and configRoot.export can now be async
generators or any AsyncIterable, and the renderer pulls one path per
free worker, so peak memory is O(concurrency × chunkSize) regardless
of total path count. Layered on top of that, a forked coordinator can
spread the render across N child processes, each running its own RSC
main thread and SSR worker thread, dispatching paths over IPC. Every
artifact (HTML, gz/br sidecars, .postponed.json,
.prerender-cache.json) is written to disk inside the child, so output
bytes never cross the IPC boundary; the coordinator only ferries small
log entries back. The two modes produce an identical artifact set —
postpone and prerender-cache work in multi-process mode because each
child uses the same setupStaticRender + emitAllArtifacts path the
single-process exporter does.
The number of render workers is exposed as a new --export-concurrency
flag (and an exportConcurrency field in react-server.config.mjs),
defaulting to a CPU-scaled value between 2 and 4. Setting it to 1
collapses to the single-process in-line exporter — cheapest for small
sites and useful for debugging. Documentation has been added in both
English and Japanese covering the flag itself and the documented "level
up" pattern for config.export as an async function* for cases where
the path list is too large to materialise (CMS pagination, large
database queries, tens of thousands of file-router slugs).
A handful of related bugs surfaced and were fixed alongside the rewrite.
The backpressure PR had moved prerenderInit after the static handlers
in lib/start/create-server.mjs, which broke the PPR resume flow
because lib/handlers/static.mjs mutates PrerenderStorage when it
serves a .postponed.json sidecar; the middleware is moved back ahead
of the static handlers. The static-worker child's fatal() was racing
process.exit(1) against an unflushed IPC send, so worker crashes
appeared to the parent as bare exit codes with no underlying error —
fixed by waiting on the send callback before exiting. The renderer
worker URL in static-runtime.mjs was incorrect (./render-stream.mjs
resolved into lib/build/); it now resolves correctly into
lib/start/. The Postponed control signal that React's prerender
machinery throws at dynamic boundaries was being printed as a red error
stack by the static exporter's logger proxy; it's now filtered by the
REACT_SERVER_POSTPONED digest. Per-path render failures are once again
printed in red on stderr above the summary line — the streaming refactor
had silently dropped the error log while keeping the count. The
compression default is restored to the documented false (a previous
attempt to fix "compressed sizes not shown" in multi-process had
inadvertently flipped the default on); the real fix lives in the IPC
plumbing instead. And the prerender-disabled check in emitHtml now
gates the entire postpone/prerender-cache machinery (the callback
wiring, the cache Set allocation, the sidecar emission) on a single
computed prerenderEnabled, derived from both the per-path and
config-level prerender state — previously a per-path prerender: false
flowed into render but still installed the onPostponed callback.
Tests cover the new pipeline at three layers. Unit tests exercise each
streaming primitive in isolation — pMapStream for bounded concurrency
and lazy pulls, toPathStream for input-shape normalisation,
buildPathStream for the generator/array branch split (including a
passthrough-then-append pattern that mirrors the docs and a fail-fast
error-propagation case), validatedPathStream for descriptor
validation, and fanout for backpressured one-chunk-in-flight delivery.
A second describe asserts the load-bearing structural invariant —
pulled - completed <= concurrency — at a modest N where the assertion
is N-independent but cheap. The end-to-end at-scale coverage lives in a
new fixture spec under test-build-start:
test/fixtures/static-export-many/ is a two-file fixture (a
single-component entry plus a react-server.config.mjs whose export
is an async function* yielding STATIC_EXPORT_MANY_COUNT paths,
defaulting to 100k), and the spec runs the full build + serve round-trip
and spot-checks four sample paths. The harness's vitestSetup.mjs now
derives options.export from a serialisable initialConfig.export flag
so a spec can opt into static export without touching internal build
options; the actual generator function still lives on disk because
functions can't cross the JSON boundary into the build worker.