Fault-Tolerant Pipeline2025Shipped
LLM Content Pipeline
One topic in, a ≈250-page structured document out, resumable.
A staged generation pipeline that turns one topic into a ≈250-page structured document through ≈186 sequenced LLM calls. It runs behind an Express API with a CSV-driven batch CLI. Long, expensive runs survive failures: a crash at call 150 resumes without regenerating finished work.
lockPrivate repositoryNext: PyTorarrow_forward
~/content-pipeline
Project Telemetry
Pages / document
≈250
LLM calls
≈186
Cost / document
≈$0.12
Resume
per unit
How it works
~/content-pipeline/architecture · step 1/3
01
topic / CSV batch
02
outline planner
03
chapter generator
04
context compressor
↺ loops to 03
05
assembler
06
checkpoint store
01Checkpoint/resume at unit granularity plus retry rounds; object-store state doubles as the idempotency key, so re-running a batch skips finished outputs.
02Context chained through micro-summaries instead of full history, keeping coherence inside a bounded window.
03Marginal cost held to ≈$0.12 per document through model tiering. Deployed on AWS EC2 with Docker Compose.
Project Info
DomainAI Systems
RoleAuthor
Year2025
StatusShipped
Tech Stack
TypeScriptNode.jsExpressOpenAIDockerAWS EC2