Disaggregated Prefill-Decode Clusters, Hardware-Enforced Confidential Compute on H200, and Local-First Episodic Memory Graphs

The Hook

For the past two years, the standard recipe for deploying large language models has been homogeneous replication: spin up identical multi-GPU instances and let each replica handle both prompt prefill and token-by-token decoding. Yet prefill and decode are fundamentally incompatible workloads—prefill is violently compute-bound and saturates tensor cores, whereas decoding is memory-bandwidth bound and sensitive to millisecond-level scheduling jitter. This weekend, production telemetry from large-scale disaggregated clusters confirmed that physically separating prefill nodes from decode nodes—and streaming KV-cache blocks over RDMA—doubles cluster goodput under strict p99 latency SLAs.

At the security layer, enterprise adoption of frontier models is increasingly gated on verifiable data confidentiality. With hardware-enforced Trusted Execution Environments (TEEs) now matured on Hopper H200 and Blackwell accelerators, cryptographic remote attestation allows clients to verify that neither the cloud provider nor the host hypervisor can inspect plaintext prompts or KV-cache memory in flight.

Rounding out the week, new advances in temporal knowledge graphs for agent episodic memory demonstrate how decay-weighted thermodynamic scoring outperforms naive cosine similarity on long-horizon user personalization.

This Week’s Signal: SplitWise-KV: Production Disaggregation of Prefill and Decode Pools

Source: Scaling Disaggregated LLM Serving with Zero-Copy KV-Cache Teleportation

An end-to-end architecture and production evaluation demonstrating how asymmetric prefill-to-decode node ratios and layer-pipelined KV transfers maximize GPU utilization while guaranteeing strict Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT) SLAs.

Architectural Analysis

When prefill and decode share the same GPU replica, a single incoming 32k-token prompt monopolizes compute resources for ratusan of milliseconds, stalling all active decode streams on that node. Conversely, during periods of long generation, tensor cores sit at under 15% utilization because HBM bandwidth limits how fast weights can be streamed into registers.

Disaggregated serving splits the cluster into two distinct pools: Prefill Nodes (optimized for high FLOP/s and aggressive chunked batching) and Decode Nodes (optimized for massive HBM capacity and high batch concurrency). Rather than waiting for all transformer layers to finish prefill before transmitting the KV cache, SplitWise-KV pipelines the transfer layer-by-layer: as soon as Layer $L$ finishes computing its key-value projections on the prefill node, an asynchronous GPUDirect RDMA write pushes that layer’s paged blocks directly into the target decode node’s HBM while Layer $L+1$ is still computing. By the time the final prefill layer completes, 97% of the KV cache is already resident on the decode node, adding less than 3 milliseconds of handoff overhead.

Implications

Even on modest 8-GPU nodes, partitioning 2 GPUs for dedicated prefill and 6 GPUs for dedicated decode via NVLink p2p transfers eliminates decode stutter completely and allows operators to tune quantization and batch sizes independently for each phase.

Sparks

Verifiable Confidential LLM Inference on NVIDIA H200 Confidential Compute

End-to-end benchmarks of hardware-enforced GPU TEEs using PCIe IDE (Integrity and Data Encryption) and bounce-buffer DMA encryption show less than 6% throughput overhead for models over 70B parameters.

Take: A 6% performance tax for hardware-guaranteed memory encryption and cryptographic remote attestation effectively removes the last compliance barrier for healthcare, legal, and defense workloads.

Sulcus & Thermodynamic Memory Decay: Why Agents Need Forgetting Curves

Modelling episodic agent memory with Ebbinghaus-inspired decay curves, synaptic reinforcement on retrieval, and typed entity-relationship edges prevents vector store pollution over months of continuous agent interaction.

Take: Dumping every raw chat transcript into an append-only vector table creates a noisy graveyard of stale context; real long-term memory requires active consolidation, contradiction pruning, and relevance decay.

Rust 1.91 Stabilizes AsyncFn Closures and Zero-Cost Pin Projections

The latest Rust toolchain release removes longstanding ergonomic friction around borrowing across async closure boundaries and simplifies high-throughput streaming middleware in Axum and Tower.

Take: Writing high-concurrency SSE streaming gateways and MCP servers in Rust just got significantly cleaner without sacrificing zero-cost state machine compilation.

AI Insider is published by Digital Forge. Forward to a founder who needs it.