From Feature Stores to Context Engineering: The Rise of Prefill-Only Decision Models

Over the last several years, applied ML has been dominated by conversational AI and autoregressive text generation. Enormous research and compute budgets have gone into optimizing the autoregressive decoding loop: distributed KV caches, speculative decoding, memory bandwidth. That’s all genuinely hard and important work. But it’s made people forget something. For a huge class of problems (real-time ranking, software control flows, RL loops) you don’t need the model to type anything. And insisting on it is costing you dearly in latency and memory.

What I want to walk through here is an architectural pattern that’s quietly gaining traction, connecting two pieces of recent research: Netflix’s GenRec and TypeSafe AI’s Jev decision model. The convergence they point toward is what I’d call prefill-only decision models: systems that replace hand-crafted numerical feature pipelines with natural language context engineering while delivering deterministic, sub-10ms mathematical evaluations.

1. The Paradigm Shift: From Numerical Feature Stores to Context Engineering

Anyone who’s run a production personalization or ranking system knows the real cost isn’t the model, it’s the feature store. You maintain sprawling ETL pipelines converting raw user behavior, telemetry, and item attributes into tens of thousands of categorical IDs, sparse crosses, and dense numerical vectors. Every time the platform adds a new product modality or content vertical, someone has to engineer new features, modify schemas, and retrain a custom tabular architecture from scratch. It’s brittle, it’s expensive, and it’s fundamentally at odds with how rich semantic information actually lives.

Both GenRec and Jev propose a different framing: treat natural language as a universal semantic data interface and shift entirely from feature engineering to context engineering.

Conventional ML:
Raw Events ──► Complex ETL Pipelines ──► 1,000s of Numerical Features ──► Tabular Model (GBDT/DLRM)

Context Engineering:
Raw Events ──► Programmatic Verbalizer ──► Optimized Natural Language ──► Transformer Backbone

In GenRec, this translation is handled by a programmatic verbalizer function:

$x = V(H, \{M_i\}, \tau)$

The verbalizer $V$ compiles the user’s historical interaction sequence ( $H$ ), catalog metadata ( $\{M_i\}$ ), and real-time operational context ( $\tau$ ) into a compact natural language string. You can’t just dump raw logging history into the model because attention has concrete memory and compute budgets. So context engineering comes down to four practical heuristics:

High-signal retention. High-intent interactions (completed watch sessions, strong affirmative signals) are kept with complete metadata.
Noise omission. Low-signal events (immediate abandonments, fleeting impressions) are dropped entirely.
Repetition summarization. Binge-watching behavior gets collapsed into concise summaries rather than listed episode by episode.
Selective elaboration. Cold-start entities get richer descriptive text to compensate for missing interaction history.

By mapping the Pareto frontier between context token length and MRR, Netflix found an elbow point that let them compress inputs from roughly 5,000 tokens down to 1,700. That’s about a 66% serving cost reduction with negligible offline ranking impact. Not a small number.

2. Core Intuition: Decoupling Understanding from Generation

Here’s the thing I keep coming back to: a Transformer’s primary utility is in its latent semantic representations, not its sequential text generation loop.

Generative inference has two phases. The prefill phase ingests all input tokens in parallel, fully saturating GPU matrix cores. The decode phase predicts output tokens one at a time, continually fetching weights and querying the growing KV cache in GPU HBM. The decode phase accounts for over 90% of end-to-end latency and puts constant pressure on memory bandwidth. In high-frequency decision systems, free-form generation also introduces structural hallucinations and forces downstream code to rely on fragile regex or JSON parsers. Anyone who’s maintained that parser in prod knows exactly what I mean.

Prefill-only models solve this by running strictly the prefill forward pass. The model ingests the verbalized context, compresses the multi-token interaction history into a dense $d$ -dimensional hidden state $h$ , and stops. No autoregressive decoding, no KV cache pressure, inference down to milliseconds.

3. Shared First Principles, Divergent Customizations

GenRec and Jev share the same prefill-only hardware mechanics but diverge at the network head based on what they’re actually solving. GenRec is a retrieval problem over a large item database; Jev is a software control flow and deterministic agent routing problem.

Architectural Dimension	Netflix GenRec	TypeSafe AI Jev
Output Head	Custom catalog-aware scoring head replacing the language modeling vocabulary head.	Schema evaluator configured for bounded types.
Answer Space	Learned $d$ -dimensional embedding matrix ( $e_i$ ) for every item in the catalog.	Bounded dynamic candidate space defined at runtime (`Choice` up to 255 options, `Score`, or `Noul`).
Scoring Formulation	Dot product of hidden state ( $h$ ) and item embeddings, normalized via Softmax: $P(i \mid h) = \frac{\exp(h \cdot e_i)}{\sum_j \exp(h \cdot e_j)}$	Hidden state ( $h$ ) projected against candidate options to output calibrated probability distributions.
Training Objective	Two-phase training with reward-weighted ranking loss ( $L_{\text{ranking}}$ ) for long-term satisfaction: $L = \alpha L_{\text{ranking}} + \beta L_{\text{language}} + \gamma L_{\text{misc}}$	Reinforcement Learning for Calibrated Decisions (RLCD) to ensure probabilities reflect empirical frequencies.
Operational Target	Real-time personalization, retrieval, and ranking over massive content catalogs.	Low-latency agent tool orchestration, triage classification, deterministic conditional branching.

4. Potential Use Cases: The Prefill Decision Engine and Contextual Bandit Stack

The application I find most compelling is using prefill-only models as the sensory front-end for reinforcement learning frameworks, specifically Contextual Bandits (LinUCB, Thompson Sampling).

Contextual Bandits have solid mathematical guarantees for balancing exploration and exploitation to maximize cumulative reward. The problem is they’ve always been hard to deploy in practice. Three concrete limitations hold them back. First, bandits need stationary, low-dimensional numerical feature vectors ( $x \in \mathbb{R}^d$ ) and can’t natively parse raw text, conversational transcripts, or heterogeneous telemetry. Second, exploring across thousands of possible actions kills sample efficiency fast. Third, traditional bandits depend on immediate binary feedback and fail to optimize for complex multi-turn satisfaction signals.

A prefill-only model cleanly decouples Perception from Execution:

[Messy Unstructured Telemetry] (Logs, dialog histories, user actions)
               │
               ▼
   [Prefill Decision Model] (Jev / GenRec Backbone)
      • Semantic compression via Context Engineering
      • Sub-10ms single forward pass
      • Emits bounded, calibrated probabilities
               │
               ▼
       [Clean Context / Action Pruning]
               │
               ▼
   [Contextual Bandit Engine] (LinUCB / Thompson Sampling)
      • Manages explore/exploit trade-offs
      • Selects optimal action from bounded set
      • Updates policy via calibrated reward feedback

Dynamic Interface Personalization

In modern web applications, the optimal UI layout (promotional banner vs. product carousel vs. interactive widget) depends on session behavior that’s genuinely hard to reduce to a few numerical features. Raw session events like scroll depth, category clicks, and dwell time get verbalized and passed through a prefill model. The model emits a low-dimensional probability vector over latent user intent, for example [price_sensitive: 0.72, feature_researching: 0.21, urgent: 0.07]. A contextual bandit then ingests this vector as its clean context space and selects the layout treatment, balancing exploration of new UI variations against exploitation of known high-converting layouts.

Agent Tool Routing

Multi-agent systems often need an LLM to decide which downstream tool, API, or retrieval strategy to invoke. Using a full generative model to type out tool calls costs you 500ms or more and introduces constant formatting risk. Instead, the conversation state gets verbalized and a prefill model (like Jev) evaluates it against available tool signatures in a single forward pass, outputting calibrated routing scores. A Contextual Bandit uses those scores as its prior to manage exploration among API tiers. If a tool fails or provides low utility, the bandit rapidly learns alternative paths without touching model weights.

Cold-Start Catalog Exploration

Recommending brand-new items is a classic bandit problem where collaborative filtering fails by definition. Following GenRec’s approach, new items get rich verbalized metadata. The prefill model projects both the user’s interaction narrative and the new item’s semantic description into the shared hidden space ( $h$ ). The bandit then uses those similarity scores to prune the catalog down to 20 or 50 plausible candidates and runs exploration strategies (Thompson Sampling) exclusively over that bounded pool. It converges exponentially faster than an unconstrained bandit.

End-to-End Reward Modeling

Bandit policies optimizing solely for immediate clicks reliably produce clickbait behavior. Similar to GenRec’s reward models that assign scalar weights to prioritize long-term satisfaction, a prefill model can ingest post-interaction session transcripts and output a calibrated score from $0.0$ to $1.0$ for resolution quality. This continuous score becomes the reward signal for policy updates, letting the system optimize for durable satisfaction rather than surface-level engagement.

Summary

The transition from conventional ML pipelines to context engineering removes the maintenance burden of brittle tabular feature stores. The transition from autoregressive generation to prefill-only evaluation removes the latency and memory overhead of the decoding loop. Pair the semantic depth of prefill-only Transformers with the disciplined exploration of Contextual Bandits, and you get systems that can make complex, real-time decisions at a fraction of the operational cost of generative systems.

Neither of these ideas is brand new in isolation. What’s interesting is seeing them converge.

References

GenRec: An LLM-Backed Recommendation Ranker at Netflix - https://arxiv.org/html/2608.10257v2
Jev (TypeSafe AI): Introducing System One Models and Jev - https://typesafe.ai/blog/introducing-system-one-models-and-jev