The Latency Floor Model

Paper · March 2, 2026 · CC BY 4.0

The Latency Floor Model

Predictive Performance Diagnostics for Real-Time AI Inference on Edge Accelerators

Amar Balutkar/DeviceNexus/DOI: 10.5281/zenodo.18840846/Zenodo #18840846

Read Paper →Engineering GuideCalculator Coming Soon

<5%

Prediction error on FP8 quantization impact

8

Optimization failures predicted before implementation

2

NVIDIA edge platforms validated (Orin & Thor)

3

Bottleneck regimes classified from model metadata alone

Abstract

The Latency Floor Model is a diagnostic framework that predicts the optimal AI inference optimization strategy from model metadata and hardware specifications alone, no profiling required. Compute three numbers: R₁ (weight bandwidth floor), R₂ (KV cache bandwidth floor), R₃ (compute throughput floor). Their relationship tells you whether to quantize weights, fuse CUDA graphs, or increase batch size, before writing a single line of optimization code. Validated on real-world LLM serving and robotics VLA pipelines, the framework correctly classified bottleneck regimes and predicted quantization outcomes within 5% of measured values.

Core Equations

$R_1 = \frac{P \cdot b}{B_w} \qquad \text{(weight-read floor)}$

$R_2 = \frac{L \cdot 2 \cdot H \cdot T \cdot d_h \cdot b_{kv}}{B_w} \qquad \text{(KV-cache floor)}$

$R_3 = \frac{F}{C_{peak}} \qquad \text{(compute floor)}$

$\tau = \max(R_1 + R_2,\; R_3) + \varepsilon$

The overhead ratio $\tau/(R_1+R_2)$ is the primary diagnostic. Compute it once and the optimization strategy follows directly:

Ratio	Regime	Action
$\approx 1.0\text{–}1.5$	Bandwidth-bound	Quantize weights
$> 2.0$	Overhead-bound	CUDA graphs, compilation
$\tau/R_3 \approx 1.0\text{–}1.5$	Compute-bound	TensorRT, batching

Key Findings

Validated on two workloads across Jetson Orin and Jetson Thor: an LLM serving pipeline and a robotics VLA pipeline.

• Predicted FP8 quantization impact within 5% of measured values, without running a profiling iteration
• Correctly predicted the failure of 8 optimization techniques before implementation, saving engineering time on dead ends
• Identified a phase transition where further quantization yields diminishing returns as the bottleneck shifts from bandwidth to compute
• Revealed heterogeneous bottleneck regimes within a single multi-stage pipeline: prefill, decode, and vision encoding each required different optimization strategies

Latency Floor Calculator Coming Soon

Plug in your model’s parameter count and hardware specs to find your latency floor in seconds. Interactive calculator launching shortly.