How Routing Works

The Problem

There are dozens of LLMs available. Some are cheap and fast, others are powerful but expensive. Manually picking which one to use for each request is tedious and usually wrong.

The Solution

Fastlane trains a small neural network (ModernBERT head) to predict which model will perform best for a given prompt. This happens locally on your machine before any request leaves.

Prompt → Ranking Model → Best Provider → Response
  ↓           ↓              ↓
 "What     gemma2-9b-it   Groq API
  is 2+2?"  (cheap)        (fast)

Architecture

┌─────────────────────────────────────────┐
│           Fastlane Proxy                │
│  ┌─────────────┐    ┌──────────────┐   │
│  │ ONNX Router  │───│ Model Router │   │
│  │ (local)      │    │              │   │
│  └─────────────┘    └──────┬───────┘   │
│                            │            │
│  ┌────────┐ ┌────────┐ ┌────────┐     │
│  │ OpenAI │ │Anthropic│ │ Groq   │     │
│  └────────┘ └────────┘ └────────┘     │
└─────────────────────────────────────────┘

Request Flow

1. Request arrives at /v1/chat/completions with model: "fastlane-auto"
2. Router extracts the prompt text
3. ONNX model scores all available providers
4. Pareto optimizer picks the best based on your mode setting
5. Request is forwarded to the winning provider
6. Response is returned to the caller

Privacy

• The ranking model runs entirely on your machine
• Your prompts are only sent to the provider you choose
• No telemetry or usage data is collected