Model Ranking

The Ranking Model

Fastlane uses a ModernBERT classification head to score each provider for a given prompt. The model is:

• Trained locally on synthetic prompts (200 samples for MVP)
• Exported to ONNX for fast inference
• Runs on CPU in under 10ms per request

What It Learns

The model learns to map prompt characteristics to provider scores:

• Simple factual questions → cheap models (gemma2-9b, llama-3.1-8b)
• Code generation → mid-tier models (mixtral, llama-3.3-70b)
• Complex reasoning → strong models (gpt-4o, claude-sonnet-4)

Scoring

Each provider gets a raw score from the neural network. These are normalized to 0-1 range where:

• Quality score - How well the model handles this prompt type
• Cost score - How cheap the provider is (normalized against all candidates)
• Objective - Weighted combination based on your mode

Retraining

The baseline model is trained on synthetic data. For better accuracy:

1. Collect real usage data (opt-in sampling)
2. Run a judge model to score responses
3. Retrain the head on your actual traffic
4. Export new ONNX and restart

This creates a flywheel where the router gets better over time.