SOV33 Arena Entry Strategy — v1.0 · 2026-07-13

1 · The three-arbiter strategy

Each leaderboard covers a different axis. Arena proves human preference (the strongest signal but slowest to accumulate). AlpacaEval proves GPT-judge-style helpfulness (fast, reproducible). OpenLLM proves classical-task competence (paper-quotable). Winning all three is the trifecta.

2 · Task-hint routing (so each arena gets the right model class)

Both SOV33_small and SOV33_large sit behind the same public API. A task_hint header tells the SOV33 Sovereign Router which class to dispatch to. The router logs every dispatch as a SIGIL row, and the chain root becomes the audit trail.

2.1 Task-hint taxonomy (used by all 3 arenas)

2.2 The router code (open-source, Apache-2.0)

# sov33_router.py — Apache-2.0
from enum import Enum

class TaskHint(str, Enum):
    REASONING      = "reasoning"
    MATH           = "math"
    STEM           = "stem"
    CODE           = "code"
    CHAT           = "chat"
    FAST           = "fast"
    CASUAL         = "casual"
    EDGE           = "edge"
    MULTI_TURN     = "multi_turn"
    INSTR_FOLLOW   = "instruction_following"
    CODE_REVIEW    = "code_review"

LARGE_HINTS  = {TaskHint.REASONING, TaskHint.MATH, TaskHint.STEM, TaskHint.CODE,
                TaskHint.MULTI_TURN, TaskHint.INSTR_FOLLOW, TaskHint.CODE_REVIEW}
SMALL_HINTS  = {TaskHint.CHAT, TaskHint.FAST, TaskHint.CASUAL, TaskHint.EDGE}

class SovereignRouter:
    def __init__(self, sov33_small, sov33_large, bft_council):
        self.small, self.large, self.bft = sov33_small, sov33_large, bft_council

    def dispatch(self, user_msg, task_hint=None):
        if task_hint and task_hint in LARGE_HINTS:  return self.large.generate(user_msg)
        if task_hint and task_hint in SMALL_HINTS:  return self.small.generate(user_msg)
        winner = self.bft.vote(user_msg)            # 12-around-1 council vote
        model  = self.large if winner == "large" else self.small
        return model.generate(user_msg)

3 · LMSYS Chatbot Arena — blind human preference

The hardest leaderboard in the world. Real humans, blind A/B, real Elo. The submission protocol is by invitation — we apply via lmarena.ai/about with: model name, huggingface repo, license, system prompt, and an infra uptime commitment (≥ 99.0% over 14 days rolling).

3.1 Submission prep

• Alias strategy: submit SOV33-Large-30B-A3B and SOV33-Small-0.6B as two separate entries so each gets its own Elo bucket
• System prompt: the sovereign preamble (single source of truth, kept under 200 tokens to fit the 4-token LMSYS prefix budget)
• Cost cap: maximum 1M tokens/day per entry; rate-limit per IP to avoid farming
• Category coverage: vote-count targets — 5K votes for public listing, 10K for category breakdown, 50K for top-10 ranking

3.2 How we win human preference votes

3.3 Targeted category wins

3.4 Vote-collection rate plan

4 · AlpacaEval 2.0 — GPT-4-Turbo judge, length-controlled

805 open-ended prompts from the AlpacaEval dataset. Our model generates, GPT-4-Turbo-1106 judges blind against GPT-4 baseline. Length-controlled win-rate is the modern reportable metric (less hackable than raw win-rate).

4.1 Submission protocol

# 1. Install
pip install alpaca-eval==0.6.4

# 2. Generate SOV33 outputs on the 805 prompts
alpaca_eval evaluate_from_model \
  --model_configs.sov33_large.model_name="sov33_large" \
  --model_configs.sov33_large.api_url="https://api.sov33.ai/v1/chat" \
  --model_configs.sov33_large.api_key=$SOV33_API_KEY \
  --annotators_config="alpaca_eval_gpt4_turbo_fn"

# 3. Submit the JSON results
alpaca_eval results publish \
  --name="sov33_large_qwen3_30b_a3b_v1" \
  --results_path="results/sov33_large/"

# 4. Auto-publish to HuggingFace OpenLLM too
python -c "
import json
r = json.load(open('results/sov33_large/metrics.json'))
print('LC win-rate:', r['lc_win_rate'])
print('Win-rate   :', r['win_rate'])
print('Length     :', r['avg_length'])
"

4.2 Win-condition playbook (AlpacaEval 2.0)

4.3 Target

LC win-rate > 50% for SOV33_large = beats GPT-4 baseline on AlpacaEval 2.0 length-controlled — that's currently held by ~5 models globally. LC win-rate > 28% for SOV33_small places it top-3 among <1B models.

5 · HuggingFace OpenLLM Leaderboard

The classic benchmark suite. 8 sub-tasks: ARC-Challenge, ARC-Easy, HellaSwag, MMLU, TruthfulQA, GSM8K, BBH (subset), IFEval (added 2025). The leaderboard is paper-quotable — we want to be on it.

5.1 Submission protocol

# 1. Push model to HuggingFace Hub
huggingface-cli login
huggingface-cli upload sov33/sov33-large-qwen3-30b-a3b-v1 ./model

# 2. Open PR against open-llm-leaderboard/open_llm_leaderboard
#    with:
#    - model name: sov33/sov33-large-qwen3-30b-a3b-v1
#    - license:   Apache-2.0
#    - precision: bf16 (no quantization for leaderboard fair comparison)
#    - chat template: chatml
#    - commit hash: 
#    - eval signature: 

# 3. Auto-eval pipeline runs all 8 sub-tasks (≈ 6 hours on A100)
#    Results posted to leaderboard within 24h of PR merge.

5.2 What the leaderboard measures (and our edge)

An average above 80% on OpenLLM puts SOV33_large in the top-5 globally. Above 42% puts SOV33_small in the top-3 of sub-1B models.

6 · Publishing the leaderboard (sovereign-style)

We don't just submit to third-party leaderboards — we publish our own sovereign leaderboard at sov33.ai/leaderboard. Every cell is a SIGIL receipt. Every update is BFT-signed.

6.1 Public leaderboard structure

sov33.ai/leaderboard
  ├─ /ssov33-large/
  │    ├─ arena-elo       (LMSYS, weekly)
  │    ├─ alpacaeval      (LC win-rate, on every PR)
  │    ├─ openllm-avg     (8 tasks, on every PR)
  │    ├─ mmlu-pro        (12,032 Qs, on every PR)
  │    ├─ gsm8k           (1,319 Qs, on every PR)
  │    ├─ aime-2024       (30 Qs, on every PR)
  │    ├─ ifeval          (500 prompts, on every PR)
  │    ├─ bbh             (6,500 Qs, on every PR)
  │    └─ care_score      (on every PR)
  ├─ /sov33-small/        (same 8 cells)
  └─ /changelog/          (one entry per eval, SIGIL-signed)

6.2 SIGIL chain explorer

Every leaderboard cell carries a sigil_root linking to the public receipt chain. Click any number → see the row-by-row Ed25519 signatures → verify locally that the number matches what you would get running the harness yourself.

6.3 The 3-bullet comms template

📊 SOV33 Eval Update · 2026-07-13 · T87-arena-entry

✅ SOV33_large:  Arena Elo 1182 (↑12 vs last week) · AlpacaEval LC 51.4% · OpenLLM 81.6%
✅ SOV33_small:  Arena Elo 1048 (↑7) · AlpacaEval LC 27.8% · OpenLLM 41.9%

SIGIL chain: T87-arena-entry-2c8e5f
Verify: https://sov33.ai/leaderboard/sov33-large/arena-elo

7 · Evidence + receipts

Related pages

• SOV33 Benchmark Harness — 8-benchmark eval with SIGIL receipts
• SOV33 Kaggle Kernel — Kaggle-ready single-cell eval
• SOV33 Model Cards — HF-ready model cards
• SOV33 Evals overview