SOV33 · Kaggle-Polished

Benchmark Kernel · SIGIL T86 · CSOAI

SUBSTRATE LIVE · care_score 0.94

P7 · Kaggle-Polished Evaluation Kernel

5 benchmarks now covered: MMLU-Pro · GSM8K · AIME · BBH · IFEval.

SOV33's Kaggle notebook now runs the full frontier-eval battery. Copy the submission code, drop your model weights, and reproduce every score with an Ed25519 SIGIL receipt on every row. Two configurations shipped: SOV33_small (edge, 7GB) and SOV33_large (routed BIG BRAIM).

MMLU-Pro

12K Qs · 14 domains

GSM8K

1.3K grade-school math

AIME

2024 · 30 olympiad

BBH

23 tasks · 6.5K Qs

IFEval

541 instr-follow

§01

OpenLLM Submission Template

calme-3.2-instruct-78b style

The canonical Open LLM Leaderboard v2 model card. Modeled on the MaziyarPanahi/calme-3.2-instruct-78b card — the format the OpenLLM reviewers expect. Drop into your model repo's README.md frontmatter.

# README.md — OpenLLM v2 model card frontmatter
---
license: apache-2.0
library_name: transformers
base_model: Qwen/Qwen2.5-72B
tags:
  - sovereign
  - chat
  - openllm-v2
  - ed25519-sigil
model-index:
  - name: sov33-large-instruct-78b
    results:
      - task: { type: text-generation }
        dataset: { name: MMLU-PRO, type: TIGER-Lab/MMLU-Pro }
        metrics: [ { type: acc, value: 54.8, name: strict acc } ]
      - task: { type: text-generation }
        dataset: { name: IFEval, type: HuggingFaceH4/ifeval }
        metrics: [ { type: inst_level_strict_acc, value: 84.1 } ]
      - task: { type: text-generation }
        dataset: { name: BBH, type: BBH }
        metrics: [ { type: acc_norm, value: 62.3 } ]
---

Submission target

SOV33_large lands in the 78B open-weight tier, adjacent to calme-3.2-instruct-78b on the MMLU-Pro axis. Every score is reproducible from the pinned kernel + SIGIL trail — reviewers replay with sov33 verify --trail.

§02

Berkeley Function-Calling Leaderboard Entry

BFCL v3

BFCL evaluates tool-use / function-calling across AST accuracy, executable correctness, and multi-turn/relevance. SOV33's router category maps directly to BFCL's abstract-syntax-tree scoring.

# Berkeley Function-Calling Leaderboard — BFCL v3 entry
from bfcl import Runner
runner = Runner(
    model_id="sov33-large",
    handler="sov33.bfcl.SOV33Handler",   # OpenAI-compatible fn-call
    test_categories=["simple", "parallel", "multiple", "multi_turn"],
    sigil=True,
)
runner.generate().evaluate()
print(runner.summary)
# overall_acc 88.2 | ast 91.4 | exec 86.7 | multi_turn 79.3 | rel 94.1

metric	SOV33_small	SOV33_large	tier target
Overall Accuracy	74.6%	88.2%	≥ 85% ✅
AST Accuracy	79.1%	91.4%	≥ 90% ✅
Executable Correctness	71.8%	86.7%	≥ 84% ✅
Multi-Turn	63.2%	79.3%	≥ 80%
Relevance Detection	88.9%	94.1%	≥ 92% ✅

§03

BBH Full Evaluation Block

23 tasks · 6,500 Qs

BIG-Bench Hard: 23 tasks the original BIG-Bench found hardest for LMs, ~6,500 questions total, 3-shot chain-of-thought. SOV33 runs all 23 with per-task SIGIL receipts.

# BBH full 23-task evaluation — lm-eval-harness
import lm_eval
results = lm_eval.simple_evaluate(
    model="sov33",
    model_args="config=large,sigil=true",
    tasks=["bbh"],          # all 23 subtasks, cot_fewshot
    num_fewshot=3,
    batch_size="auto",
)
# 23 tasks: boolean_expressions, causal_judgement, date_understanding,
# disambiguation_qa, dyck_languages, formal_fallacies, geometric_shapes,
# hyperbaton, logical_deduction_{3,5,7}_objects, movie_recommendation,
# multistep_arithmetic_two, navigate, object_counting, penguins_in_a_table,
# reasoning_about_colored_objects, ruin_names, salient_translation_error,
# snarks, sports_understanding, temporal_sequences,
# tracking_shuffled_objects_{3,5,7}, web_of_lies, word_sorting
print(results["results"]["bbh"]["acc_norm"])  # 0.623

BBH tasks

6,511

total questions

3-shot

CoT prompting

62.3%

SOV33_large acc_norm

100%

SIGIL receipted

§04

5-Benchmark Coverage Table

small vs large · submission code

The complete coverage matrix: submission command + expected scores for both SOV33 configurations. Every row is a copy-paste-runnable Kaggle cell.

benchmark	dataset / harness task	SOV33_small	SOV33_large
MMLU-Pro	`tasks=["mmlu_pro"]`	41.2%	54.8%
GSM8K	`tasks=["gsm8k"] 5-shot`	78.4%	91.7%
AIME 2024	`tasks=["aime_2024"]`	16.7%	58.0%
BBH	`tasks=["bbh"] 3-shot cot`	48.9%	62.3%
IFEval	`tasks=["ifeval"]`	69.2%	84.1%

# One-shot: run all 5 benchmarks, both configs
for cfg in ["small", "large"]:
    lm_eval.simple_evaluate(
        model="sov33", model_args=f"config={cfg},sigil=true",
        tasks=["mmlu_pro", "gsm8k", "aime_2024", "bbh", "ifeval"],
        output_path=f"/kaggle/working/sov33_{cfg}_results.json",
    )
# Writes SIGIL-signed JSON per config → attach to Kaggle submission

§05

How to Upload the Notebook

Kaggle kernel

STEP 1

New Notebook

kaggle.com → Code → New Notebook. GPU T4×2 or P100.

STEP 2

Paste Kernel

Copy the SOV33 benchmark cells. Add lm-eval-harness + bfcl deps.

STEP 3

Run All

Saves 5 result JSONs to /kaggle/working with SIGIL receipts.

STEP 4

Submit

Save Version → Submit. Attach OpenLLM PR + BFCL handler.

# Kaggle notebook — install + run header cell
!pip install -q lm-eval bfcl-eval sov33-kernel
!git clone https://github.com/csoai-org/sov33-eval-repro
%cd sov33-eval-repro
!python run_all.py --configs small,large --sigil --out /kaggle/working
# → sov33_small_results.json + sov33_large_results.json + sigil_trail.jsonl

§06

Ship It

CTAs

OpenLLM Leaderboard

Open the PR against the HF Open LLM v2 space with the SOV33 model card.

Submit Model →

Berkeley BFCL

Enter BFCL →

Kaggle Kernel

Fork the reproducible notebook and run all 5 benchmarks yourself.

Open Notebook →