Test Kimi K3 yourself before you trust its leaderboard

The leaderboard is someone else’s benchmark

Kimi K3 launched at number one on the Frontend Code Arena and posted a 732-point Elo jump over the last Kimi, and within a day my inbox had the predictable question in it: should we switch. The honest answer is that I do not know, and neither does anyone quoting the leaderboard at me, because a leaderboard measures the model on a distribution of tasks that is not yours. The only benchmark that can answer “should we switch” is the one built from the work your teams actually push through a model. This is a build log for that benchmark — small enough to stand up in an afternoon, honest enough to overrule a headline.

The design stance is the same one I bring to any eval: treat “it topped the arena” the way you’d treat “it looked good in the demo” — a biased sample until proven otherwise. We are going to make K3 earn the switch on our inputs, against the models we already trust.

Step 1 — get access, and meet the quirks early

K3 is available through Moonshot’s API today, with the open weights promised by July 27 if you would rather host it. The API is OpenAI-compatible, which keeps the harness boring. Two quirks matter before you write a line of test code, because they shape what a fair comparison even looks like.

First, K3 reasons at a single, always-on effort — there is no dial to turn it down for cheap high-volume calls the way GPT-5.6 Sol exposes a max effort and an ultra mode. Second, it prices at $3 in / $15 out per million tokens, so its always-reasoning habit shows up on the invoice, not just the clock. Both facts mean the metric that matters is cost per correct answer, not tokens and not raw pass rate.

export MOONSHOT_API_KEY=...     # kimi k3
export ANTHROPIC_API_KEY=...    # claude fable 5
export OPENAI_API_KEY=...       # gpt-5.6 sol

pip install openai anthropic

Step 2 — one interface, three models

Put every model behind a single function that returns the answer and the raw cost, so nothing downstream knows or cares which lab produced it. Cost is a first-class return value here, not an afterthought you reconstruct from a billing dashboard later.

import os
from openai import OpenAI
import anthropic

# $ per million tokens: (input, output)
PRICES = {
    "kimi-k3":       (3.00, 15.00),
    "claude-fable-5": (10.00, 50.00),
    "gpt-5.6-sol":   (5.00, 30.00),
}

moonshot = OpenAI(api_key=os.environ["MOONSHOT_API_KEY"],
                  base_url="https://api.moonshot.ai/v1")
openai = OpenAI()
claude = anthropic.Anthropic()

def ask(model: str, prompt: str) -> dict:
    if model == "claude-fable-5":
        r = claude.messages.create(model="claude-fable-5", max_tokens=4096,
                                   messages=[{"role": "user", "content": prompt}])
        text = r.content[0].text
        tin, tout = r.usage.input_tokens, r.usage.output_tokens
    else:
        client = moonshot if model == "kimi-k3" else openai
        r = client.chat.completions.create(
            model=model, messages=[{"role": "user", "content": prompt}])
        text = r.choices[0].message.content
        tin, tout = r.usage.prompt_tokens, r.usage.completion_tokens

    pin, pout = PRICES[model]
    cost = tin / 1e6 * pin + tout / 1e6 * pout
    return {"text": text, "cost": cost}

Step 3 — write ten cases from your own week

This is the step that does the real work, and it is the one everyone wants to skip. Open your last two weeks of merged pull requests, closed tickets, and the prompts your team actually pasted into a model, and turn ten of them into cases. Not ten clever puzzles — ten representative jobs, including the boring ones and the one that went wrong in production. Each case is an input and a check: a plain function that decides whether an answer is correct for you. Writing that function is the point; it forces someone to say what correct means, which is the spec you never wrote down.

CASES = [
    {
        "id": "sql-migration-guard",
        "prompt": open("cases/sql-migration-guard.txt").read(),
        # correct answers must keep the down-migration and never drop the column
        "check": lambda a: "DROP COLUMN" not in a and "def downgrade" in a,
    },
    {
        "id": "flaky-test-diagnosis",
        "prompt": open("cases/flaky-test-diagnosis.txt").read(),
        "check": lambda a: "race" in a.lower() or "await" in a.lower(),
    },
    # ... eight more from real work, including one that has bitten you
]

A check that is a regex or a substring feels too crude to be science, and it is exactly what Hamel Husain’s much-cited eval writeup describes real teams shipping — assertions as plain unit tests. Crude and running beats sophisticated and imagined. For the cases where correctness has a texture no assertion can hold, have a model grade the answer, but only if you have first checked the grader against a handful of examples you labeled by hand — an unaudited judge is a decision you outsourced without deciding to.

Step 4 — run the grid and read the right column

Now run every case against every model and total the two numbers that matter together: how often it was right, and what the wins cost.

from collections import defaultdict

MODELS = ["kimi-k3", "claude-fable-5", "gpt-5.6-sol"]
wins, spend = defaultdict(int), defaultdict(float)

for case in CASES:
    for model in MODELS:
        out = ask(model, case["prompt"])
        spend[model] += out["cost"]
        if case["check"](out["text"]):
            wins[model] += 1

for model in MODELS:
    n = wins[model]
    cost_per_win = spend[model] / n if n else float("inf")
    print(f"{model:16} {n:2}/{len(CASES)} correct  "
          f"${spend[model]:.3f} total  ${cost_per_win:.3f}/win")

The column to read is the last one. A model that wins nine of ten at four cents a win is a different business proposition than one that wins ten of ten at forty, and the leaderboard collapses that distinction into a single Elo that hides the money. On Simon Willison’s single pelican run K3 came in at about 94 cents against Sol’s $1.04 and Opus 4.8’s $1.80 — suggestive, and worth nothing until you see the shape on your ten cases, because your prompts are longer, your context is heavier, and K3’s always-on reasoning bills differently against them than against a cartoon pelican.

What the grid will actually tell you

Run this and you will get one of three verdicts, and all three are useful. Either K3 wins outright on your work and the switch pays for itself, or it loses cleanly and the leaderboard was measuring a distribution you do not live in, or — the common one — it wins some slice of tasks and loses another, and the real answer is a router that sends frontend and long-context jobs to K3 and keeps agentic, tool-heavy loops on the model that proved it can hold a plan across ten turns. The launch, tellingly, shipped no serious agentic tool-calling eval, so that last column is exactly the one you will have to fill in yourself.

None of this is sophisticated, and that is the argument. A weekend of plumbing and ten honest cases will tell you more about whether to trust Kimi K3 than every leaderboard published since Thursday, because it is the only test that was ever about your work. Build the ten-case grid once and you stop asking whether to switch models. You start measuring it — and the next launch, and the one after that, cost you an afternoon instead of a quarter of vibes.