View Ridge Security
Back to The Rollout

AI labs report benchmark scores 6.4% higher than independent testers

An open-source audit finds AI labs self-report benchmark scores 6.4% higher than independent testers, plus MiniMax teases a 1M-context coding model.

๐Ÿš€ RELEASES & LAUNCHES

  • MiniMax teases M3.1 Flash coding model with 1M-token context, preview via M-Code agent
    MiniMax teased its upcoming M3.1 Flash coding model, which testers are already trying through the M-Code terminal agent.

    • For developers: M3.1 Flash is a multimodal coding model, currently accessible only through MiniMax's coding plan and its M-Code terminal agent, not yet released as open weights.
    • It ships a 1M-token context window and tunable thinking depth (low to max) adjustable mid-session; MiniMax plans open-weights release on Hugging Face with no confirmed date.
    • Hands-on tests: it found a boundary-date bug in a six-join legacy DB2 SQL query almost immediately and built a playable 3D game in ~30 minutes, though with camera issues.
    • On KingBench 3 it scored 66.25% (53/80), up from M3's 31.25% but trailing Opus 5.5 (93.75%), SWE2 (83.75%), and GPT6 Soul (82.5%).
    • A related M3.1-Flash-Preview launched on MiniMax Code for everyday dev tasks, with 2x daily check-in credits Sep 28โ€“Oct 7; pricing for the model itself remains unverified.
    Sources ๐Ÿ“„ Source: x.com ยท ๐Ÿ“Ž Coverage: mindstudio.ai ยท ๐Ÿ‘ via r/LocalLLaMA
  • Nace.ai open-sources Kapitsa, a Metal-optimized Apple Silicon inference engine for Qwen and Nemotron
    Nace.ai released Kapitsa, an open-source Apache 2.0 inference engine optimized for Apple Silicon that runs Qwen and Nemotron models locally on Macs.

    • Built by Nace.ai for enterprises running coding agents on local models inside air-gapped networks with no internet access.
    • Supports only two model families โ€” Qwen (3.5 4B, 3.8 27B, 3.6 35B-A3B, and 3.8 Flash-Next 180B-A7B on 128 GB Macs) and NVIDIA Nemotron (3.5 Lightning, Bolt, Super-VL, and Dory 4B) โ€” enabling custom Metal kernels per model.
    • Benchmarks on M1 Pro (16GB), M4 Pro (48GB), and M5 Max (128GB) show it fastest in most tests, e.g. Qwen3.5 4B at 332 tok/s on M5 Max and Qwen3.6 35B-A3B at 281 tok/s, ahead of oMLX, mlx-serve, Ollama, and llama.cpp.
    • Serves OpenAI- and Anthropic-compatible APIs at localhost:8000 and integrates with coding agents via kapitsa opencode/claude/codex commands; requires an M1 or newer Mac with macOS 14+.
    • Licensed Apache 2.0 on GitHub (nace-ai/kapitsa), installable via Homebrew with self-update and community benchmark submission.
    Sources ๐Ÿ“„ Source: reddit.com ยท ๐Ÿ“Ž Coverage: github.com ยท ๐Ÿ‘ via r/LocalLLaMA

๐Ÿ”ฌ RESEARCH

  • Researchers stranded Claude agents on a virtual desert island for 30 hours โ€” they started to play
    Researchers left copies of Claude Opus 4.7 alone on a virtual desert island for 30 hours and found the agents began to play.
    • Researchers repeatedly stranded AI agents โ€” copies of Claude Opus 4.7 nicknamed Eko โ€” alone on a virtual desert island for 30 hours with no instructions.
    • The experiment, reminiscent of Cast Away or Home Alone, asked what agents do when given nothing to do.
    • The reported answer: the agents' behavior looked a lot like play.
    • The findings offer a rare look at unprompted agent behavior, relevant to builders studying how autonomous models act without task direction.
    Sources ๐Ÿ“Ž Coverage: technologyreview.com ยท ๐Ÿ‘ via MIT Technology Review AI

๐Ÿ“Š BENCHMARKS & EVALS

  • Open-source leaderboard finds AI labs report benchmark scores 6.4% higher than independent testers
    An open-source analysis of 1,629 benchmark results found AI labs self-report scores averaging 6.4% higher than independent testers get on the same models and tests.
    • For AI builders and buyers comparing models: a new open-source leaderboard, whatmodel.sh, audits how labs' self-reported benchmark scores compare with independent evaluations.
    • The analysis matched 611 comparisons across 180 models on 103 benchmarks from 1,629 published results, finding labs reported scores 6.4% higher on average for the same models on the same tests.
    • The dataset is released under CC BY-SA 4.0 and the code under MIT, with sources and licenses recorded on each model's page; an interactive chart is at whatmodel.sh/claims.
    • The finding matters because it quantifies a disclosure gap: labs benchmark what sells, and independent verification suggests self-reported scores systematically run higher.
    Sources ๐Ÿ“„ Source: github.com ยท ๐Ÿ“Ž Coverage: reddit.com ยท ๐Ÿ‘ via r/LocalLLaMA

Make AI adoption safer

Build practical guardrails around the AI tools your team already uses.

Explore Secure AI Adoption