vLLM v0.32.0 adds structured decisions endpoint, FP8 KV cache

🚀 RELEASES & LAUNCHES

• vLLM v0.32.0 ships 814 commits with structured decisions endpoint and low-precision KV cache
  vLLM released v0.32.0, adding a structured decision-serving endpoint, generic HF parsers, NVFP4/FP8 KV cache across GPUs, and Model Runner V2 features.

  • Aimed at LLM inference/serving users: release includes 814 commits from 363 contributors (121 new).
  • New POST /v1/systemone endpoint answers choice, noul, and score questions over structured state, served by default and runs on Cloudflare/clef-omni, LiquidAI/d1-3B, and DiffusionGemma.
  • New generic hf tool and reasoning parsers read the checkpoint's response_template, removing the need for model-specific parsers.
  • NVFP4 KV cache now runs on SM8x, SM12x, SM90 and MiniMax-M3 sparse attention on SM100; GLM-5.3-Flash gains up to 31% more output throughput via fp8_ds_mla on SM90.
  • New models include EmbeddingGemma 2 omni-modal embeddings, MiniCPM-V 4.7, and Nemotron 3.5 ASR; Python 3.10 dropped and several security fixes (CORS, ZMQ socket exhaustion) landed.
  • Matters for builders: --api-server-count >1 yields 43-85% more req/s on short requests, and GLM-5.x FP8 KV cache halves KV memory per rank with 2.3-5.5% more throughput.
  Sources 📎 Coverage: github.com · 👁 via vLLM releases

• New IQ2_K–IQ6_K quant types proposed for mainline llama.cpp in new PR
  A llama.cpp pull request proposes adding new IQ2_K through IQ6_K quantization types that match Q-quant file sizes while substantially reducing quantization error.

  • For local LLM users running GGUF models in llama.cpp, a new PR (ggml-org/llama.cpp #30715) proposes IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K quant types.
  • The quants come from ikawrakow's ik_llama.cpp fork, where they were introduced to cut the higher quantization error seen on LLaMA-3-era models (~1 bit-per-weight more information than LLaMA-2).
  • Reported results: file sizes same as Q-quants, KLD/perplexity error substantially lower (e.g., IQ5_K error 2.1x lower than Q5_0 and 40% lower than Q5_K_S), with slightly slower inference and slow quantization creation.
  • Test models for evaluation are published on Hugging Face (eaddario/IQ2_K-IQ6_K-Test-Models).
  • The IQ_K designs use small lookup tables that fit in SIMD registers, giving near-linear-mapping CPU performance unlike codebook-based i-quants.
  Sources 📄 Source: github.com · 📎 Coverage: reddit.com · 👁 via r/LocalLLaMA

• Blockway ships Agens Volundr 32B v1, fixing Preview flaws with 262K context in 16 GB
  Blockway released Agens Volundr 32B v1, fixing repetition loops and adding session continuation to its 32B agentic model.

  • Aimed at local/agent users: the model targets reasoning, coding, and agentic work, with GGUF builds for llama.cpp, vLLM, Ollama and other local apps.
  • v1 fixes Preview issues: SWE-bench repetition loops dropped from 36/50 to 0/50, and reasoning steps are no longer cut at 32K (median step 120 tokens, longest 12K).
  • An MIT-licensed engine runs the model with 262K context (measured to 239K) within a 16 GB memory budget.
  • SGLang build adds session continuation, reusing model state instead of re-prefilling — follow-up turns are 2.9x faster (1.6 s median).
  • The 32.7B-parameter model is Apache-2.0 licensed, speaks English, Chinese, and Cantonese, and accepts text, images, and tool calls.
  Sources 📄 Source: huggingface.co · 📎 Coverage: reddit.com · 👁 via r/LocalLLaMA