120B Parameters, 12B Active — The New Economics of AI Inference

When NVIDIA unveiled Nemotron 3 Super at GTC 2026 on March 11, it didn't just release another large language model. It introduced a fundamentally different approach to scaling AI: a 120-billion parameter hybrid Mamba-Transformer Mixture-of-Experts model that activates only 12 billion parameters per inference pass. The result? 5x higher throughput, 2x improved accuracy over its predecessor, and a 60.47% score on SWE-Bench Verified — obliterating GPT-OSS's 41.90%.

For developers building agentic AI systems — autonomous agents that reason, use tools, and execute multi-step workflows — Nemotron 3 Super represents a paradigm shift. It's open-weight, comes with full training recipes, and runs across every major inference platform. Here's everything you need to know to put it to work.

The GTC 2026 Nemotron Agent Stack

Nemotron 3 Super didn't arrive alone. NVIDIA launched a complete agent stack at GTC 2026, purpose-built for the agentic AI era. The lineup includes Nemotron 3 Nano (4B parameters) for on-device and consumer hardware, Nemotron 3 Content Safety (4B) for multimodal content screening at ~84% accuracy, and Nemotron 3 VoiceChat (12B) for sub-300ms latency full-duplex voice conversations.

Perhaps more significant is the Nemotron Coalition — a first-of-its-kind collaboration with Mistral AI, Perplexity, LangChain, Cursor, Black Forest Labs, Reflection AI, Sarvam, and Thinking Machines Lab. This coalition will develop the foundation for the upcoming Nemotron 4 family, signaling NVIDIA's aggressive push to become the gravitational center of open-source AI.

The entire Nemotron 3 Super pipeline is open: over 10 trillion tokens of pre- and post-training datasets, 15 reinforcement learning environments, full evaluation recipes, and weights on Hugging Face — all under the permissive NVIDIA Nemotron Open Model License.

Architecture Deep Dive: Three Innovations in One Model

What makes Nemotron 3 Super architecturally unique is the convergence of three distinct innovations that have never been combined at this scale.

Hybrid Mamba-Transformer Backbone

Traditional Transformers suffer from quadratic complexity with respect to sequence length — doubling your input more than doubles your compute cost. Nemotron 3 Super deploys Mamba-2 layers (based on State Space Models) for the majority of sequence processing, achieving linear-time complexity. Transformer attention layers are strategically interleaved only where precise associative recall is required. The result: 4x improvement in memory and compute efficiency compared to pure attention architectures.

This hybrid approach is what enables the model's native 1-million-token context window — without the prohibitive costs that would make such context lengths impractical for production workloads.

Latent MoE: More Experts, Same Cost

Standard Mixture-of-Experts models route tokens to a subset of expert networks. Nemotron 3 Super's Latent MoE compresses token embeddings before routing, meaning the model can activate 4x more specialist experts for the same inference cost. In practice, this translates to finer-grained specialization — distinct experts handle Python generation versus SQL queries versus natural language reasoning, all within a single forward pass.

Multi-Token Prediction (MTP)

Rather than predicting one token at a time, MTP predicts multiple future tokens simultaneously in a single forward pass. This delivers up to 3x wall-clock speedups for long-form generation and enables built-in speculative decoding without requiring a separate draft model. The shared-weight design across prediction heads maintains training stability while dramatically accelerating inference.

Deployment Guide: vLLM, SGLang, TensorRT-LLM, and Ollama

NVIDIA provides official deployment cookbooks for three major inference engines, plus community support through Ollama for local experimentation.

vLLM (High-Throughput Serving)

vLLM offers continuous batching and streaming, ideal for high-concurrency API serving. The default configuration targets 4x H100 GPUs:

python -m vllm.entrypoints.openai.api_server \
  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 131072

SGLang (Agent-Optimized)

For multi-agent tool-calling workloads, SGLang is the recommended engine. It supports tensor parallelism (--tp), expert parallelism (--ep), tool-call parsing, reasoning parsers, and EAGLE-based speculative decoding — all critical for agent orchestration where function calls and structured outputs are the norm.

TensorRT-LLM (Production-Grade)

For lowest-latency production deployment, TensorRT-LLM includes dedicated Latent MoE kernels that are specifically optimized for the model's architecture. On Blackwell GPUs (B200) with NVFP4 precision, this delivers 4x faster inference compared to FP8 on H100.

Ollama (Local Experimentation)

You can run Nemotron 3 Super locally with a 4-bit quantized version requiring approximately 64-72GB of RAM/VRAM. A Mac Studio with M2 Ultra or a workstation with dual RTX 4090s can handle it:

ollama run nemotron-3-super

Full FP16 precision requires ~240GB VRAM, making cloud or multi-GPU setups necessary for unquantized deployment.

Building Multi-Agent Systems with Nemotron 3 Super

Nemotron 3 Super addresses the two fundamental bottlenecks in multi-agent AI systems.

Context explosion is the first. When agents exchange messages, invoke tools, and accumulate reasoning traces, context grows rapidly. Most models either truncate history (losing critical information) or slow to a crawl. Nemotron 3 Super's 1M-token context window with linear-time Mamba processing means agents maintain full workflow state without goal drift — even across extended multi-step tasks like generating a 10-slide presentation that requires coordinating between code execution, image generation, and layout decisions.

Cost-efficient tiering is the second. NVIDIA recommends a hierarchical agent pattern: simple tasks (routine merge requests, basic lookups) are handled by Nemotron 3 Nano, while complex reasoning tasks (architectural decisions, multi-step research) escalate to Nemotron 3 Super. This pattern mirrors how human teams work — not everyone needs to be a senior engineer for every task.

The approach is already proving its worth: NVIDIA's AI-Q research agent, powered by Nemotron 3 Super, claimed the #1 position on both DeepResearch Bench and DeepResearch Bench II leaderboards — benchmarks that specifically measure multi-step research capability.

Benchmark Performance: Setting New Standards

Here's how Nemotron 3 Super stacks up against comparable models:

| Metric | Nemotron 3 Super | GPT-OSS-120B | Qwen3.5-122B | |--------|-----------------|--------------|---------------| | Inference Throughput (8k/16k) | Baseline | 2.2x slower | 7.5x slower | | SWE-Bench Verified | 60.47% | 41.90% | — | | RULER (1M tokens) | 91.75% | 22.30% | — | | PinchBench (Agentic) | 85.6% | — | — |

The RULER benchmark result is particularly striking: 91.75% versus 22.30% at 1-million-token context length demonstrates the dramatic advantage of the hybrid Mamba-Transformer architecture for long-context tasks. This isn't an incremental improvement — it's a different capability class.

Fine-Tuning and Customization

NVIDIA has released the complete training pipeline, making Nemotron 3 Super one of the most reproducible frontier-class models available:

• Pretraining: 25 trillion tokens (10 trillion unique curated tokens) across a two-phase curriculum on a Slurm GPU cluster
• Supervised Fine-Tuning: 7 million samples from a 40-million-sample post-training corpus covering reasoning, coding, instruction-following, safety, and multi-step agent tasks
• Reinforcement Learning: 1.2 million+ environment rollouts across 21 configurations using NeMo Gym

For practical fine-tuning, two paths are supported: LoRA SFT via NeMo Megatron-Bridge or NeMo Automodel for efficient adaptation, and GRPO/DAPO reinforcement learning via NeMo RL for behavior alignment. Amazon Bedrock reinforcement fine-tuning support is coming soon, enabling domain-specific adaptation for legal, healthcare, and finance applications.

Ecosystem and Availability

Nemotron 3 Super is already available across an impressively broad ecosystem. Cloud platforms include Google Cloud Vertex AI, Microsoft Azure, Oracle Cloud, with Amazon Bedrock coming soon. Inference providers include Perplexity (Pro), OpenRouter, DeepInfra, Fireworks AI, Together AI, Modal, Baseten, Cloudflare Workers AI, FriendliAI, and more.

For enterprise on-premises deployment, the model ships as an NVIDIA NIM microservice with integrations for Dell Enterprise Hub and HPE Agents Hub. Early adopters already in production include Perplexity, CodeRabbit, Factory, Greptile, Palantir, Siemens, Dassault Systèmes, and Cadence.

Practical Recommendations

Getting started: The fastest path is through build.nvidia.com for API access. For local experimentation, start with Ollama's 4-bit quantized version on a 64GB+ machine.

For multi-agent workflows: Choose SGLang as your inference engine for optimized tool-calling. Implement the Nano-Super tiered pattern to balance cost and capability. Use the 1M context window strategically — preload entire codebases or document sets rather than relying on RAG for everything.

For production deployment: If you have access to Blackwell GPUs (B200), NVFP4 precision with TensorRT-LLM's Latent MoE kernels delivers maximum performance. On Hopper hardware (H100), FP8 with vLLM's continuous batching remains excellent.

For fine-tuning: Start with LoRA SFT on your domain-specific data before attempting full RL alignment. The open training recipes mean you can inspect exactly how NVIDIA trained the base model and adapt accordingly.

Looking Ahead

Nemotron 3 Super isn't just a model — it's an infrastructure layer for the agentic AI era. The combination of hybrid Mamba-Transformer architecture, Latent MoE, and Multi-Token Prediction proves that open models can compete with — and in key benchmarks, surpass — proprietary alternatives. With the Nemotron Coalition developing the next generation, expanding cloud integrations, and a vibrant community pushing domain-specific adaptations, the trajectory is clear. If you're serious about building agentic AI systems in 2026, Nemotron 3 Super should be at the top of your evaluation list.