Speed up agentic AI with NVIDIA Nemotron models on Amazon Elastic Kubernetes Service and SageMaker HyperPod Elastic Kubernetes Service.
NVIDIA Nemotron 3.5 Super VL powers multimodal agents with long context and hybrid Mamba-Transformer on Amazon EKS, Auto Mode, and SageMaker HyperPod.
Key Takeaways
- NVIDIA Nemotron 3.5 Super VL is an open 120B-parameter multimodal mixture-of-experts model that activates 12B parameters per token and supports text, image, document, and video understanding with up to a 1M-token context.
- It is built for agentic AI workflows that combine reasoning, tool use, and visual understanding, including coding, web research, document intelligence, cybersecurity, and video analysis.
- Its hybrid Mamba-Transformer MoE architecture helps make long-context serving more affordable by keeping Mamba state memory constant during generation and supporting native speculative decoding.
- The model can be deployed on Amazon EKS, Amazon EKS Auto Mode, and SageMaker HyperPod EKS using shared Kubernetes manifests, with minimum GPU configurations ranging from 16 H100 GPUs to 8 B200 GPUs depending on precision and hardware.
- AWS deployment guidance covers NVIDIA Dynamo serving, cluster validation with NCCL all-reduce benchmarks, and performance testing with NVIDIA aiperf to measure latency, throughput, and capacity needs.
Meta description: NVIDIA Nemotron 3.5 Super VL, a 120 billion parameter multimodal mixture-of-experts model, now runs on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS.
Tags: NVIDIA Nemotron 3.5 Super VL, AWS EKS, multimodal AI, mixture-of-experts, SageMaker HyperPod
Featured image: Abstract friendly illustration of a multimodal AI model connecting text, image, document, and video flows into Kubernetes and SageMaker HyperPod infrastructure, with no real people, no logos, and no readable text.
NVIDIA Nemotron 3.5 Super VL is for teams building AI agents that must reason over text while also understanding images, documents, and video. The model has joined NVIDIA Nemotron 3 Nano, Nano Omni, Super, Ultra, and Nemotron 3.5 Lightning models already available on AWS. The accompanying AWS deployment guidance explains its architecture, use cases, minimum GPU configurations, and reference deployment patterns on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon EKS Auto Mode, and Amazon SageMaker HyperPod EKS.
What the model changes for multimodal agents
NVIDIA Nemotron 3.5 Super VL is an open multimodal model distilled from Nemotron 3 Ultra. NVIDIA publishes the weights, training datasets, and post-training recipes, so teams can customize the model for domain-specific workloads and run it on infrastructure they control.
The model is a 120 billion-parameter mixture-of-experts system that activates 12 billion parameters per token. It combines reasoning and tool use with text, image, document, and video understanding, allowing one model to serve both text-only and visual workloads. Its architecture uses hybrid Mamba-2, mixture-of-experts, and attention layers together with a RADIO vision encoder. It supports input from text, image, document, and video, produces text output, and offers a context length up to 1 million tokens. BF16 and NVFP4 checkpoints are available.
According to the model’s technical report, NVIDIA reports 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations, and 80.8 percent on the CharXiv chart-reasoning benchmark.
Why the architecture matters for long context
The model uses a hybrid Mamba-Transformer mixture-of-experts architecture that interleaves Mamba-2, attention, and expert layers. Sparse routing activates only 12 billion of the 120 billion parameters for each token. The result is that compute per token resembles a much smaller model, while GPU memory still stores the full weight set.
Mamba-2 layers carry a constant-size state during generation. That means their memory does not expand with sequence length in the same way a key-value cache does, which helps keep long-context work affordable. A recursively shared Multi-Token Prediction head also supports native speculative decoding.
Where teams can use Nemotron 3.5 Super VL
The model is positioned for agent workflows that combine reasoning with visual understanding. The published use cases include:
- Coding: test and validate code, analyze errors, and recommend fixes.
- Web research: cross-reference and verify sources returned by search tools.
- Document intelligence: read tables, charts, and page layouts to produce summaries.
- Cyber security: identify vulnerabilities and triage threats.
- Video understanding: search, review content, and support incident analysis.
Nemotron 3.5 Super VL integrates with major open agent frameworks and orchestration systems through NVIDIA reference implementations, cookbooks, and the NVIDIA Agent Toolkit. The full list of supported frameworks is available on the NVIDIA developer blog.
GPU floors and AWS deployment architecture
For vLLM serving, the minimum GPU configurations are:
| GPU model | Minimum count, BF16 | Minimum count, NVFP4 |
|---|---|---|
| B200/GB200 | 8 | 2 |
| B300/GB300 | 4 | 1 |
| H200 | 8 | 4 |
| H100 | 16 | 8 |
The reference deployment pattern runs on standard Amazon EKS, Amazon EKS Auto Mode, and Amazon SageMaker HyperPod EKS. With Amazon EKS Auto Mode, AWS manages scaling, GPU, and Elastic Fabric Adapter support in the cluster on the customer’s behalf. The Kubernetes manifests used to deploy the model are the same for each orchestrator option.
A typical cluster has a control plane hosting the Kubernetes API and a data plane made of CPU and GPU nodes. CPU nodes run system and test pods, while GPU nodes run the model. Amazon Elastic Container Registry stores the container images, a shared Amazon FSx for Lustre volume holds the model weights, and Elastic Fabric Adapter connects the GPU nodes.
NVIDIA Dynamo serves the model either aggregated on a single node or disaggregated, with prefill and decode stages running on different nodes. For vision-language workloads, the image and video encoding stage can additionally run disaggregated on its own node.
The architecture has been validated on both p6-b200.48xlarge instances, which include 8 x NVIDIA B200 GPUs, and p5en.48xlarge instances, which include 8 x NVIDIA H200 GPUs. Teams can validate the cluster and its network with a NVIDIA Collective Communications Library all-reduce micro-benchmark. Seeing utilization on all GPUs and traffic on all Elastic Fabric Adapter interfaces while the test is running verifies that the cluster is working as expected.
Performance validation, next steps, and credits
After deployment, teams can measure performance with the NVIDIA aiperf tool. Its output includes key performance indicators such as time to first token, inter-token latency, and throughput in tokens per second. By taking these measurements at various levels of concurrency, teams can study how latency and throughput depend on the number of concurrent requests. That information is essential for planning capacity and choosing a model deployment topology.
The aws-do-eks project provides a detailed walkthrough for deploying Nemotron 3.5 Super VL and Nemotron 3 Ultra on AWS. It covers prerequisites, cluster creation, model weights download, deployment, testing, benchmarking, and cleanup. The same pattern can be applied to other models in the NVIDIA Nemotron family. The aws-do-eks container also automates visualization of GPU and Elastic Fabric Adapter utilization, as well as running aiperf tests and concurrency sweeps with single commands.
A concurrency sweep comparing aggregated and disaggregated deployments shows that Nemotron 3.5 Super VL inference has lower first-token delay and about the same inter-token latency between the two layouts on two nodes. This type of chart helps with deployment decisions. For example, a latency budget can be drawn as horizontal lines, such as time to first token under 600 milliseconds, about 1 second, and inter-token latency under 100 milliseconds per token. The rightmost point under both lines is the throughput one replica of that topology can serve within the service level objective. In the example, that point is c32, or about 320 tokens per second.
To plan capacity, divide the target tokens per second by this per-replica throughput and determine how many prefill-decode pairs are needed. Re-running the same sweep after a change, such as precision, tensor parallelism and pipeline parallelism shape, aggregated versus disaggregated layout, or a new NVIDIA Dynamo or vLLM release, makes the trade-off visible immediately. A curve that moves right and down is an improvement. Because time to first token and inter-token latency are plotted separately, the chart also indicates which side to scale: rising inter-token latency suggests adding decode capacity, while rising time to first token suggests adding prefill capacity.
Builders can start by deploying Nemotron 3.5 Super VL in their own account using the reference implementation. More inference blueprints are available in the AI on EKS project, and teams can register for upcoming AWS workshops for additional hands-on experience with inference on EKS.
The post credits Eliuth Triana, Jiahong Liu, Kshitiz Gupta, Abhijeet Patil, Mike Stefaniak, Chris Splinter, Sheetal Joshi, Aditya Ramakrishnan, Natarajan Kumar, and Anish Maddipoti, among others. The authors are Alex Iankoulski, Principal Solutions Architect in the Worldwide Specialist Organization at AWS; Anton Alexander, senior specialist in generative AI at AWS; Leah Tucker, Senior AI and ML Specialist Solutions Architect; Curt Lockhart, AI Solutions Architect at NVIDIA; Abdullahi Olaoye, Senior AI Solutions Architect at NVIDIA; and Wenhan Tan, solutions architect at NVIDIA.
Comments (0)