Execute AI Models for Edge Application Development Using Only a Few Commands
A few commands set up TensorRT Edge-LLM, run Nemotron-3.5 Lightning locally, serve an API, and compare speculative decoding for edge AI.
Key Takeaways
- NVIDIA DGX Spark 64GB enables edge developers to run Nemotron-3.5 Lightning locally with TensorRT Edge-LLM, reducing setup friction for physical AI prototypes.
- The workflow supports a few CLI steps: install TensorRT Edge-LLM, download and build engines, test prompts in a terminal chat, then serve an OpenAI-compatible local API.
- DSpark speculative decoding can be enabled to accelerate generation, improving token speed while keeping time-to-first-token broadly similar.
- The same runtime can be used for development on DGX Spark and deployment on edge platforms such as Jetson Thor, DRIVE AGX Thor, and IGX Thor.
Meta description: Learn how to run NVIDIA Nemotron-3.5 Lightning on DGX Spark 64GB with TensorRT Edge-LLM, test prompts, serve a local API, and compare DSpark performance.
Tags: NVIDIA DGX Spark 64GB, TensorRT Edge-LLM, Nemotron-3.5 Lightning, DSpark speculative decoding, edge AI
Featured image: An abstract friendly illustration of a compact developer workstation connected to a robot arm, a production line, and a driver cabin, with no real people, no logos, and no text.
NVIDIA DGX Spark 64GB now gives embedded developers a practical way to run NVIDIA Nemotron-3.5 Lightning with TensorRT Edge-LLM, turning a text-only model into a local chat test and an API-backed prototype before moving workloads to edge hardware. The setup is aimed at physical AI systems that need to perceive, reason, and act inside tight memory, power, and latency budgets.
Why This Matters for Edge AI Developers
Edge AI is moving beyond the data center as developers build agents that monitor driver attention and cabin conditions, help robots understand their surroundings and plan tasks, and coordinate fleets of machines on production lines. These applications need models that can respond quickly without relying on a distant server.
Getting a model ready to run locally can involve substantial setup. TensorRT Edge-LLM is designed to reduce that friction. It is a lightweight C++ inference runtime built for embedded workloads. The same inference framework can be used for development on DGX Spark and deployment on platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
For developers, the value is simple: prototype near the target environment, test prompts locally, serve the model through an API, and compare the effect of speculative decoding before moving to a production device.
Start From DGX Spark and TensorRT Edge-LLM
To begin, you need a DGX Spark with a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12. The examples were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.
- Open a terminal on your DGX Spark.
- Create and activate a virtual environment.
- Install TensorRT Edge-LLM.
- Check the TensorRT version and confirm that the TensorRT Edge-LLM runtime loads.
- Use the trtedge command-line interface, or CLI, to download the selected model and build TensorRT engines optimized for inference on Spark.
The virtual environment keeps newly installed packages separate while still allowing access to system Python packages, including TensorRT. If compatible system TensorRT bindings are unavailable, follow the NVIDIA TensorRT pip installation instructions to install a compatible TensorRT 10.x version inside the activated environment, then rerun the verification command.
Test Nemotron-3.5 Lightning in a Terminal
Once the environment is ready, start a chat with Nemotron-3.5 Lightning. On the first launch, wait for the model download, which is approximately 21GB, and for the engine build to finish. Files are stored in ~/.cache/tensorrt-edgellm by default, so you should allow space for both the model and the engines. Later launches reuse matching cached engines.
When the chat prompt appears, type a question and review the response. You can continue the conversation with follow-up questions or type /exit to end the session.
This step is useful because it isolates the model behavior from application plumbing. You can confirm that the prompt, response quality, and latency are acceptable before adding API serving or deployment logic.
Serve the Model Through a Local API
After the terminal chat works, serve the model through a local OpenAI-compatible API on Spark for application development and testing.
- Keep the virtual environment active.
- Start the server.
- The server builds or reuses an engine for this configuration.
- Once startup completes, open a second terminal on your DGX Spark and check that the server is ready.
- A response containing "status":"ready" confirms readiness.
- Send a prompt to the chat completions endpoint.
- The response streams as JSON events.
- Keep the server running for more requests, or press Ctrl+C in the server terminal to stop it.
This API layer is where prototyping becomes more application-like. A local service can be connected to a test client, a dashboard, or a robot control loop without changing the model runtime itself.
Add DSpark Speculative Decoding
You can speed up generation with DSpark speculative decoding. This method uses a draft model to propose several tokens for the main model to verify in a single pass.
To enable it with three draft tokens:
- Press Ctrl+C in the server terminal.
- Restart the server with DSpark enabled.
- Wait until the server reports readiness.
- Send another prompt to the chat completions endpoint to try DSpark-enabled generation.
You can experiment with num_speculative_tokens and other settings to tune generation speed for your workload. The speculative decoding documentation covers additional details.
Performance on DGX Spark 64GB
TensorRT Edge-LLM delivers low response latency and fast token generation with NVIDIA Nemotron-3.5 Lightning on DGX Spark 64GB. The table below reports time to first token, or TTFT, and generation speed with and without DSpark speculative decoding.
| Decoding | Workload | Median TTFT | Mean generation speed |
|---|---|---|---|
| DSpark OFF | SpeedBench 1K | 201.0 ms | 73.3 tokens/s |
| DSpark OFF | SpeedBench 2K | 329.5 ms | 72.8 tokens/s |
| DSpark OFF | HumanEval prompts | 98.1 ms | 73.4 tokens/s |
| DSpark ON | SpeedBench 1K | 203.0 ms | 108.5 tokens/s |
| DSpark ON | SpeedBench 2K | 331.6 ms | 105.3 tokens/s |
| DSpark ON | HumanEval prompts | 98.7 ms | 126.9 tokens/s |
Table 1. Nemotron-3.5 Lightning performance on DGX Spark 64GB with and without DSpark speculative decoding.
Generation speed is the average token generation rate per request, excluding time to first token. DSpark uses three draft tokens. Peak total system memory use during inference with DSpark was 27.9GB, including the OS.
The measurements were taken on DGX Spark 64GB running DGX OS 7.6.0, which is Ubuntu 24.04.4 LTS. The setup used NVFP4 with thinking enabled, batch size one, one concurrent streaming request, and a 128-token output limit. Average input lengths were 1,153 tokens for SpeedBench 1K, 2,247 tokens for SpeedBench 2K, and 139 tokens for HumanEval prompts.
The pattern is straightforward. TTFT changes very little, while generation speed rises when DSpark is enabled. For interactive assistants, the first token often matters most. For longer responses, throughput can change the perceived experience.
Next Steps
Try another model from the supported models list and continue developing your application on Spark. Options include vision-language models that interpret images and video, as well as speech recognition and text-to-speech models.
Explore the TensorRT Edge-LLM documentation for model-specific workflows and requirements on your target embedded platform. Visit the Nemotron developer page for resources to get started, and explore open Nemotron models and datasets on Hugging Face and Blueprints on build.nvidia.com.
Common Mistakes
- Running outside the virtual environment and missing the intended package context.
- Forgetting that the first launch needs space for both the approximately 21GB model download and built engines.
- Assuming every launch rebuilds the engine when matching cached engines can be reused.
- Checking the API too early and treating an unfinished startup as a failure.
- Turning on DSpark speculative decoding without testing the workload, because speed gains depend on the task and token length.

Comments (0)