Tutorials

Develop ultra-low-latency voice AI applications on AWS.

Cartesia uses State Space Models on AWS SageMaker and GPU instances to enable sub-100 millisecond voice AI for natural real-time applications.

Key Takeaways

  • Cartesia uses purpose-built State Space Models on AWS infrastructure, including Amazon SageMaker AI and GPU instances, to deliver sub-100 millisecond voice AI for text-to-speech and speech-to-text at global scale.
  • Ultra-low latency is essential because users expect human-like responsiveness in voice interactions; excess delay can break conversational flow and reduce trust in applications such as contact centers, telehealth, gaming, accessibility tools, and sales.
  • State Space Models address the quality-versus-latency trade-off of traditional transformer-based models by processing sequences step by step with a compact, fixed-size internal state, enabling efficient streaming and lower memory use.
  • Sonic, Cartesia’s streaming text-to-speech model, delivers first audio bytes in about 90 milliseconds, supports natural prosody and emotion across 44+ languages, and can clone voices from only 10-second samples.
  • Ink, Cartesia’s streaming speech-to-text model, includes native turn detection, context-aware end pointing, real-time partial transcripts, and robustness in noisy environments with overlapping speech.

Meta description: Cartesia, an AWS Partner, uses State Space Models on Amazon SageMaker AI and Amazon Elastic Compute Cloud GPU instances to deliver sub-100 millisecond voice AI.

Tags: real-time voice AI, State Space Models, AWS, Amazon SageMaker AI, Cartesia

Featured image: Abstract friendly illustration of a voice waveform moving through cloud GPU instances, no real people, no logos, no text.

Cartesia, an AWS Partner specializing in real-time voice AI, introduced how it uses purpose-built State Space Models and AWS infrastructure to deliver sub-100 millisecond text-to-speech and speech-to-text at global scale. For developers building voice assistants, AI-powered contact centers, real-time translation, accessibility tools, telehealth, gaming, and sales workflows, the goal is voice output that arrives within the same temporal window as human turn-taking, making interactions feel closer to human conversation.

Why latency is the core problem

Conversational AI is changing how people interact with technology. Voice-first interfaces are becoming a default mode for millions of users worldwide, but delivering a truly natural voice experience remains one of the hardest challenges in AI infrastructure. Users expect human-like responsiveness. Excess latency can break conversational flow and erode trust. For AI-generated speech to feel natural, it must arrive almost as quickly as a human reply.

The demand for real-time voice AI is growing across industries. Contact centers need AI agents to handle millions of customer interactions with instantaneous speech generation. Telehealth consultations require low-latency voice for accurate, empathetic communication. Gaming and virtual worlds need real-time voice synthesis for non-player characters and player interactions. Accessibility tools such as screen readers and voice interfaces must respond instantly for users with visual impairments. Inbound and outbound sales applications require AI sales agents to respond without awkward pauses while qualifying leads, handling objections, and keeping prospects on the line.

Traditional transformer-based text-to-speech models struggle with streaming latency because they process entire sequences before producing output. That architectural limitation creates a fundamental tension: higher quality often means higher latency. State Space Models address this trade-off by using a different way to process sequences.

State Space Models change the trade-off

Transformers, the architecture behind most of today's AI, use attention, a mechanism that compares every element in a sequence to every other element. As a sequence gets longer, the amount of computation grows quadratically. State Space Models instead treat a sequence as a system that evolves continuously over time, processing it step by step and carrying a compact internal state forward. This can be converted into a form computers handle efficiently, keeping cost low for certain long sequences. That efficiency makes State Space Models well suited to real-time, streaming data such as live speech.

For audio processing, State Space Models offer three key advantages. Structured State Space Models are a form of linear recurrence, but their state parameterization preserves information over long audio sequences, avoiding the long-range signal decay that limits classic recurrent neural networks such as long short-term memory networks. Unlike transformers, which cache all previous tokens, State Space Models maintain a fixed-size hidden state, giving them a constant memory footprint. Their recurrent structure also processes input token-by-token, so they support true streaming without architectural modifications.

These properties make State Space Models especially useful for real-time voice applications where both quality and latency are non-negotiable. Cartesia puts these advantages into two production voice models: Sonic for speech generation and Ink for transcription.

Sonic and Ink cover both sides of conversation

Cartesia is a real-time multimodal intelligence company that built a voice AI application purpose-built on State Space Model architecture. The company was founded by researchers who pioneered State Space Model advances, and its application is designed for streaming inference at the edge of human perception. The core models are Sonic for text-to-speech and Ink for speech-to-text, both optimized for sub-100 milliseconds first-byte latency while maintaining studio-quality output.

Sonic is Cartesia's flagship streaming text-to-speech model. It delivers ultra-realistic speech synthesis with a first audio byte in approximately 90 milliseconds, so users hear the voice respond almost instantaneously. It supports natural prosody, emotion, and expressiveness across over 44 languages. Its voice cloning works with minimal data, allowing custom voices to be created from 10-second audio samples. The streaming-native architecture begins generating audio before the full text input is received, while context-aware synthesis maintains consistent tone and style across long passages. Cartesia recommends Sonic for teams that want the fastest time-to-production with minimal operational overhead: call the API and receive streaming audio in real time.

Ink is Cartesia's streaming speech-to-text model, purpose-built for conversational AI applications. It includes native turn detection, automatically identifying when a speaker has finished their turn and alleviating awkward pauses without a separate voice activity detection or end pointing model. Context-aware end pointing uses semantic understanding to distinguish between mid-sentence pauses and true conversation boundaries. Streaming transcription emits partial transcripts in real time as the user speaks, allowing a downstream large language model to begin forming its response before the turn ends. Ink also maintains accuracy in noisy environments common in real-world deployments and handles overlapping speech and speaker transitions gracefully through multi-speaker support.

Together, Sonic and Ink provide the complete voice AI stack needed to build conversational agents that feel human.

Two AWS deployment paths for Sonic

Teams that need data sovereignty, latency control, or compliance can deploy Cartesia models directly into their own AWS account. Sonic is delivered as an Amazon SageMaker AI model package through AWS Marketplace, which can be deployed to a SageMaker AI endpoint running entirely within a virtual private cloud. Inference data does not leave the customer account. Ink is currently available through the Cartesia API, which can be purchased through the Cartesia Voice AI Platform listing in AWS Marketplace. Two primary patterns are available for Sonic.

The first option is Amazon SageMaker AI. The deployment must run from an Amazon SageMaker Notebook Instance or SageMaker Studio, and the AWS Identity and Access Management execution role needs AmazonSageMakerFullAccess. Before deployment, users must choose a supported instance type from the model package's SupportedRealtimeInferenceInstanceTypes field. The supported options are ml.g6e.xlarge or larger, ml.g7e.2xlarge or larger, ml.g7.2xlarge or larger, ml.p5.48xlarge, and ml.p5en.48xlarge. If the chosen instance type's applied value is below current usage plus the planned endpoint instance count, a service quota increase may be required, with details available in SageMaker AI quotas.

To deploy Sonic 3.6 to a SageMaker endpoint, open AWS Marketplace resources, then All products, and search for Sonic 3.6 SageMaker. Choose the Cartesia listing, select View purchase options, review the pricing and terms, and subscribe. The subscription grants the AWS account access to the model package but does not deploy resources. From Deployments & inference, choose Deployable models, then Create model, select Use a model package subscription from AWS Marketplace, and choose Sonic 3.6 SageMaker. Select an execution role and create the model. Then create a real-time endpoint from the model using a supported GPU instance, such as ml.g6e.2xlarge. The endpoint first appears with Creating status; wait for InService before invoking it.

The model package Amazon Resource Name is Region-specific. To verify the Sonic 3.6 package in US East (N. Virginia) with the SageMaker API, describe the model package and query ModelPackageStatus, InferenceSpecification.Containers[0].ProductId, and InferenceSpecification.SupportedRealtimeInferenceInstanceTypes. The package Amazon Resource Name value is `arn:aws:sagemaker:us-east-1:865070037744:model-package/sonic36-sagemaker-v0-c6e9cacffcdf31178d1bd7c98c587480`. After the endpoint reaches InService, invoke it with the SageMaker Runtime InvokeEndpointWithResponseStream API and process the returned audio stream in the application.

The second option is self-managed deployment on Amazon Elastic Compute Cloud GPU instances. Some teams have existing GPU infrastructure, strict networking requirements, or a need to co-locate Cartesia with other services. For those cases, the model container can be run directly on Amazon Elastic Compute Cloud GPU instances, giving full control over runtime, scaling, and network path. This approach uses Cartesia's container image, obtained from Cartesia, rather than the AWS Marketplace model package.

The process begins by obtaining the container image. Teams request the Sonic container image and licensing terms from Cartesia, then replicate it into a private Amazon Elastic Container Registry repository so the compute plane can pull it with Identity and Access Management scoped credentials rather than reaching out to an external registry at scale.

For GPU selection, Sonic runs on NVIDIA GPU instances. The G6 with L4 and G6e with L40S families are described as the optimal balance for real-time text-to-speech, while the earlier G5 with A10G also works. Teams should match GPU memory and virtual CPU to target concurrency and prefer a single-GPU instance per model replica for the simplest scaling.

The compute plane can be provisioned with Amazon Elastic Container Service or Amazon Elastic Kubernetes Service. For Amazon Elastic Container Service, register a GPU task definition, set the container's GPU resource requirement, for example resourceRequirements with type GPU, and run it as a service behind an Application Load Balancer. For Amazon Elastic Kubernetes Service, create a GPU node group or Karpenter provisioner using the GPU-optimized Amazon Machine Image, install the NVIDIA device plugin, and deploy the container as a Deployment fronted by a Service or Ingress.

Networking should be configured for low latency. Inference nodes can be placed in a cluster placement group to minimize inter-node latency, with replicas kept in a single Availability Zone for the tightest path. Virtual private cloud endpoints can be used for Amazon Elastic Container Registry and Amazon Simple Storage Service pulls. For hybrid or on-premises callers, traffic can be terminated over AWS Direct Connect or a private Network Load Balancer to avoid public-internet hops.

Model weights should be loaded efficiently. Store weights on a shared, low-latency volume such as Amazon Elastic File System or Amazon FSx for Lustre, mounted across replicas so new pods or tasks start without each re-downloading multi-gigabyte artifacts. Warm the model at container start and expose a health check so the load balancer routes only to ready replicas.

Batching and concurrency need tuning. Set the container's max batch size and concurrent-stream limits to match the traffic profile. Higher batch size improves GPU utilization but adds first-byte latency, so it must be tuned against the sub-100 milliseconds target. Benchmark streams per GPU to find the replica capacity used for scaling.

Custom auto scaling should rely on signals that reflect real load, such as active audio-stream count or GPU utilization, rather than CPU. On Amazon Elastic Container Service, the guidance mentions AWS Application Auto Scaling.

What this means for builders

The practical takeaway is that latency is not only a model problem. It is an architecture problem, and it is also an infrastructure problem. State Space Models provide a path to streaming speech without the quadratic attention cost that can make long sequences expensive. Cartesia's Sonic and Ink cover the two sides of a conversation: speaking and listening. AWS deployment then determines how much control, compliance, and operational overhead a team takes on.

For developers choosing between paths, the managed Amazon SageMaker AI model package offers a clearer route to production with inference data staying inside the customer account. The Amazon Elastic Compute Cloud path offers deeper control over GPU instances, network topology, scaling, and co-location with existing services. Ink remains available through the Cartesia API and AWS Marketplace listing, while Sonic has both SageMaker and self-managed options. Teams evaluating real-time voice AI will need to compare first-byte latency, language coverage, voice cloning requirements, turn detection, transcription streaming, and deployment constraints before deciding which path fits their workload.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.