Salesforce's Agentforce Foundation team serves in-house and open-source models across 30+ global Regions, where per-Region hosting fragmented capacity and turned traffic spikes into availability problems. Using cross-Region inference routing in Amazon SageMaker AI, Salesforce now routes each request through a single invocation path to any of its own deployed endpoints across Regions and accounts, adding capacity securely while honoring data residency commitments and retaining visibility into where and why each request was served.
Amazon SageMaker AI
AWS fully managed machine learning platform covering training, deployment and real-time inference endpoints.
Curated Intelligence
Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, using cache state reported by the model container to send each request to an instance holding a long cached prefix match that is not overloaded. It targets multi-turn chat, long shared prompts and RAG workloads, and is enabled via one environment variable plus the KV_AWARE routing strategy in the endpoint configuration, with no client API changes.
AWS has added AI data preparation capabilities to Amazon SageMaker AI, enabling teams to generate synthetic examples, assess and improve existing datasets, and plug them into model development workflows. The feature targets the data bottleneck in fine-tuning, reinforcement learning, and evaluation suites, helping teams build high-quality task-specific data at lower cost.
AWS introduced usage-based metering for Amazon SageMaker AI bidirectional streaming inference, letting model providers bill AWS Marketplace customers for actual consumption during live streams while endpoint infrastructure charges remain separate. Deepgram is the launch partner, bringing usage-based speech-to-text and text-to-speech billing to AWS Marketplace.