Termsslug: prompt-caching

Prompt Caching

提示词缓存

Reuses computed state of a matching prompt prefix to avoid reprocessing long context, cutting latency and cost.

Curated Intelligence

27 items · Page 1/3
AWS Machine Learning BlogT1·Agents
62 pts
Amazon Bedrock AgentCore Harness Adopts Strands with Lifecycle Hooks

Amazon upgraded the Bedrock AgentCore harness to build on its open-source Strands harness, giving managed agents recommended models, task-oriented system prompts, automatic prompt caching, and proactive context management by default. It also adds built-in tools for planning, web retrieval, filesystem access, context offloading, and task delegation, plus lifecycle hooks that let developers inject deterministic checks and notifications at defined points in the agent loop.

AWS Machine Learning BlogT1·Infra
58 pts
ArisGlobal Cuts Bedrock Inference Costs by Reordering Prompt Cache

Life-sciences SaaS vendor ArisGlobal runs its NavaX engine on Amazon Bedrock to automate pharmacovigilance adverse-event case processing. As multi-tenant compliance requirements pushed the shared prompt prefix up roughly 150% in six months and per-case inference costs up over 200%, the team reversed its caching strategy to cache the shared prefix instead of each source document, cutting effective input cost and enabling tenant growth without proportional spend.

AWS Machine Learning BlogT1·Tools
38 pts
Amazon Bedrock Adds Prompt Caching for Open-Weight Models

Amazon Bedrock now supports explicit prompt caching for open-weight models, letting applications reuse repeated prefixes such as system prompts, reference documents, tool definitions and conversation history to cut input costs and latency. The cache is fully managed by Bedrock, offers implicit and explicit modes, and exposes cache read/write metrics for monitoring.

宝玉 (@dotey)T2·Agents
58 pts
Layered Main-Session and Thread Routing Cuts Agent Token Costs

Baoyu outlines a layered routing pattern for multi-agent sessions: the main session only classifies and dispatches tasks, so a mid-tier model like Sonnet 5.5 suffices, while stronger models such as Opus 5.5 handle work inside threads. Paired with differentiated prompt-caching tactics, sending only recent messages plus session retrieval after cache expiry and compressing thread context near cache TTL, overall token consumption stays manageable.

AWS Machine Learning BlogT1·Infra
58 pts
Four Amazon Bedrock Inference Settings That Directly Affect Cost

AWS's official blog notes that even after prompt and application tuning, four request-level settings—cache duration, service tier, reasoning effort, and maxTokens—still materially change Amazon Bedrock's token volume and billing rates. It documents how each setting was tested and how to verify impact via response metadata, logs, and billing data.

AWS Machine Learning BlogT1·Agents
58 pts
Amazon Bedrock Introduces Framework for Agent Cost per Task and ROI

AWS published the second part of its Bedrock cost series, arguing that agent economics should be measured by cost per task rather than cost per token, with caching, model routing and output constraints as the main levers. It also outlines a discipline for proving agent ROI through human baselines, holdout experiments and a five-dimension scorecard, aligned with the Well-Architected Agentic AI Lens.

IT之家 · 智能时代T2·Tools
58 pts
Anthropic Adds Mid-Conversation Effort Control to Claude API

Anthropic has introduced a beta feature to the Claude API that lets developers adjust effort mid-conversation by inserting an empty system message with output_config.effort. The setting applies from the next user message onward without altering already-generated content, and prompt caching still hits, helping balance quality, latency and cost when switching between casual chat and complex analysis.

AWS Machine Learning BlogT1·Infra
45 pts
Amazon Bedrock Adds India-Based Inference for Kimi K3

Amazon Bedrock introduces in-country inference for the open-weight multimodal model Kimi K3 in India to meet strict data residency requirements. This allows customers to process sensitive local workloads while integrating frontier coding and agentic capabilities via existing APIs.

宝玉 (@dotey)T2·Tools
78 pts
Claude API Launches Message Threads Beta to Speed Up Long Conversations

Anthropic has launched a beta for 'Message Threads' in the Claude Messages API, allowing the server to cache conversation context so developers no longer need to resend full history each turn. This feature significantly reduces request payload size (from ~800KB to 1.8KB) and cuts time-to-first-token by ~35%, though token pricing remains unchanged. It is particularly beneficial for long-running agents and multimodal interactions.

Arena.ai (@arena)T1·Agents
68 pts
Arena.ai: Missing Cache Increases Agent Costs by 260%

Arena.ai analyzed over 26K agent sessions, revealing that removing prompt caching increases typical task costs by 260%. For models like Claude Fable 5.1, a single task's cost jumps from $7.44 to $27 without caching, while cache invalidation adds another 5% to bills. The study highlights the critical role of caching in optimizing agent economics.