Skip to content

feat(cache): add TTL support to CachePoint for prompt caching#1660

Merged
mkmeral merged 11 commits intostrands-agents:mainfrom
kpx-dev:feat-support-prompt-caching-ttl
Apr 24, 2026
Merged

feat(cache): add TTL support to CachePoint for prompt caching#1660
mkmeral merged 11 commits intostrands-agents:mainfrom
kpx-dev:feat-support-prompt-caching-ttl

Conversation

@kpx-dev
Copy link
Copy Markdown
Contributor

@kpx-dev kpx-dev commented Feb 10, 2026

Description

Add optional ttl field to CachePoint TypedDict to support AWS Bedrock's cache TTL configuration. The field accepts "5m" or "1h" values as specified in the Bedrock API. Updated BedrockModel to preserve ttl field when formatting cache point content blocks. Includes tests for TTL preservation and backward compatibility.

Doc shows API supports ttl option now:
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_CachePointBlock.html

launch announcement:
https://aws.amazon.com/about-aws/whats-new/2026/01/amazon-bedrock-one-hour-duration-prompt-caching/

more details:
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

Related Issues

#1243

Documentation PR

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

  • [x ] I ran hatch run prepare

Checklist

  • [x ] I have read the CONTRIBUTING document
  • [ x] I have added any necessary tests that prove my fix is effective or my feature works
  • [x ] I have updated the documentation accordingly
  • [ x] I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • [x ] My changes generate no new warnings
  • [ x] Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

mkmeral
mkmeral previously approved these changes Feb 17, 2026
@mkmeral mkmeral enabled auto-merge (squash) February 17, 2026 17:00
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

zastrowm
zastrowm previously approved these changes Feb 17, 2026
Comment thread src/strands/types/content.py Outdated
Comment thread tests/strands/models/test_bedrock.py
auto-merge was automatically disabled February 17, 2026 18:09

Head branch was pushed to by a user without write access

@kpx-dev kpx-dev dismissed stale reviews from zastrowm and mkmeral via 230ed7c February 17, 2026 18:09
@github-actions github-actions Bot added size/s and removed size/s labels Feb 17, 2026
@kpx-dev kpx-dev requested review from mkmeral and zastrowm February 17, 2026 18:43
@kpx-dev
Copy link
Copy Markdown
Contributor Author

kpx-dev commented Feb 17, 2026

FYI this PR fails label size: https://github.com/strands-agents/sdk-python/actions/runs/22110060966 . Not sure if CI needs update or label input needs update

Comment thread tests_integ/models/test_model_bedrock.py Outdated
Comment thread tests_integ/models/test_model_bedrock.py Outdated
zastrowm
zastrowm previously approved these changes Feb 18, 2026
Copy link
Copy Markdown
Member

@zastrowm zastrowm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last nit (imports) - looks good

Comment thread tests_integ/models/test_model_bedrock.py Outdated
mkmeral
mkmeral previously approved these changes Feb 18, 2026
Copy link
Copy Markdown
Contributor

@mkmeral mkmeral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll approve. I want to try following up with strands command for nits, let's see how that works. the only nit is the import in the test

@github-actions github-actions Bot added size/m and removed size/m labels Apr 6, 2026
@mkmeral mkmeral temporarily deployed to manual-approval April 6, 2026 20:41 — with GitHub Actions Inactive
@github-actions github-actions Bot added size/m and removed size/m labels Apr 6, 2026
@ElliottJW
Copy link
Copy Markdown
Contributor

ElliottJW commented Apr 17, 2026

Great fix for the Bedrock model path! There's an analogous issue in the LiteLLMModel path that would be worth including here or as a follow-up PR.

The problem

LiteLLMModel._format_system_messages() in src/strands/models/litellm.py also drops the ttl field from CachePoint when converting to cache_control:

# Current code — TTL silently dropped
elif "cachePoint" in block and block["cachePoint"].get("type") == "default":
    if system_content:
        system_content[-1]["cache_control"] = {"type": "ephemeral"}

The fix:

elif "cachePoint" in block and block["cachePoint"].get("type") == "default":
    if system_content:
        cache_control: dict[str, Any] = {"type": "ephemeral"}
        if ttl := block["cachePoint"].get("ttl"):
            cache_control["ttl"] = ttl
        system_content[-1]["cache_control"] = cache_control

Why it matters

When using LiteLLMModel pointed at a Databricks external model endpoint (which proxies to AWS Bedrock), the cache_control field is forwarded verbatim to the endpoint via LiteLLM's DatabricksConfig. Bedrock supports "5m" and "1h" TTL values on Claude 4.5+ models. Without this fix, the TTL specified in CachePoint(ttl="1h") is silently dropped and Bedrock falls back to the default 5-minute TTL.

Suggested tests (mirroring the structure of the new test_bedrock.py tests in this PR):

  • test_format_system_messages_preserves_cache_point_ttl — verify that CachePoint(type="default", ttl="1h") produces cache_control: {"type": "ephemeral", "ttl": "1h"} on the preceding content block
  • test_format_system_messages_cache_point_without_ttl — verify backward compatibility: CachePoint(type="default") still produces cache_control: {"type":"ephemeral"} with no ttl key

Comment thread src/strands/types/content.py Outdated
Comment thread tests_integ/models/test_model_bedrock.py
Comment thread tests/strands/models/test_bedrock.py
Comment thread tests_integ/models/test_model_bedrock.py
Comment thread tests_integ/models/test_model_bedrock.py
@github-actions
Copy link
Copy Markdown

Assessment: Comment

Clean, well-scoped feature addition that correctly adds optional TTL support to CachePoint and preserves it through the Bedrock model formatting path. The backward compatibility approach (using NotRequired + conditional inclusion) is solid. Unit and integration tests cover the key scenarios well.

Review Categories
  • API Design: The Literal["5m", "1h"] type for ttl provides good IDE guidance but may hinder forward compatibility if AWS adds new values — consider str with documentation instead.
  • Testing: Integration tests for 5m and 1h TTL are nearly identical and could benefit from @pytest.mark.parametrize. The backward compatibility test uses a different model and smaller context size than the TTL tests, which may affect caching behavior.
  • Scope: The AGENTS.md changes include unrelated directory structure updates from the rebase — please verify these are intentional.

Nice contribution that addresses a real customer need (issue #1243) with a minimal, clean implementation.

# Conflicts:
#	src/strands/types/content.py
@github-actions
Copy link
Copy Markdown

Assessment: Comment

The key feedback from the previous review (Literal["5m", "1h"]str) has been addressed, and the LiteLLM model path was also updated to preserve TTL — great follow-through. The remaining open threads from Round 1 are minor and non-blocking.

Remaining items from previous review
  • Redundant assertions (unit tests lines 2156, 2179): The assert cache_point_block == expected already validates the full dict equality, making the per-field assertions on the next lines redundant. Minor, non-blocking.
  • Integration test duplication: test_prompt_caching_with_5m_ttl and test_prompt_caching_with_1h_ttl could be combined with @pytest.mark.parametrize. Minor, non-blocking.
  • Backward compat test flakiness risk: test_prompt_caching_backward_compatibility_no_ttl uses the default model (Sonnet 4) with * 200 repetitions (~1000 tokens), which may be near/below the minimum cache threshold. The other tests use Haiku 4.5 with * 1000 (~5000+ tokens). If this test flakes in CI, aligning model and context size would fix it.

Overall, this is in good shape for merge. The core changes are clean and well-tested across both Bedrock and LiteLLM paths.

@mkmeral
Copy link
Copy Markdown
Contributor

mkmeral commented Apr 23, 2026

Thank you for your contribution @kpx-dev ! I've enabled auto-merge, once MacOS tests pass, it will merge.

ps: Integ tests are failing due to setup issue, the added tests passed previously, so I am okay merging

@kpx-dev
Copy link
Copy Markdown
Contributor Author

kpx-dev commented Apr 23, 2026

thank you @mkmeral . Looks like MacOS tests passed but not merging yet? Something is blocking it?

@kpx-dev
Copy link
Copy Markdown
Contributor Author

kpx-dev commented Apr 24, 2026

thank you so much team!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants