Skip to content

Commit 22b6008

Browse files
chore: [js] add prompt caching docs (#500)
--------- Co-authored-by: BrigittaK307 <brigitta.kovacs@sap.com>
1 parent fb52ad3 commit 22b6008

1 file changed

Lines changed: 72 additions & 0 deletions

File tree

docs-js/orchestration/chat-completion.mdx

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -762,6 +762,78 @@ const response = await orchestrationClient.chatCompletion({
762762
You can send a single chat completion request with multiple images by defining multiple content blocks with `image_url` type in the template.
763763
The model will process each image and use the information from all of them to respond.
764764

765+
### Prompt Caching
766+
767+
:::note
768+
This section covers explicit cache control for AWS Bedrock-hosted models (Anthropic Claude and Amazon Nova).
769+
Other model providers, such as OpenAI and Gemini, use implicit context caching that is enabled by default and requires no additional configuration.
770+
For a full overview, refer to the [Prompt Caching documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/prompt-caching?locale=en-US).
771+
:::
772+
773+
Cache control improves performance by caching requests at the model provider level.
774+
Add a `cache_control` breakpoint to a content block to mark a cache breakpoint in the prompt.
775+
Availability of different time-to-live (TTL) options may depend on the model, refer to the [orchestration prompt caching documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/prompt-caching?locale=en-US) for details.
776+
777+
:::note
778+
Prompt caching requires a minimum number of tokens in the cached content.
779+
If the prompt is too short, the cache breakpoint may not take effect.
780+
:::
781+
782+
Set the `cache_control` property as part of message contents.
783+
Some models can also apply `cache_control` to tool calls.
784+
Use the `getTokenUsage()` method to inspect cache activity.
785+
For supported models, the `prompt_tokens_details` property on the tokens usage object will contain the amount of tokens read and written to the cache.
786+
787+
```ts
788+
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';
789+
790+
const orchestrationClient = new OrchestrationClient({
791+
promptTemplating: {
792+
model: {
793+
name: 'anthropic--claude-4.7-opus'
794+
},
795+
prompt: {
796+
template: [
797+
{
798+
role: 'system',
799+
content: [
800+
{
801+
type: 'text',
802+
text:
803+
'You are an expert news article classifier.' +
804+
' Classify each article into exactly one category.'.repeat(512),
805+
cache_control: { type: 'ephemeral' }
806+
}
807+
]
808+
},
809+
{ role: 'user', content: 'input: Comcast launches prepaid plans' },
810+
{ role: 'assistant', content: 'Business' },
811+
{
812+
role: 'user',
813+
content: [
814+
{
815+
type: 'text',
816+
text: 'input: {{?article}}',
817+
cache_control: { type: 'ephemeral' }
818+
}
819+
]
820+
}
821+
]
822+
}
823+
}
824+
});
825+
826+
const response = await orchestrationClient.chatCompletion({
827+
placeholderValues: {
828+
article:
829+
'Scaling up neural models has yielded significant advancements in language generation'
830+
}
831+
});
832+
833+
const usage = response.getTokenUsage();
834+
console.log('Cache details', usage.prompt_tokens_details ?? usage);
835+
```
836+
765837
### File Input
766838

767839
Some models in the orchestration service accept files as input, such as PDFs, CSV files, Word documents, and audio files.

0 commit comments

Comments
 (0)