How do we compress the context?

[SanthoshCheekoti]

Checklist

• My issue is not an AI Core service issue, but an SDK issue.
• My issue is a JS/TS related issue, not a Java related issue.
• My issue is not already reported or documented.

Describe the Question

Hello - I'm using CloudSDK + Langgraph. I have a node in my agent that calls a tool hosted by a remote MCP server. The tool is called repeatedly as part of the agent loop and is generating a lot of context, and hence my token count is very large.

1. Does SDK support the ability to compress the context based on a trigger ?
2. Secondly, we are having a system prompt that is a bit heavy. Does SDK prompt caching as shown below? If yes, please provide details on how to use it.
   https://docs.langchain.com/oss/javascript/integrations/middleware/anthropic

[davidkna-sap]

Does SDK support the ability to compress the context based on a trigger ?

The SAP Cloud SDK for AI is currently not an agent framework by itself, so it does not automatically compress conversation history or summarize context based on a trigger within an agent loop.

You will need to add appropriate handling to your langgraph that automatically forward the message history to a summarisation agent.
It is likely that some implementations of this pattern for langgraph are publicly available, and e.g. when using langchain.js you can use the built-in summarisation middleware.

Does SDK prompt caching as shown below? If yes, please provide details on how to use it.

The SAP Cloud SDK for AI langchain package does not yet fully expose this feature.

Nevertheless, you can already cache your system prompt with something like the following:

import { OrchestrationClient } from '@sap-ai-sdk/langchain';

const client = new OrchestrationClient({
    promptTemplating: {
      model: {
        name: 'anthropic--claude-4.5-haiku'
      },
      prompt: {
        template: [
          {
            role: 'system',
            content: [
              {
                type: 'text',
                // Repeated filler text to exceed Anthropic's minimum cacheable token threshold (4096 tokens for Haiku 4.5).
                text:
                  'You are a knowledgeable assistant. ' +
                  'You have deep expertise in science, technology, history, literature, mathematics, and many other fields. '.repeat(
                    220
                  ),
                cache_control: { type: 'ephemeral', ttl: '5m' }
              }
            ]
          }
        ]
      }
    }
  });

I will create a BLI to investigate a more comprehensive integration of this feature for the SAP Cloud SDK for AI langchain integration.
As an alternative is my understanding that for other model providers caching will usually happen automatically as part of the orchestration service (service documentation).

For general documentation on the feature for the base orchestration client please have a look at the sample-code, the current draft of the documentation page, and the orchestration service documentation on prompt caching.

[SanthoshCheekoti]

I tried implementing system prompt like below

import type { LangChainOrchestrationModuleConfig } from '@sap-ai-sdk/langchain';
import { buildPrompt } from '../utilities/knowledgeLoader.js';

    const systemPrompt = await buildPrompt(
        ['./prompts/roleDefinition.md', './prompts/datamodel_explorer.md'],
        { Introduction: './MDGknowledge/Introduction' }
    );

export const classifyIntentConfig: LangChainOrchestrationModuleConfig = {
  promptTemplating: {
    model: {
      name: 'anthropic--claude-4.6-sonnet',
      params: { temperature : 0.1 }
    }
  }
};

export const DataModelExplorerConfig: LangChainOrchestrationModuleConfig = {
  promptTemplating: {

    model: {
      name: 'anthropic--claude-4.6-sonnet',
      params: { temperature : 0.1 }
    },
    prompt :{
      template :[
        {
          role : 'system',
          content : [
            {
              type : 'text',
              text : systemPrompt,
              cache_control: { type: 'ephemeral', ttl: '5m' }
            }
          ]
        }
      ] 
    }
  }
};

I'm getting below error. Can you please help me with the error?

"request_id":"3c0af636-a77a-9a65-8644-cc9accf874a5","code":400,"message":"400 - LLM Module: tools: Tool names must be unique.","location":"LLM Module",
    at OrchestrationStream.iterator (/Users/XXXX/Documents/Agents/MDAgents/node_modules/@sap-ai-sdk/core/src/stream/sse-stream.ts:46:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async _processChunk (/Users/XXX/Documents/Agents/MDAgents/node_modules/@sap-ai-sdk/orchestration/src/orchestration-stream.ts:44:22)
    at async _processOrchestrationStreamChunkResponse (/Users/XXX/Documents/Agents/MDAgents/node_modules/@sap-ai-sdk/orchestration/src/orchestration-stream.ts:58:22)
    at async _processStreamEnd (/Users/XXX/Documents/Agents/MDAgents/node_modules/@sap-ai-sdk/orchestration/src/orchestration-stream.ts:71:22)
    at async OrchestrationClient._streamResponseChunks (/Users/XX/Documents/Agents/MDAgents/node_modules/@sap-ai-sdk/langchain/src/orchestration/client.ts:378:22)
    at async OrchestrationClient._generateUncached (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/core/src/language_models/chat_models.ts:666:26)
    at async OrchestrationClient.invoke (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/core/src/language_models/chat_models.ts:288:20)
    at async baseHandler (/Users/XX/Documents/Agents/MDAgents/node_modules/langchain/src/agents/nodes/AgentNode.ts:395:25)
    at async AgentNode.#invokeModel (/Users/XX/Documents/Agents/MDAgents/node_modules/langchain/src/agents/nodes/AgentNode.ts:734:22)
    at async AgentNode.#run (/Users/XX/Documents/Agents/MDAgents/node_modules/langchain/src/agents/nodes/AgentNode.ts:258:7)
    at async AgentNode.invoke (/Users/X/Documents/Agents/MDAgents/node_modules/langchain/src/agents/RunnableCallable.ts:71:25)
    at async RunnableSequence.invoke (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/core/src/runnables/base.ts:1988:25)
    at async _runWithRetry (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/langgraph/src/pregel/retry.ts:114:16)
    at async PregelRunner._executeTasksWithRetry (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/langgraph/src/pregel/runner.ts:330:27)
    at async PregelRunner.tick (/Users/XX/Documents/Agents/MDAgents/node_modules/@langchain/langgraph/src/pregel/runner.ts:138:37)

[SanthoshCheekoti]

I also tried below and used anthropicPromptCachingMiddleware,

import { OrchestrationClient } from '@sap-ai-sdk/langchain';
import type { LangChainOrchestrationModuleConfig } from '@sap-ai-sdk/langchain';

export function createClient(config: LangChainOrchestrationModuleConfig): OrchestrationClient {
  // eslint-disable-next-line no-process-env
  const resourceGroup = process.env.resourceGroup;
  return new OrchestrationClient(config, {}, { resourceGroup });
}
    const agent = createAgent({
        model: createClient(DataModelExplorerConfig),
        tools,
        systemPrompt,
        middleware: [
            summarizationMiddleware({
                model: createClient(DataModelExplorerConfig),
                trigger: { tokens: 400, messages: 3 },
                keep: { messages: 5 },
            }), 
            anthropicPromptCachingMiddleware({ ttl: "5m" })
        ]
    });

I'm getting the following message

PromptCachingMiddleware: Skipping caching for OrchestrationClient. Consider switching to an Anthropic model for caching benefits.

[davidkna-sap]

Both the cache_control hints as well as the summarizationMiddleware() middleware appear to work as expected in my testing (see below).
The anthropicPromptCachingMiddleware() middleware is not yet expected to work at this time.
As per the error, please double-check the tools you are providing to the agent and ensure that no duplicate tools or tools sharing the same names are provided.

  const orchestrationConfig: LangChainOrchestrationModuleConfig = {
    promptTemplating: {
      model: {
        name: 'anthropic--claude-4.5-haiku'
      },
      prompt: {
        template: [
          {
            role: 'system',
            content: [
              {
                type: 'text',
                // Repeated text to exceed Anthropic's minimum cacheable token threshold
                text:
                  'You are a knowledgeable assistant. ' +
                  'You have deep expertise in science, technology, history, literature, mathematics, and many other fields. '.repeat(
                    220
                  ),
                cache_control: { type: 'ephemeral', ttl: '5m' }
              }
            ]
          }
        ]
      }
    }
  };

  return new OrchestrationClient(orchestrationConfig)
    .pipe(new StringOutputParser())
    .invoke([
      {
        role: 'user',
        content: 'What is the speed of light?'
      }
    ]);

  const llm = new OrchestrationClient({
    promptTemplating: {
      model: {
        name: 'anthropic--claude-4.5-haiku'
      }
    }
  });

  const agent = createAgent({
    model: llm,
    tools: [],
    checkpointer: new MemorySaver(),
    middleware: [
      summarizationMiddleware({
        model: llm,
        trigger: { messages: 6 },
        keep: { messages: 2 }
      })
    ]
  });

  const config = { configurable: { thread_id: randomUUID() } };

  const questions = [
    'What is SAP Cloud SDK?',
    'What programming languages does it support?',
    'What is the main use case?',
    'How does authentication work?'
  ];

  let lastResponse = '';
  for (const question of questions) {
    const output = await agent.invoke(
      { messages: [{ role: 'user', content: question }] },
      config
    );
    // eslint-disable-next-line no-console
    console.log(`Message count after turn: ${output.messages.length}`);
    lastResponse = output.messages.at(-1)!.content as string;
  }

  return lastResponse;

[SanthoshCheekoti]

Hello - It is still failing for me.
I took Claude's help and I got below message. It is still failng for me. Can you check by using the tool in createAgent(), and a scenario that requires ReAct loop( so that orchestration client will be instantiated more than once )

The root cause is confirmed — XXX.promptTemplating.prompt is a singleton object that gets mutated by mergeOrchestrationConfig on the first tool call (appending explore_dataModel to .tools). On the second call (after the tool result), the same object already has [explore_dataModel] and appends again → duplicate.

const config = {
    ...orchestrationConfig,
    promptTemplating: {
        ...orchestrationConfig.promptTemplating,  // spreads top level
        model: { ... }                            // but prompt is NOT spread
    }
};
// This mutates the original orchestrationConfig.promptTemplating.prompt
config.promptTemplating.prompt.tools = [
    ...(config.promptTemplating.prompt.tools || []),
    ...tools.map(...)
];

if you see the langsmith, the issue is occuring when calling the createAgent method second time, the first time it is fine

[davidkna-sap]

Hello @SanthoshCheekoti,

I have implemented a potential solution for your issue in #1926, please confirm whether the canary release @sap-ai-sdk/langchain@2.11.1-20260611092914.0 resolves your issue.

[SanthoshCheekoti]

Hello @davidkna-sap ,
I am not seeing any error now. I can also see below as cached token

Thank you very much for the help