Anthropic Just Made Code‑Execution Agents Easy (But Dangerous)

Too Long; Didn't Watch — Summary

Anthropic has introduced advanced tool use features for AI agents, including tool search, programmatic tool calling, and tool use examples, which significantly reduce token costs, improve accuracy, and enable complex multi-step workflows by allowing agents to discover tools on demand and execute code within a sandbox; however, these powerful capabilities also introduce new risks related to determinism, debugging, latency, safety, and implementation complexity, necessitating robust safeguards.

Main Takeaways

• Anthropic's new advanced tool use features (tool search, programmatic tool calling, tool use examples) address the inefficiencies and high token costs of traditional MCP (Multi-tool Code Execution) setups.
• Tool search allows agents to discover and load tools on demand, drastically reducing initial token consumption and improving accuracy by providing a more focused context.
• Programmatic tool calling enables agents to write and execute code (e.g., Python) to orchestrate tool usage, process data, and return only final results, leading to significant token savings, reduced latency, and enhanced accuracy.
• While these features offer immense power and flexibility, they also introduce challenges such as decreased determinism, more difficult debugging, potential latency for simple tasks, and significant safety concerns due to increased agent autonomy.
• Proper implementation requires setting up secure custom sandboxes for code execution and careful consideration of safeguards to prevent unintended or destructive actions.

Detailed Summary

Intro (0:00 - 1:25)

The video introduces Anthropic's new advanced tool use features, which address the limitations of classic MCP (Multi-tool Code Execution) servers that consume excessive tokens for tool definitions. The core idea is to let agents generate and run code for tools they need, rather than loading all tools upfront. Anthropic's new features include tool search, programmatic tool calling, and tool use examples, which are game-changers for builders but also pose dangers without proper safeguards. The video aims to explain these features, update previous advice on MCP, and demonstrate their application in a custom agent framework.

Anthropic's New Features (1:25 - 9:08)

• Problem with Traditional MCP: Traditional MCP setups load dozens of tools, burning 50-100k tokens on tool definitions before any user request.
• Tool Search: Agents now discover and load tools on demand, keeping only what's relevant for the current task, a significant mindset shift from loading everything upfront.
• Mechanism of Tool Search: Instead of all tool definitions, a single 'tool search tool' (around 500 tokens) is passed. Claude calls this tool with a query, and only relevant tool definitions (e.g., 3 out of 200) enter the context.
• Token Savings and Accuracy: This approach reduces token consumption from 77k to 8.7k tokens (95% context window preservation) and improves accuracy by preventing the model from being overwhelmed by a large list of similar tools.
• Implementation: To use, add tool_search_tool_regex to the tools array and set defer_loading to true for other tools. For MCP servers, deferring the entire server and overriding hot tools (e.g., search files) is possible.
• Programmatic Tool Calling: Addresses the issue of intermediate tool results polluting the context. Instead of multiple natural language tool calls, the model writes a Python script to call tools inside a sandbox.
• Example: For tasks like finding team members over budget, Claude writes a script that calls get_team_members, get_expenses, and get_budget_by_level, processing data internally and only printing the final JSON result.
• Benefits: Leads to 37% average token savings on complex research tasks, reduced latency due to fewer model round trips, and improved accuracy through explicit loops and conditionals in code.
• API Confirmation: This approach is now officially recommended by Anthropic, with a first-class API contract, making it more reliable and efficient than manual prompting.
• Implementation: Add code execution to tools; Claude writes orchestration code, which is then executed, and only the final result is provided back to Claude.
• Use Cases: Best for agents processing large datasets or running multi-step workflows with many tool calls.
• Tool Use Examples: A way to attach small, realistic examples of tool calls directly into the tool definition, showing the model what a good tool call looks like beyond just the JSON schema, improving performance, especially for tools with many optional fields.
• Combined Picture: Tool search for on-demand loading, programmatic tool calling for code-based orchestration, and tool use examples for proper call demonstration.

Why This is Dangerous (9:08 - 11:30)

• Less Deterministic: Workflows become less predictable; tools might not be found, or the agent might write incorrect orchestration code.
• Harder to Debug: Tracing platforms will mainly show final results, making it difficult to see and debug intermediate tool outputs within the executed code.
• Adds Latency: Introduces an additional code execution and sometimes a search step, which might slow down simple, single-tool tasks.
• Harder to Guarantee Safety: Increased agent autonomy allows for complex, un-designed combinations of tools, potentially leading to destructive actions like erasing tool ecosystems or wiping data if safeguards are absent.
• Harder to Implement Correctly: Requires setting up custom, secure sandboxes for code execution and result storage, which is non-trivial infrastructure work.

MCP Code Execution (11:30 - 15:17)

• The speaker demonstrates applying these ideas in their own agent framework with models that don't natively support Anthropic's new API features yet.
• The framework uses a 'cold email OS agency' that previously used 36 tools and 10-20 tool calls per task, leading to high token consumption.
• Problem Example: Asking the agent to find inboxes with incorrect settings previously resulted in over 20 tool calls and nearly 40,000 tokens consumed, even though it could iterate in a loop and only log relevant accounts.
• Solution: A new mcp_code_execution command in the framework converts the MCP server into individual tools, creates a schema file (similar to Anthropic's cookbook), and saves it as knowledge for the agent.
• Process: The framework automatically converts the server, generates schema files, tests tools, adds code execution tools, and adjusts agent instructions to use the new pattern, all step-by-step.
• Deployment: The agency is deployed on the Agency AI platform, which provides sandboxes.
• New Workflow: The agent now searches files (tool definitions uploaded as knowledge), then runs functions directly in code using an IPython interpreter tool, filtering inboxes and providing a summary.

Final Results (15:17 - 16:17)

• The token consumption for the demo task was significantly reduced from almost 40,000 tokens to 14,000 tokens using the MCP code execution approach.
• Agents have become proficient at writing code, making it more efficient to let them generate and use tools rather than building them manually.
• The speaker hopes more providers will integrate these features directly into their APIs to simplify development.
• Current challenges include the need for extensive prompting to guide agents and the infrastructure overhead of setting up secure sandboxes for code execution.
• The Agency AI platform offers secure sandboxes and persistent storage, allowing agents to evolve skills and reuse files across different interactions.

Notable Quotes

"Antropic has now taken this exact idea and baked it directly into Clot. They just released what they call advanced tool use, a tool search tool, so Claude can discover tools on demand instead of loading everything up front." "Agents should discover and load tools on demand, keeping only what's relevant for the current task." "If you have multiple MCPS and add 50 plus tools, tool search is a no-brainer." "From a builder perspective, this is basically official confirmation that just let your agents run the code is not a hack anymore. It's the recommended approach now." "The agent has way more autonomy now. It can write loops, conditionals, and combine multiple tools in ways you didn't explicitly design. The worst case, without proper safeguards, it can even erase your entire tool ecosystem or wipe out all the data if destructive tools are exposed." "The reason is because agents have become just way too good at writing the code. So it makes no sense for us to build the tools for our agents ourselves because they can literally do it better than we can."