Simplifying Context Engineering for AI Agents in Production

Detailed Summary

Introduction and Housekeeping (0:08 - 2:39)

The session begins with housekeeping notes, including the webinar's estimated 45-minute duration, instructions for Q&A, and confirmation that a recording will be sent to registrants. Cornelia Davis, a Developer Advocate at Temporal, introduces herself, highlighting her background in distributed systems and microservices. Josh Smith, a Solution Architect at Temporal, also introduces himself, noting his two-year anniversary at Temporal and his experience in AI and building distributed systems.

Understanding Context Engineering (2:39 - 11:29)

This section defines context engineering as a superset of prompt engineering, focusing on managing all contextual information provided to an LLM, not just user prompts. It emphasizes the importance of structuring and curating this context to help LLMs make reasonable decisions and avoid issues like context poisoning, distraction, and clashes. The discussion highlights that context engineering is akin to a new form of programming, requiring careful consideration of what information to include and exclude from the LLM's context window. An image from Anthropic is used to illustrate the complexity of context engineering compared to simpler prompt engineering.

Common Context Engineering Struggles and Solutions (11:29 - 13:50)

The presentation outlines three key areas of context engineering that will be addressed: scoping agents, providing the right data at the right time, and managing conversation history. Josh Smith takes over to demonstrate the first two points, emphasizing the importance of giving agents focused tasks to ensure reliability.

Demo: Monolithic vs. Micro-Agents (13:50 - 29:35)

Josh demonstrates the pitfalls of monolithic agents by attempting to make a single agent perform multiple tasks. He first shows a working system with a micro-agent architecture for an order management system, where separate agents handle detection, analysis, planning, and reporting, with a human approval step. This system reliably identifies and resolves issues like low inventory or late payments. He then contrasts this by running a "monolith agent" that attempts to do everything. While the demo agent unexpectedly worked well, Josh explains that in previous tests, monolithic agents often became unpredictable, hallucinated problems, gave confusing responses, or were unsure about tool usage. He highlights that monolithic agents are harder to debug, more expensive to retry due to larger context windows, and lack crucial human-in-the-loop steps.

Design Guidance for Agents (29:35 - 35:36)

Josh provides design guidance based on his experience: agents should handle one specific task well (micro-agents), workflows should orchestrate multiple agents and their context, and agents should be given only the necessary tools and information. He strongly recommends separate planning, human-in-the-loop, and execution steps for better visibility and reliability. Creating reports is crucial for understanding agent actions, especially for monolithic agents. Temporal's child workflows or Nexus workflows are suggested for managing long-running sub-agents or tool calls. He concludes by stating that Temporal makes it easy to manage context effectively by returning only necessary information from activities.

Managing Conversation History and Context Pruning (35:36 - 43:00)

Cornelia Davis takes over to discuss managing long-running agent conversations and the need for history pruning. She explains that excessive conversation history can lead to LLM confusion and slow response times. Summarization is introduced as a key pruning technique, where a long history is condensed into a summary to seed future conversations, often using a separate LLM. Other techniques include trimming the oldest parts of the history. Additional concerns like auditability, provenance, and the mechanics of when and how to prune are also highlighted.

Demo: Conversation History Management with Temporal (43:00 - 53:26)

Cornelia demonstrates a chat interface where an LLM acts as both the assistant and a simulated user. She shows how a long conversation history can lead to the LLM (acting as the user) becoming confused and repetitive. To address this, she introduces a summarize command that, when triggered, uses Temporal's continue as new feature. This feature stops the current workflow run and starts a new one with the same workflow ID, but with the conversation history replaced by a summary generated by an LLM. This preserves auditability of the original conversation while providing a concise context for the new workflow run. She also demonstrates a durable timer that automatically triggers summarization after a period of user inactivity, even if the worker is temporarily offline, showcasing Temporal's ability to manage long-running processes and context reliably.

Conclusion and Resources (53:26 - 1:01:59)

Cornelia and Josh summarize the key takeaways: scoping micro-agents with Temporal activities or child workflows, managing durable and ephemeral context within workflows and activities, and using continue as new and durable timers for conversation history management. They provide resources including a blog post, demo code repositories, and invite attendees to a future webinar on "Human in the Loop" on November 6th. They also address audience questions, discussing how to prevent summarization from losing key details (using guard rails and evals) and the tools used (Temporal, Python, Light LLM). They explain how Temporal's workflow and worker architecture inherently handles concurrency and context isolation, preventing clashes between different agent conversations.

Notable Quotes

"Context Engineering is critical for making useful, production-ready agents." — Cornelia Davis

"This context engineering problem of deciding exactly what the right context is that you should be putting in and taking out it is if you will the new programming." — Cornelia Davis

"Agents work best when you give them a very specific, concise, focused uh amount of data and prompt and and if you can narrow that context in your context engineering and make give the agents focus, they tend to perform more reliably." — Josh Smith

"You have to realize that these models are trained to be helpful and to give kind of authoritative responses." — Cornelia Davis

"The other thing that you'll probably notice is that it's taking longer and longer to get the responses. And that's one of the other downsides of excessively long conversation histories is that it slows everything down." — Cornelia Davis

"Temporal makes it easy to fall into the pit of success. So when you're building with temporal agents and you follow these guidelines, it makes it pretty easy to manage context well because you just return back what you need um if you're used to building with that programming model." — Josh Smith

"You can use an LLM to implement some guard rails as well. That is one of the the most important techniques." — Cornelia Davis

"You want to establish evals... so that you can constantly be well evaluating how your LLMs are performing so that you can get alerts if they start going off the rails." — Cornelia Davis

"The agentic frameworks are great and we do in fact integrate with a number of them including pyantic AI and the open AI agents SDK and where we can bring durability to those. They're generally not durable otherwise. They just are libraries that run in a single process." — Cornelia Davis