AI's 70% problem w/ Addy Osmani

Too Long; Didn't Watch — Summary

Addy Osmani from Google's Chrome Developer Experience team discusses the "70% problem" in AI coding, where AI generates most code quickly but the remaining 30% (edge cases, security, integration) still requires significant human effort, leading to declining trust despite high adoption. He emphasizes the critical need for human understanding and review of AI-generated code, highlights that productivity gains are typically 1x-2x, not 10x, and notes that code review is becoming a new bottleneck. The discussion also covers the value of tests as feedback loops for AI, the concept of asynchronous agents and orchestration, and advice for junior developers on using AI as a learning tool while maintaining fundamental skills.

Main Takeaways

• AI coding tools can generate about 70% of a solution, but the remaining 30% involves complex tasks like edge cases, debugging, and integration, which are still highly time-consuming for humans.
• Despite widespread adoption of AI in coding, trust in AI-generated code is surprisingly low and declining, with many developers reporting little to no trust.
• The "two steps back" pattern frequently occurs, where AI's attempts to fix one bug can inadvertently create multiple new problems, necessitating careful human oversight.
• Human engineers must deeply understand AI-generated code, its architecture, and underlying decisions to ensure quality, security, and maintainability.
• Actual productivity gains from AI are modest (1x-2x), not the often-claimed 10x or 100x, especially for established, complex projects.
• Code review is emerging as a significant bottleneck due to the increased volume of AI-generated code, requiring new strategies to manage human oversight.
• Tests serve as crucial safety nets and feedback mechanisms for AI agents, helping to derisk AI coding and provide immediate insights into errors.
• Junior developers should use AI as a motivating learning aid to build quickly, but critically, they must dedicate time to understand the generated code and foundational concepts.

Detailed Summary

Introduction to AI in Coding and the 70% Problem (0:00 - 5:47)

• Addy Osmani, from Google's Chrome Developer Experience team, introduces the concept of the "70% problem" in AI coding, which he discussed at the Lead Dev conference.
• He differentiates between "vibe coding" (high-level prompting, rapid experimentation, prioritizing speed over correctness) and "AI-assisted engineering" (AI as a collaborator, human engineer in control, responsible for architecture, review, security, and maintainability).
• While AI adoption for coding is high (up to 90% of developers), trust in AI-generated code is declining, with favorable views dropping from 70% to 60% in two years, and 30% reporting little to no trust.
• The "70% problem" states that AI can quickly generate about 70% of an application or feature (scaffolding, obvious patterns), but the remaining 30% (edge cases, debugging, integration with production systems, security, API keys) remains as time-consuming as ever.

Navigating the Remaining 30% and Understanding Generated Code (5:47 - 12:32)

• The remaining 30% of the work can still leverage AI, but it requires a deeper, more iterative approach, often zooming in on specific edge cases.
• A fundamental step is for engineers to understand the AI-generated code, its architecture, and the decisions made, rather than just continuously layering prompts.
• The "two steps back" pattern describes how fixing one AI-suggested bug can lead to two or more new problems, creating a cycle of fixes.
• Maintaining critical thinking skills and learning from mistakes is crucial, suggesting that developers might benefit from occasional non-AI coding sprints to keep skills sharp.
• A "learnings.md" file can serve as a memory anchor, distilling insights after each task to improve subsequent AI runs and provide a reference for human understanding.
• Context engineering is vital for the remaining 30%, ensuring the AI model has sufficient information (docs, URLs, examples, team workflows) to handle complex, specific problems beyond straightforward tasks.

The Role of Tests and Productivity Claims (12:32 - 18:40)

• Tests are amplified in value with AI, acting as a feedback loop for agents and a safety net to derisk AI coding, catching regressions and providing instant feedback.
• While AI can write tests, human review is non-negotiable to ensure their quality and prevent a scenario where AI generates code, tests it, and even reviews it without human oversight.
• Productivity claims often cited (10x, 100x) are typically for greenfield projects without technical debt; real-world gains for established projects are more realistically 1x-2x, or about 20% more tasks completed.
• Increased code velocity from AI is making code review a new bottleneck, as finite senior engineers struggle to review the higher volume of generated code.

Team Dynamics, Responsibilities, and Asynchronous Agents (18:40 - 39:01)

• The example of Shopify shows designers using AI to vibe code prototypes, which are then implemented by developers, highlighting the need for clear roles and responsibilities and developer review for security and quality.
• Less technical individuals using AI may not recognize issues like scalability or exposed API keys, emphasizing the need for experienced human review or robust tooling.
• Teams need to establish processes for reviewing AI-generated code, assuming its use, and ensuring manual human sign-off before merging.
• Demarcating AI-generated code is generally not recommended, as engineers are responsible for 100% of the code they commit, regardless of its origin.
• AI can be surprisingly useful as a "learning buddy," allowing developers to ask questions about codebases, explore alternative approaches, and deepen their understanding.
• When AI provides information, especially about a codebase, it's crucial to verify its accuracy and guard against hallucinations, potentially by asking it to generate tests to prove its claims.
• The evolution from "conductor" (managing a single AI task) to "orchestrator" (managing a fleet of agents for multiple, interdependent tasks) is emerging, particularly for small to medium-sized projects.
• Orchestration can handle tasks like rewriting codebases or updating dependencies, but for large, legacy enterprise codebases, context window limitations and complexity still pose significant challenges.

Advice for Junior Developers and Team Personas (39:01 - 46:17)

• For junior developers, using AI for "vibe coding" to build quickly can be highly motivating, but it's essential to go back and understand the generated code and underlying fundamentals.
• AI can serve as an effective learning aid for foundational knowledge, but it should complement traditional learning methods like studying documentation.
• It's normal for learning to be imperfect, and continuous curiosity and openness to learning are key to overcoming initial misunderstandings or AI hallucinations.
• Teams often consist of different personas regarding AI: skeptics (fear replacement, distrust models), blockers (privacy concerns, lack of perceived value), practitioners (comfortable with AI tools, follow research), and evangelists (understand AI's power and weaknesses, pragmatic).
• The goal is to shift skeptics towards an "evangelist" mindset, balancing an understanding of AI's value with a pragmatic awareness of its current limitations.

Conclusion and Resources (46:17 - 47:34)

• Addy Osmani thanks the host and mentions his two new books: "Beyond Vibe Coding" (covering topics discussed in the interview) and "Building Web Apps with Bolt" (for new vibe coders).
• These books and free tips are available on his website, adosmi.com.

Notable Quotes

"AI coding tools can get you most of the way to a solution but not all the way." — Addy Osmani

"AI can rapidly produce maybe 70% of the code... but the remaining 30%... can be just as time consuming as it ever was." — Addy Osmani

"The fix is going to break something else. You're going to ask AI to fix that issue and it's going to create two more problems. Rinse, repeat. Sometimes it's five new problems." — Addy Osmani

"If you're using AI to generate the code, using AI to test the code... at some point you're probably going to try throwing AI into the code review loop as well. And at that point, AI is just doing the entire thing. You don't really know what's happening at all." — Addy Osmani

"Code review is becoming the new bottleneck." — Addy Osmani

"You need to take responsibility for 100% of that code regardless of how it was originally written." — Addy Osmani

"AI is actually really good as a learning aid." — Addy Osmani

"It's okay to be imperfect just as long as you continue to be curious and are open to learning." — Addy Osmani