Stanford AI Club: Jason Wei on 3 Key Ideas in AI in 2025

Too Long; Didn't Watch — Summary

Jason Wei, a leading AI researcher, outlines three core concepts for understanding AI in 2025: intelligence becoming a commodity, the 'verifier's law' which links AI's task-solving ability to the ease of solution verification, and the 'jagged edge of intelligence' suggesting AI progress will be highly uneven across different tasks, with digital, data-rich, and human-easy tasks advancing fastest.

Main Takeaways

• Intelligence and knowledge are rapidly becoming commoditized, leading to near-zero cost and instant access to information, democratizing fields and increasing the relative value of private information.
• The 'Verifier's Law' posits that AI's ability to solve a task is directly proportional to how easily its solution can be verified, making tasks with objective, fast, scalable, low-noise, and continuously rewarded verification metrics prime for AI conquest.
• AI's development will not lead to a sudden 'fast takeoff' of general superintelligence; instead, progress will be task-specific, creating a 'jagged edge' where some tasks improve rapidly while others remain largely untouched.
• Tasks that are digital, relatively easy for humans, and have abundant data are most susceptible to rapid AI advancement and automation.
• Future innovation opportunities lie in developing new measurement methods that allow AI to optimize previously unquantifiable tasks.

Detailed Summary

Introduction (0:00 - 1:12)

Jason Wei, a research scientist at Meta Superintelligence Labs and former OpenAI and Google Brain researcher, is introduced. He is recognized for his influential work on Chain of Thought reasoning, instruction tuning, and emergent phenomena, with over 90,000 citations. Wei plans to discuss three fundamental ideas for navigating AI in 2025.

The Spectrum of AI Impact (1:13 - 1:51)

Wei highlights the wide range of opinions on how AI will change the world, from skepticism about its current capabilities in specific jobs to predictions of job displacement within a few years. This broad spectrum underscores the need for a structured understanding of AI's future trajectory.

Three Core Ideas (1:52 - 2:41)

Wei introduces his three key ideas:

• Intelligence as a Commodity: The cost and accessibility of knowledge and reasoning will approach zero.
• Verifier's Law: AI's ability to train for a task is proportional to how easy it is to verify that task.
• Jagged Edge of Intelligence: AI's capability and improvement rate will vary significantly based on task properties.

Intelligence as a Commodity (2:42 - 10:11)

AI progress occurs in two stages: pushing the frontier (unlocking new abilities) and commoditization (making existing abilities cheaper). The cost of achieving a particular level of intelligence (e.g., on MMLU benchmark) decreases annually. This trend is driven by adaptive compute, allowing varied compute usage based on task difficulty, and the ability to retrieve public information almost instantly. Examples include finding specific historical data, which has evolved from hours (pre-internet) to minutes (internet era) to instant (chatbot era) and even complex data queries (agents era). OpenAI's browse comp benchmark demonstrates AI's progress in solving complex browsing tasks that challenge humans. Implications include the democratization of fields (e.g., coding, personal health), increased relative value of private information, and frictionless access to personalized internet.

Asymmetry of Verification and Verifier's Law (10:12 - 18:50)

Asymmetry of verification refers to tasks where verifying a solution is much easier than finding it (e.g., Sudoku, running Twitter). Other tasks, like factual essays, exhibit the opposite asymmetry (easy to generate, hard to verify). This concept can be visualized on a plane of generation ease vs. verification ease. Crucially, providing privileged information (like an answer key or test cases) can improve a task's verifiability. The 'Verifier's Law' states that AI's ability to solve a task is proportional to its verifiability. Verifiability is a function of objective truth, verification speed, scalability, low noise, and continuous reward. Most AI benchmarks are easy to verify, illustrating this law. DeepMind's Alpha Evolve is a prime example, solving complex tasks by leveraging this asymmetry with extensive compute and smart algorithms, generating and grading candidate solutions iteratively. This approach sidesteps generalization issues by focusing on problems where train and test are the same.

Implications of Verifier's Law (18:51 - 19:57)

Key implications of Verifier's Law are that the first tasks to be automated will be those trivial to verify. Furthermore, a significant area for future innovation and company building will be creating new ways to measure things, enabling AI to optimize them.

The Jagged Edge of Intelligence (19:58 - 28:10)

Wei argues against a 'fast takeoff' superintelligence scenario, where AI suddenly becomes vastly superior. Instead, he proposes a 'jagged edge' of intelligence, meaning AI's self-improvement will be gradual and task-specific. Different tasks will have varying rates of improvement; some (like hard math problems) will see rapid progress, while others (like speaking Flingit, a rare language) will improve slowly due to data scarcity or inherent complexity. He offers heuristics for predicting AI improvement:

• Digital tasks: AI excels due to faster iteration speeds and scalability of compute.
• Human-easy tasks: Generally easier for AI.
• Human-impossible tasks: AI may surpass humans in tasks requiring massive data processing (e.g., breast cancer prediction).
• Data abundance: AI thrives when data is plentiful, as seen in language model performance across different languages.
• Single objective metric: Tasks with clear evaluation metrics allow for synthetic data generation via reinforcement learning (e.g., AlphaZero).

Wei provides a table predicting AI's ability to perform various tasks, from debugging code (2023) and competition math (2024) to AI research (2027), chemistry research (later), and making a movie (2029). He also identifies tasks AI will struggle with, such as translation to rare languages, plumbing, hairdressing, traditional carpet making, and personal relationship management.

Conclusion (28:11 - 29:26)

In summary, Wei reiterates the three key ideas: intelligence and knowledge will become fast and cheap; Verifier's Law highlights measurement as a driving factor for AI progress; and the edge of intelligence is jagged, meaning AI's impact will be highly variable across different fields. Fields like software development will be heavily accelerated, while others like hairdressing will remain largely untouched.

Notable Quotes

"intelligence is going to become a commodity." — Jason Wei

"the ability to train AI to do a particular task is proportional to how easy it is to verify that task." — Jason Wei

"both the capability um and the rate of improvement of AI on a particular task is going to vary based on certain properties of those tasks." — Jason Wei

"once we sort of achieved abilities with AI, the cost of it will be driven towards zero." — Jason Wei

"any solvable easily verifiable task will eventually be conquered by AI." — Jason Wei

"nobody should give or receive any career advice right now. Everyone has broadly underestimated the scope and scale of change and the high variance of your future." — Jason Wei

"every task will have a different rate of improvement." — Jason Wei

"AI is good at digital tasks." — Jason Wei

"AI tends to thrive when data is abundant." — Jason Wei

"Any benchmark can be rapidly solved as long as the task provides a clear evaluation metric that can be used as a reward signal during fighting." — Danny do (quoted by Jason Wei)

"taking a girlfriend on a date that she's happy with. Uh, impossible, not digital, not easy to get data. So, I don't think we'll have this. I think we're we'll be in business for a while." — Jason Wei