An Evaluation of Deepseek and How We Underestimate the Chinese
aired [03.31.2025]
Host: Harry Stebbings
Guest: Kevin Scott, CTO of Microsoft
Key Insights
Product Over Models: AI models hold value only when they fuel products that users need.
Scaling Still Has Legs: AI scaling laws haven’t hit their ceiling, promising more capability ahead.
Data Quality Wins: High-quality and synthetic data outshine raw volume for training smarter models.
Inference Drives Efficiency: Usage (inference) improvements are slashing costs as models grow.
Open and Closed Coexist: Both systems will shape AI’s future, offering flexibility and scale.
AI Code Boom: In five years, 95% of new code could be AI-generated, with humans steering the ship.
Societal Game-Changer: AI can revolutionize healthcare and beyond—if deployed responsibly.
1. Product Rules the AI Game
Kevin Scott stresses that AI’s real worth lies in products, not raw models. Technical fascination often overshadows user needs, but success hinges on delivering solutions people actually want.
Startups and big firms alike must iterate fast and test convictions against real data. Infrastructure matters, but only as a means to enable product creation.
Quote: "Models aren't products... the only thing that really matters is making good product."
“Models aren't products... the only thing that really matters is making good product.”
2. Scaling Laws: Room to Grow
Scott sees no immediate end to AI scaling laws. Models keep getting better at reasoning over complex tasks, with no plateau in sight—though he admits diminishing returns might kick in eventually.
Current progress is clear, with next steps already mapped out. Limits may exist, but they’re not yet visible or pressing.
Quote: "I don’t see the limit to the scaling laws... I can very clearly see what we’re doing now and what we’re doing next."
“I don’t see the limit to the scaling laws... I can very clearly see what we’re doing now and what we’re doing next.”
3. Data: Less Junk, More Gold
High-quality data trumps sheer quantity, especially in refining models post-training. Scott highlights synthetic data’s rise and the need for expert human feedback to boost reasoning over rote recall.
Low-quality web data is losing its edge. Measuring data’s true value remains an unsolved puzzle.
Quote: "High quality data is becoming much more useful in... the post-training parts of the model production pipeline than low quality data."
4. Inference: Cheaper, Bigger, Better
Inference—how models perform in real use—is outpacing training in focus and impact. Scott points to relentless optimization, making API calls cheaper even as models balloon in size.
Software stack improvements far outstrip hardware gains. Recent launches like DeepSeek R1 hint at this ongoing trend.
Quote: "Over time, the models have gotten bigger and the API calls have gotten cheaper."
5. Open vs. Closed: Best of Both Worlds
Scott predicts a future where open-source and proprietary AI systems thrive together. Like search engines, both will serve distinct needs—freedom for tinkerers, scale for big players.
Developers crave choice, pushing demand for variety. Neither model will dominate; coexistence is the norm.
Quote: "I think there’s going to be lots of both (open and closed systems)."
“I think there’s going to be lots of both (open and closed systems).”
6. AI Takes Over Coding—Almost
In five years, Scott bets 95% of new code will be AI-generated, but humans will retain authorship. Programming shifts to a higher abstraction, with engineers guiding AI rather than typing line-by-line.
Think less coding, more problem-solving. Top programmers will still debug and refine AI outputs.
Quote: "95% (of net new code) is going to be AI generated... very little is going to be line-by-line human written code."
7. AI’s Big Promise: Health and Beyond
Scott envisions AI transforming healthcare, already outdiagnosing average doctors. But its potential spans education, climate, and more—unlocked only with bold, responsible action.
Quote: "It’s already the case that... frontier models are probably better health diagnosticians than your average GP is... we have a whole world of people who have inadequate access to high-quality health care."