The Incentive Problem in Shipping AI Products — and How to Change It

Roberto Medri, VP of Data Science at Instagram, explains why most experiments fail, how misaligned incentives warp product development, and what it takes to drive real impact with data science. He shares what teams get wrong about launches, why ego gets in the way of learning, and how Instagram turned Reels from a struggling product into a global success. A candid look at product, data, and decision-making inside one of the world’s most influential platforms.

Shipping is not impact.
Roberto challenges the fetishization of launches, arguing that celebrating deployment over outcomes distorts incentives and leads to wasted effort.

Most experiments won’t move the needle.
He normalizes failure in experimentation, stressing that 90% of tests will be neutral or negative and that’s healthy. The value lies in the rare, high-leverage wins.

Impact comes from clarity, not cleverness.
Data science becomes strategic when it's paired with domain judgment. The goal isn’t to sound smart, it’s to change decisions.

Sunsetting is as important as shipping.
Roberto critiques the “ship and forget” mindset. Good product culture means knowing when to kill things quickly without ego or fear of embarrassment.

Reels was a bet, not a sure thing.
He recounts how Reels launched with weak metrics, but Meta backed the strategic bet anyway. Continued investment, especially in recommender quality, turned it into a global success.

Ego limits learning.
He argues that fear of failure, common among high achievers, suppresses experimentation. The most effective people optimize for progress, not personal validation.

Data science is a product role.
Roberto resists being cast as a support function. The best data scientists don’t just analyze, they help shape what gets built.

Most ideas should be killed faster.
He advocates for “impact accounting”: a mental model to quickly rule out low-leverage ideas before investing time.

Culture scales; incentives compound.
He emphasizes that early choices, about how teams define success, reward impact, and handle failure, set the trajectory for long-term product quality.

**Product judgment is earned, not inherited.**From Etsy to Instagram, Roberto highlights how thinking clearly about value, context, and outcomes is more powerful than frameworks or credentials.

You can read the full transcript here.

00:00 Introduction: The Limits of Data

02:21 Roberto's Career Journey: Etsy to Instagram and more!

08:59 How Ensure Data Science and Product Management Actually Work

14:12 Experimentation and Impact Accounting

19:37 Organizational Incentives and Product Lifecycle

26:18 Reframing Failure as Learning

26:54 Experimentation Culture in Software

31:10 The Importance of Bold Decisions

36:03 Evaluating Emerging Technologies

47:29 The Role of Data in Product Development

51:18 Practical Lessons for Smaller Organizations

53:18 Conclusion and Final Thoughts