Why Good Metrics Still Lead to Bad Decisions — and How to Fix It
Eoin O'Mahony—data science partner at Lightspeed, former Uber science lead, and co-designer of the system that kept NYC’s Citi Bikes available across the city—argues that positive metrics are meaningless if you don’t understand the mechanism behind them. At Uber, he was careful to make sure his launches both looked good on paper and made sense in practice. Now in venture, he’s applying that same rigor to unstructured data—using GenAI to scale a kind of work that’s long resisted systematization.
Eoin O'Mahony—data science partner at Lightspeed, former Uber science lead, and one of the early architects of the system that kept NYC’s Citi Bikes available across the city—argues that positive metrics are meaningless if you don’t understand the mechanism behind them. At Uber, he was careful to make sure his launches both looked good on paper and made sense in practice. Now in venture, he’s applying that same rigor to unstructured data—using GenAI to scale a kind of work that’s long resisted systematization.
Guest
Eoin O'Mahony
Partner, Data Science at Lightspeed Ventures
Key Takeaways
- Positive Metrics Can Mislead — Mechanism Matters More**
Eoin blocked product launches at Uber when teams couldn’t explain why* a metric had improved. Without a clear causal explanation, those “wins” often turned out to be losses.
- Early-Stage Products Don’t Need A/B Tests — They Need Impact**
For new products, Eoin didn’t bother with small-effect experiments. If a change didn’t create a visible spike in time series data, it wasn’t worth pursuing.
- Simple Algorithms Win in the Real World**
At Citi Bike, Eoin scrapped complex optimization for plain, actionable routing instructions that teams could follow. He optimized for execution, not elegance.
- Complex Systems Break Simple Assumptions**
Marketplace dynamics often invalidate basic experimentation. Eoin shows how network effects, time lags, and interactions can make even clean A/B tests misleading.
- GenAI Might Finally Scale Judgment Work in VC**
At Lightspeed, Eoin is building tooling to extract signal from unstructured data — applying scientific rigor in a domain that’s long resisted systematization.
You can read the full transcript here.
00:00 Positive Metrics Can Mislead — Mechanism Matters More
03:18 Eoin's Journey with Citi Bike
05:56 Challenges and Solutions at Citi Bike
14:05 Uber: From Two Wheels to Four
19:47 Navigating the COVID-19 Crisis at Uber
26:21 Transition to Venture Capital at Lightspeed
28:05 Joining Lightspeed: A New Opportunity
29:19 Exciting Projects at Lightspeed
30:08 Exploring Generative AI Tools
31:27 The Rapid Evolution of AI
37:01 Leveraging AI in Venture Capital
41:41 Future of AI and Personal Insights
52:29 Predictions and Closing Thoughts
Ready to unleash your data?
Discover how Delphina can transform your data science.
