Why Great Data Still Leads to Bad Decisions (And How to Fix It)

Amy Edmondson (Harvard Business School) and Mike Luca (Johns Hopkins) join High Signal to unpack what actually drives good decisions in data‑rich organizations. Using contrasts like the Bay of Pigs vs. the Cuban Missile Crisis and product cases such as Airbnb’s work on measuring discrimination, they show how decision quality tracks conversation quality—framing options, surfacing uncertainty, and challenging assumptions. We cover common failure modes (correlation vs. causation, anchoring, hierarchy, false precision), practical meeting designs that raise the signal, and where algorithms and LLMs help or hinder human judgment.

Decision quality is conversation quality.
Bay of Pigs vs. Cuban Missile Crisis shows the delta wasn’t data: iit was the rigor of debate, exploration of options, and pressure-testing assumptions.

Design the decision, not just the analysis.
Good outcomes come from structured meetings: name the uncertainty, surface dissent, switch sides, and expand the option set before choosing.

Measurement choices shape strategy.
What you track becomes what you optimize. Airbnb’s shift (measuring discrimination, adding guardrails, and changing product flows) shows how metrics redirect behavior.

Causality is earned, not assumed.
Correlation doesn’t settle managerial bets. Leaders must ask how the claim was identified (experiment? quasi-experiment?), what confounders remain, and what would change the decision.

Treat uncertainty as a first-class input.
Ranges, confidence intervals, and priors beat single point estimates. Decide for robustness across plausible outcomes, not for a fragile best guess.

Culture can overpower evidence.
Anchoring, hierarchy (the HiPPO), and confirmation bias routinely swamp data. Create norms and language to call these out in the room before they harden into decisions.

Weak signals deserve investigation.
“Not significant” isn’t “not real.” Probe subgroup effects, moderators, and time horizons: early warnings often hide in heterogeneity and short windows.

Generalize with care.
Ask what worked, why it worked, and where the mechanism holds. Internal validity is not external validity; portability requires understanding the “why,” not just the “what.”

Right-size rigor to stakes.
Save heavyweight process for high-stakes, high-uncertainty choices. Trivial decisions can run on heuristics; consequential ones need designed debate.

Algorithms assist; humans are accountable.
LLMs and models expand options, but outsourcing judgment is a risk. Use machines for scale and patterning; use humans for context, values, and novel possibility.

You can read the full transcript here.

Timestamps

00:00 Introduction to High-Stakes Decision Making

01:06 Meet the Experts: Amy Edmondson (Harvard Business School) and Mike Luca (Johns Hopkins)

02:25 The Importance of High-Quality Conversations

11:06 Case Study: Airbnb and Discrimination

13:57 Strategies for Effective Decision-Making

26:40 Pay Attention to Weak Signals

27:15 Statistical Significance and Variability

28:21 Confirmation Bias in Analytics

29:26 Generalizing Findings Across Markets

29:54 Bayesian Approach to Data Integration

32:59 Leadership and Decision-Making

36:09 The Role of Algorithms and Human Judgment

40:50 Data-Driven Leadership

46:34 Behavioral Economics and Decision Pitfalls

48:53 Concluding Thoughts and Takeaways