The AI Paradox: Why Your Data Team’s Workload is About to Explode

Chris Child, VP of Product, Data Engineering at Snowflake, joins High Signal to deliver a new playbook for data leaders based on his recent MIT report, revealing why AI is paradoxically creating more work for data teams, not less. He explains how the function is undergoing a forced evolution from back-office “plumbing” to the strategic core of the enterprise, determining whether AI initiatives succeed or fail. The conversation maps the new skills and organizational structures required to navigate this shift.

Chris Child, VP of Product, Data Engineering at Snowflake, joins High Signal to deliver a new playbook for data leaders based on his recent MIT report, revealing why AI is paradoxically creating more work for data teams, not less. He explains how the function is undergoing a forced evolution from back-office “plumbing” to the strategic core of the enterprise, determining whether AI initiatives succeed or fail. The conversation maps the new skills and organizational structures required to navigate this shift. We dig into why off-the-shelf LLMs consistently fail to generate useful SQL without a semantic layer to provide business context, and how the most effective data engineers must now operate like product managers to solve business problems. Chris provides a clear framework on the shift from writing code to managing a portfolio of AI agents, why solving for AI risk is an extension of existing data governance, and the counterintuitive strategy of moving slowly on foundations to unlock rapid, production-grade deployment.

Guest

Chris Child

VP of Product, Data Engineering at Snowflake

Key Takeaways

AI Currently Creates More Work Than It Saves for Data Engineering.

While AI assistants offer productivity gains, they unlock vast new datasets and drive demand for new pipelines, resulting in a net increase in data engineering work. A recent MIT survey found 77% of leaders see their data engineers’ workloads growing despite AI adoption.

LLMs Fail Without a Semantic Layer.

Off-the-shelf LLMs generate poor SQL because they lack business context. Chris found the key wasn't more training data, but providing a semantic model that defines core business concepts like "customer" or "revenue" for the AI to use.

Your CIO Doesn't Get Data Engineering's Value.

While 72% of executives see data engineers as integral to the business, only 55% of CIOs do. Chris suggests this is because CIOs remain platform-focused, while business owners and Chief Data Officers see the direct impact on revenue and insights.

The Future Data Engineer Manages Agents.

The role is evolving from an individual contributor writing code to a manager of agents. This involves setting budgets, defining goals, and providing architectural oversight to ensure a portfolio of automated pipelines delivers business value without creating chaos.

AI Risk Is a Governance Problem, Not a New Problem.

Trusting AI-generated outputs and pipelines doesn't require new technology. It requires extending existing data governance frameworks (like access controls, data quality checks, and security rules) to apply to agents, ensuring they inherit the same permissions as a human would.

To Move Fast on AI, First Move Slowly on Foundations.

Companies with a solid data foundation (unified data with strong governance) are deploying production AI applications far more quickly. They can bypass foundational work and focus on building, as Snowflake did with its internal go-to-market agent.

Data Engineers Must Become Product Managers.

The future value of data engineering isn't writing code, but asking "why" to understand the underlying business problem. This shifts the role from a technical executor fulfilling pipeline requests to a strategic partner who architects solutions for business outcomes.

The Most Important Skill is Killing Pipelines.

As AI makes it easy to spin up new pipelines, the critical skill shifts from building them to rigorously calculating their ROI. Data leaders must get good at identifying and shutting down low-value experiments to avoid sprawl and wasted cost.

You can read the full transcript here.

Timestamps

00:00 Introduction to Data Engineering Challenges

01:04 The Role of Data Engineers in AI

02:09 Chris Child's Insights on AI and Data Engineering

02:14 MIT Report and Data Engineering Evolution

03:12 The Growing Demands on Data Engineering

05:29 AI's Impact on Data Engineering Workloads

07:56 The Future of Data Engineering with AI

10:55 Challenges in AI-Assisted Data Engineering

21:12 Business Leaders' Perspectives on Data Engineering

26:03 Evaluating Business Value in Data Pipelines

27:33 The Evolving Role of Data Engineers

28:17 Addressing Risks and Governance in AI

31:55 Speed vs. Quality in AI Data Applications

35:32 Organizational Changes in an AI-First World

43:28 Career Advice for Data Engineers

45:48 Making Organizations AI Ready

49:14 Conclusion and Final Thoughts

Links From The Show

Ready to unleash your data?

Discover how Delphina can transform your data science.

Book a demo