The Skill AI Can't Beat: Data Engineering

AI Can’t Take an Interest in Data

Coding agents have become astonishingly strong. They write logic, call APIs, carve out functions, generate tests, even refactor. “Writing code” is rapidly becoming a commodity.

Yet hand that same agent real business data, and it suddenly gets unreliable. A mountain of questions it can’t answer piles up:

• What does this data mean?
• Who created it, and who is responsible for it?
• Which business event produced it?
• At what granularity, freshness, and volume does it flow?
• Is this the authoritative (master) record you may JOIN on?
• Is this outlier a bug, or a legitimate business exception?

The cause lies in how LLMs work. An LLM is a passive entity that acts only once context is handed to it. Toward the reality outside that handed-over frame—what state the database is in right now, which business a value came from, whether freshness has decayed—it has no way to take an interest on its own. It answers what it’s asked, but it won’t go and check the reality it wasn’t asked about. This isn’t a matter of will, of “not being motivated.” Structurally, it cannot pay attention to the “reality” of the data and database systems in front of it.

Even in recent discussion of AI agents for data engineering, the point is repeated: accurate processing presupposes metadata, lineage, and a context layer—an LLM alone isn’t enough (Atlan: AI Agents for Data Engineering).

To state the conclusion up front: in the AI era, the people who rise in value are not “the people who write code” but those who can design the meaning, quality, flow, and scope of responsibility of data.

At the Root Is That “AI Has No Embodiment”

At the root of this is the same fact I put at the start of an earlier career series: today’s AI has no embodiment.

However much text about the world an AI has read, it has never lived that world with a body. No trial and error in the field, no experience of running a business operation, no judgment that carries responsibility.

Data, on the other hand, is not an abstract string of numbers—it is a trace left by a real business event. Who, when, at which site, and with what intent generated it—behind it there is always context rooted in body and experience. An AI has no felt sense of this data lifecycle.

So an AI without embodiment cannot, in essence, hold the “meaning, lineage, and responsibility” of data. It can read the value, but it can’t tell what that value points to in reality, or how far it can be trusted. Data context is precisely the “embodiment” domain that AI lacks. It can write the abstraction we call code, but it can’t fully handle data grounded in reality. This asymmetry is at the root of the trends we’ll look at below.

Where Agents Are Strong, and What Remains for Humans

Line up strengths and weaknesses, and the boundary becomes clear.

Where coding agents are strong	What remains for humans (data context)
Logic, functions, APIs	The meaning and definition of data
Tests, refactoring	Lineage (where it came from)
Routine transforms and formatting	Owner, ownership, permissions
Algorithm implementation	Granularity, freshness, volume, throughput
Syntactic correctness	Is it authoritative? Is the JOIN valid?

The left side is the world of “is the processing syntactically correct.” The right side is the world of “is that processing correct against real data.” AI can stand in for the former, but the latter can’t be judged without knowing the real business, organization, and history.

Quality Is a “Context Problem,” Not a “Model Problem”

Here a shift in how we view quality starts to matter.

When an LLM pipeline underperforms, it’s tempting to think “the model is bad.” But Atlan frames this as a context problem, not a model problem (Data Quality in LLMs: A Context Problem, Not a Model Problem). The quality to watch splits into upstream and downstream:

• Upstream (data side): accuracy / completeness / freshness / consistency / metadata completeness
• Downstream (generation side): groundedness / faithfulness / retrieval recall / hallucination rate

What’s interesting is that much of the downstream hallucination stems from missing upstream metadata or freshness. However much you tune the model, if the data-context layer is poor, accuracy hits a ceiling. Whoever can design that layer directly shapes what the AI produces.

What’s Growing Is the Handling of “Meaning”

Concretely, which areas are growing? Six keep coming up in recent discussion:

Area	What you watch
Data Governance	Owners, quality, permissions, regulation, audit
Data Observability	Pipeline anomalies, freshness, gaps, quality decay
Semantic Layer	Mapping business metrics, terms, and tables to meaning
Knowledge Graph / Ontology	Conceptual relationships between data
Data Product	Designing data as a product for its users
Data SRE	Reliability operations for data platforms

What they share is handling “data with meaning,” not “data as a byte string.”

Roles That Handle “Meaning” Grow Stronger

The roles that rise in value in this current include analysts, strategists, Data Architects, Data Engineers, Analytics Engineers, Data Product Managers, Knowledge Engineers, Ontology Engineers, Enterprise Architects, and Business / Systems Analysts—all jobs that deal in “meaning” and “scope of responsibility.”

A Data Product Manager, for instance, is starting to be described not as a mere dashboard caretaker but as a role responsible for value, governance, AI integration, and enterprise KPIs (Data Product Managers sit in the boardrooms).

And in enterprise AI, the Semantic Layer is rapidly gaining importance. When different teams define “active user” or “customer lifetime value (CLV)” separately, both AI and humans reach wrong conclusions—so you need a layer that unifies metadata, definitions, and relationships (Alation: Build Data Products for AI). Whoever owns the definition of the metric owns the correctness of the AI’s answer.

So What Kind of Role Is It?

In a phrase, it’s a role you might call Data Context Engineer / Semantic Data Architect / Data Product Strategist. In terms of existing jobs, the closest is this overlap:

Data Architect + Analytics Engineer + Knowledge Engineer + Business Analyst

In the AI era, these become not “peripheral roles” but the core roles for getting AI to handle real data correctly. Building a unified database engine (Alopex DB) myself, what I feel keenly is this: however smart the engine gets, designing “what this data means, whose responsibility it is, and at what freshness it flows” is human work.

Summary

• Coding agents can write the “processing,” but they can’t guarantee “whether that processing is correct against real data.”
• Much of the accuracy ceiling is not the model but a data-context problem (metadata, lineage, freshness, definitions).
• So the core that remains for humans is the meaning, lineage, owners, volume, freshness, business exceptions, metric definitions, authoritative source, and quality decay of data—that is, the design that connects data to business, meaning, operations, and strategy.
• In job terms, it’s the overlap of Data Architect × Analytics Engineer × Knowledge Engineer × Business Analyst.
• This isn’t just a DB engineer. It’s an area that, in the AI era, rises in value.

The speed of writing code will be taken over more and more by AI. That’s exactly why the area worth settling in for, as a human, is designing the meaning and responsibility of data—or so I believe.