Before answering, the question has to be split, because "AI engineer" currently names three jobs that share almost nothing except a hiring keyword.
Job | What it actually is | Entry requirement | How many roles |
|---|---|---|---|
Research | Advancing what models can do; publishing | Doctorate or equivalent, usually | Very few, globally |
Training and fine-tuning | Adapting models to a domain, running the pipeline | Strong maths plus systems, often a master's | A genuine but small speciality |
Building on models | Shipping products that call models | Software engineering, plus new failure modes | Nearly all of the demand |
Almost everyone asking the question means the third one and thinks they're being asked about the first. That confusion is where the bad advice comes from — people talk themselves out of an accessible role because they think it requires a doctorate, or into a mathematics curriculum they didn't need.
What does the labour data actually show?
The Stanford AI Index, drawing on Lightcast's billions of US job postings collected since 2010, puts AI skills in 2.5% of all US job postings. Python leads the specialised skills at 258,674 postings, up 391% from the 2013-15 baseline. Readers who reach machine learning through a growth or RevOps role will want the XenGrowth practice alongside this.
But the sentence worth acting on is a different one. The report notes that some of the fastest long-term growth came from deployment-oriented capabilities — Amazon Web Services, scalability, workflow management — and reads this as AI moving beyond experimentation into infrastructure, operations and execution.
The fastest-growing AI skills are cloud, scalability and workflow. Those are not AI skills. They are the skills of someone who can get a system into production and keep it there.
That is a shortage of engineers, not of researchers. The models are largely a solved procurement problem — you call an API. What almost nobody can do reliably is build a system around one that works, stays within budget, and fails safely when the model does something unexpected. On the operations side of this specifically, The XenGrowth resource library is worth reading.
What's genuinely different about building on models?
One thing, and it's substantial enough to justify calling it a speciality: the system is non-deterministic and can fail while appearing to succeed.
Conventional software mostly fails loudly. It throws, it returns a 500, a test goes red. You find out. A model returns a fluent, well-formatted, confident answer that is wrong, and nothing in your stack objects — which is exactly the failure mode Veracode measured at 45% for security specifically, and that Stack Overflow's respondents named as their top frustration.
Everything difficult about this speciality follows from that one property:
You cannot test it conventionally, because the correct output isn't a fixed string. You need evaluation suites that score behaviour statistically, which is closer to an experimental discipline than to unit testing
You cannot pin the dependency. The model changes underneath you, sometimes silently, and your regression suite is the only thing that will tell you your product got worse
Cost and latency are per-request and variable, so capacity planning is a live commercial concern rather than an afterthought
Failure has to degrade rather than crash, because the failure is a plausible wrong answer that will reach a user unless something catches it
Context supply is the real engineering. Everything the model needs and cannot see has to be found, ranked and delivered — which is retrieval, which is a systems problem
Read that list again and notice how little of it is machine learning. It's testing, dependency management, capacity planning, graceful degradation and data plumbing. It is conventional engineering applied to an unusually badly-behaved dependency. On AI agents and marketing automation specifically, XenGrowth on AI agents and marketing automation is worth reading.
Why does the evaluation skill matter so much?
Because it is the difference between engineering and hoping, and almost every team currently shipping AI features is doing the second one. Ask a team how they know their retrieval change was an improvement and the usual answer is that they tried a few queries and it seemed better. That is not a small gap in rigour — it is the same class of error METR measured when experienced developers judged themselves 20% faster while being 19% slower. Impressions of non-deterministic systems are close to worthless, and a system whose quality you cannot measure is one you cannot deliberately improve.
An evaluation suite does not have to be sophisticated to change this. A hundred real cases with expected properties, a scoring function you agree on in advance, and a number you record on every change is enough to convert arguments into measurements. It also gives you the only defence that exists against a model provider silently changing behaviour underneath your product, which is a dependency risk with no equivalent in conventional software: your library does not usually get quietly worse at its job between one Tuesday and the next.
This is why I would push an engineer toward evaluation before anything fashionable. It is unglamorous, nobody writes conference talks about it, and it is the single capability that separates teams shipping AI products that hold up from teams shipping demos that degrade. DORA's finding that AI amplifies whatever a team already is applies with particular force here: a team with measurement gets compounding improvement from every model release, and a team without gets a slow drift they cannot even detect.
Is the salary premium real?
Probably directionally, and I'd be careful with the numbers. Recruiting firms report US AI engineer medians somewhere between roughly $145,000 and $200,000 for 2026, with claimed premiums of 50-70% over comparable software roles.
Those come from job-posting scrapes and placement data rather than a statistical agency. The sample isn't controlled — AI roles cluster in well-funded companies in expensive cities, which inflates any raw comparison — and the figures vary enormously between sources, which is itself a signal about their reliability. Treat the direction as informative and any specific figure as soft. XenGrowth on AI search, GEO and discovery works through AI search, GEO and discovery in more operational detail.
There's also a timing argument worth stating plainly: a premium that large in a fast-growing category is partly a scarcity rent, and scarcity rents compress as supply arrives. Choosing a specialisation because of a wage gap measured today is a bet that the gap survives everyone else making the same observation.
One more consideration that rarely makes it into these decisions: the third category is the only one of the three where your existing experience is an asset rather than a sunk cost. A decade of shipping production systems counts for very little in a research hiring process, where the currency is publications. In product engineering on top of models it is close to the whole qualification, because the hard parts — reliability, cost, failure behaviour, knowing what production does to a design — are exactly what a decade teaches and what a model cannot supply. Switching fields discards that. Adding to it compounds.
So what should most engineers actually do?
Add, don't switch. The transition into the third category is much shorter than people assume, and it doesn't require abandoning what you already have — which is precisely the asset the market is short of.
Build an evaluation suite for something real before you build anything else. A set of cases, a scoring method, a number you can watch across model versions. This is the skill nobody has and everyone needs, and it is the one that makes every subsequent decision measurable rather than argued
Learn retrieval properly — chunking, embedding, ranking, and how each one fails. Context supply is the structural gap, and it stays a problem regardless of how capable models get
Instrument cost and latency per request from day one. A feature that works and costs more per user than it earns is a commercial failure, and it is the most common way these projects die quietly
Design the degradation path before the happy path. What does the product do when the output is confidently wrong? If the answer is 'show it to the user', you have not finished designing
Ship one such system end to end, at work if you can, on something small if you can't. One shipped system beats any certificate, because the thing you are demonstrating is that you know whether it works
If you're deciding based on... | That's a reason to... | Because |
|---|---|---|
Genuine interest in the internals | Consider research, seriously and slowly | It is a real career, but a poor hedge — do it for the work |
Wanting to build products | Add skills, don't retrain | This is your existing job with a new dependency |
Fear of being left behind | Stop and reframe | Fear picks the most visible option, which was prompting |
The salary premium | Be cautious | Recruiting-firm data, and scarcity rents compress |
Your current work being automatable | Move within your field first | Stanford's substitution split is about task shape, not job title |
The framing I'd resist is the one where this is a binary — stay a software engineer and become obsolete, or become an AI engineer and be safe. The data doesn't support either half. The BLS still projects 15% growth for software developers, and the Stanford AI Index's own finding is that what's growing fastest inside AI is the ability to deploy and operate systems, which is what software engineers already do.
The models are the easy part. They arrive working, behind an API, maintained by someone else. Everything around them is still just engineering, and engineering is still hard.
Further reading from XenGrowth
The XenGrowth resource library — what you'll learn: how the commercial side of this work is run, across search, automation and revenue operations.
XenGrowth on AI agents and marketing automation — what you'll learn: how the teams who own AI agents and marketing automation plan and measure it.
XenGrowth on AI search, GEO and discovery — what you'll learn: how the teams who own AI search, GEO and discovery plan and measure it.
Where this work meets go-to-market
If machine learning is part of a growth programme rather than a standalone build, XenGrowth's revenue operations work is the companion reading.
Four questions to separate the three very different careers that share the label. The advice diverges completely depending on which one you mean, and most people asking the question have not yet distinguished them.
