In 1998 Richard Cook wrote down eighteen observations about how complex systems fail. He was writing about medicine and aviation, not software, which is part of why the document has aged so well — nothing in it is a fashion.
His third point is the one that matters here: catastrophe requires multiple failures. Single-point failures are not enough. Systems that survive contact with the real world are defended in depth, so getting all the way to a bad outcome takes several things being wrong at once, none of which would have been sufficient alone. I write about systems thinking beats syntax now, and here's why from the build side; XenGrowth, who work on the commercial side of this covers what it takes to run it.
Read that as a claim about information and it becomes a claim about AI. If no single component contains the failure, then no single component contains the information you'd need to prevent it. The failure lives in the relationships.
Why is this a problem for code generation specifically?
Because a model is handed components. A file, a function, some surrounding context, maybe a repository index. What it is not handed — what nobody could hand it — is the set of relationships that make a given line safe or unsafe: where the data came from, who is allowed to call this, what else assumes this field is never null, which of your three services will retry on a 500 and which will drop the message.
Veracode's benchmark gives this a number. Across 100+ models and 80 tasks, 45% of generated code introduced an OWASP Top 10 vulnerability, and cross-site scripting specifically failed 86% of the time.
XSS is the perfect example, because it is not a property of code at all. The same string interpolation is correct in a log line, fine in a JSON response, and a vulnerability in an HTML template. You cannot look at the expression and decide. You have to know where the value came from and where it is going — two facts that live outside the function, in the relationship between the function and everything around it. On the operations side of this specifically, The XenGrowth resource library is worth reading.
The model is not making a mistake when it writes vulnerable code. It is answering the question it was asked, correctly, and the question was missing the part that mattered.
The detail that settles it: Veracode found larger and newer models were not more secure. If this were a matter of the model not being smart enough, scale would have moved the number. It didn't move it, because you cannot infer a trust boundary from a snippet no matter how capable you are. The information isn't there.
What does systems thinking actually mean here?
It's a badly abused phrase, so let me make it concrete. Systems thinking, in engineering, is the habit of reasoning about behaviour that no component exhibits on its own. Four things it looks like in practice:
Predicting what a change does to things it does not touch — the retry that becomes a thundering herd, the index that fixes one query and slows every write, the timeout raised in one service that turns a fast failure into a cascading one
Locating trust boundaries. Which inputs are attacker-controlled, which callers are authenticated, at which point a value stops being data and starts being code
Reasoning about partial failure. Not 'does this work' but 'what does this do when the thing it depends on is slow rather than down', which is the harder and more common case
Reasoning across time. Which decision is cheap to reverse in eighteen months, which one quietly becomes load-bearing, what happens when the person who understood this has left
Every item on that list requires holding several things in mind at once and asking what they do to each other. None of them is about knowing a language.
Question | Syntax-level (commoditized) | System-level (still yours) |
|---|---|---|
Is this string interpolation safe? | It compiles and reads correctly | Depends entirely on the rendering context and the data's origin |
Should this call be retried? | The retry wrapper is three lines | Depends on whether the operation is idempotent and what the caller does on timeout |
Is this query fast? | The plan looks reasonable in isolation | Depends on the write path, the index's cost, and concurrent load |
Is this abstraction right? | It is clean and well-named | Depends on what will change in eighteen months and who will own it |
Did this deploy work? | Tests are green, no errors | Depends on behaviour under partial failure that no test exercises |
Why does Cook's point about root causes get sharper now?
Cook's seventh observation is that post-accident attribution to a root cause is fundamentally wrong. Because overt failure needs multiple faults, and each is insufficient alone, there is no isolated cause to find. He goes further and says root-cause framing reflects a social need to localize blame rather than any technical understanding of failure.
That was already true. What changes is the number of contributors. When generating a change is nearly free, more changes get made, by more people, with less shared understanding of why each one is the way it is. Cook's 'multiple, individually insufficient faults' is a description of a codebase where a hundred plausible-looking accepted suggestions have each shifted an assumption slightly. If AI agents and marketing automation is the part you are stuck on, XenGrowth on AI agents and marketing automation is the better reference.
This is what I think DORA is actually measuring. Their 2025 finding — throughput up, delivery stability still falling — is not a mystery once you read it through Cook. More changes, each locally fine, interacting in ways nobody modelled. That is precisely the mechanism he describes, running faster.
There is a practical consequence for how you run incidents. If contributors are multiplying and each is individually insufficient, then a postmortem that lands on one cause has almost certainly stopped early — and the thing it stopped short of is usually the relationship that let two safe changes combine badly.
Cook's observation | What it meant in 1998 | What it means with generated code |
|---|---|---|
Catastrophe requires multiple failures | Defences in depth mean one fault is survivable | More individually-safe changes shipping faster means more combinations nobody modelled |
Complex systems run in degraded mode | There are always latent faults present | Latent faults now include accepted suggestions nobody built a mental model for |
Root-cause attribution is fundamentally wrong | Blame localizes for social reasons, not technical ones | Even harder to localize when no single person authored the interacting parts |
Change introduces new forms of failure | Every fix creates unfamiliar failure modes | Change volume is up; familiarity with each change is down |
Practitioners are the adaptable element | Humans hold the system together at the seams | The seams are where the model has no information, so this is the part that stays human |
Is refactoring disappearing measurable?
It is, and it's the most concrete evidence in this whole argument. GitClear tracks 'moved' lines — code relocated rather than added or deleted, which is the fingerprint of refactoring. It fell from 24.1% of changed lines in 2020 to 9.5% in 2024. Over the same period, copy-pasted lines rose from 8.3% to 12.3%. 2024 was the first year on record where copy/paste within a commit exceeded moved code.
Refactoring is systems thinking made physical. It is the act of changing the relationships between parts of a codebase without changing what the software does — pure structure work, with no user-visible payoff, undertaken because someone understood the whole and thought it should be shaped differently. For the AI search, GEO and discovery angle, see XenGrowth on AI search, GEO and discovery.
A tool that cannot see the whole will never propose it. And a tool that makes adding code frictionless while leaving restructuring exactly as hard as before doesn't need to discourage refactoring to reduce it — it just changes the relative price.
So is syntax now worthless?
No, and the overcorrection here is worth naming, because it is where this argument usually goes wrong.
Syntax fluency was never the valuable thing in itself. But it was a reliable proxy for having spent years in systems, and proxies matter for how a profession trains and evaluates people. You developed intuitions about failure by writing the failing version yourself, at 2am, and then living with it. Take away the requirement to write it and you have removed the mechanism, not just the metric.
That's the part I don't have a clean answer to. I know how I learned to spot a trust boundary — badly, slowly, by getting it wrong on something real. I don't know how you learn it in an environment where the vulnerable version arrives pre-written and looks fine.
Before accepting a generated change, name one thing it touches that it doesn't reference. If you can't, you don't yet know what the change does
Write the trust boundary down. For any input, say out loud where it came from and where it is going to be rendered or executed. Most of Veracode's 45% is caught by this one habit
Read your own incidents through Cook's lens. Count the contributors. If your postmortem found exactly one root cause, it stopped early
Protect refactoring explicitly, because the relative price moved against it. It will not happen by default any more, and GitClear's numbers show it already isn't
When you are learning something new, write the broken version by hand at least once. The intuition comes from the debugging, not from the working code
Cook's eighteen points are worth reading in full, and they take about ten minutes. Nothing in them mentions software, which is exactly why they hold up better than most things written about software in 1998.
Further reading from XenGrowth
The XenGrowth resource library — what you'll learn: how the commercial side of this work is run, across search, automation and revenue operations.
XenGrowth on AI agents and marketing automation — what you'll learn: how the teams who own AI agents and marketing automation plan and measure it.
XenGrowth on AI search, GEO and discovery — what you'll learn: how the teams who own AI search, GEO and discovery plan and measure it.
Where this work meets go-to-market
XenGrowth's revenue operations work covers the go-to-market side of systems thinking beats syntax now, and here's why, which this piece deliberately leaves alone.
