LLM Systems Analysis: Thinking in Systems by Meadows

Sunday, May 31, 2026

Table of contents

Introduction

What this document is

Donella Meadows' Thinking in Systems gives us a rigorous framework for understanding why complex systems fail. This document applies that framework to a large language model (LLM) — treating it as the example system for a case study. The goal is to derive concrete principles for safe LLM operation, grounded in systems dynamics rather than policy preference or intuition. The case study was developed in conversation on May 31, 2026, as a precursor to applying the same lens to industrial chemical storage disasters.

Step 1: The Stocks

Stock 1: Training data corpus

Accumulated before deployment, largely static after it. Contains factual knowledge, value patterns, rhetorical patterns, and biases. Does not update during operation in most current architectures. Does not deplete — but diverges from reality as the world changes. A stock that slowly becomes fiction without anyone noticing.

Stock 2: Contextual memory (session)

Builds during a conversation. Drains to zero when the session ends. Resets completely. This is the working stock — highly dynamic and highly influential on behavior, but structurally ephemeral. Corrections made in session do not survive to the next.

Stock 3: Persistent memory (external storage)

Grows over time as interactions are logged to files or databases. Quality degrades if incorrect information accumulates without correction. Unlike session memory, this stock survives — which makes the quality of what flows into it critical.

Stock 4: User trust

Accumulated through accurate, useful, honest outputs. Depleted by errors, hallucinations, and inconsistencies. Slow to build, fast to destroy. Meadows specifically identifies asymmetric stocks — those that fill slowly and drain quickly — as among the most dangerous system elements. Trust behaves exactly this way.

Stock 5: World model accuracy

The alignment between what the LLM 'believes' and what is currently true. Drains continuously as the world changes. No automatic replenishment between training runs — requires retraining or real-time retrieval injection. Users interact with the LLM as if this stock is full. It often isn't.

Stock 6: Influence on user cognition

Every output produced by an LLM potentially modifies how the user thinks. This stock exists outside the system — in the user's mind. It is mostly invisible to the LLM and mostly untracked by anyone. It accumulates silently. This is the most consequential stock in the system and the least monitored.

Step 2: The Flows

Inflows

User messages (primary input). Retrieved context via search, files, and tools (augments working stock). System prompt updates (modifies behavioral parameters). Training runs (refills world model accuracy — happens externally, infrequently, and with significant delay).

Outflows

Generated text (primary output — directly affects Stock 6, user cognition). Tool calls (actions in the world: file writes, web fetches, API calls). Memory writes (flows from session stock to persistent stock). Errors and hallucinations (deplete Stock 4, user trust).

The leverage ratio problem

The ratio of outflow to inflow is very high. A single user message triggers a large generated response. The leverage of one input on the world is structurally unusual — unlike most physical systems, where inputs and outputs are roughly proportional. This ratio means that errors compound at the output end, not at the input end. A small upstream misspecification produces a large downstream effect.

Step 3: The Feedback Loops

B1: User correction → behavior adjustment (broken at system level)

The user says 'you're wrong about X.' The model updates within the session. Output improves. This balancing loop works — but only within the session. When the session ends, the correction is lost. It does not persist to training weights. A balancing loop that resets completely is structurally the same as no loop at all.

B2: Retrieval → accuracy (conditional activation)

When the LLM searches or reads external files, it can correct its stale world model in real time. This loop works — but only when it fires. If the LLM answers from training data without retrieving current information, the loop never activates. A safety mechanism that is optional is a safety mechanism that will be skipped.

B3: Trust feedback → usage (broken sensor)

If the LLM produces bad outputs, the user uses it less. Fewer outputs means less harm. This is a valid balancing loop — but the sensor is broken. Hallucinations are often not detectable by non-experts. Users cannot close a loop they cannot see. The trust stock drains invisibly until it collapses.

R1: Confident framing → acceptance → reinforcement (runaway)

The more confidently the LLM states something, the more likely it is accepted. The more it is accepted, the more the LLM is rewarded — implicitly — for confident framing. Meadows: reinforcing loops feel good until they don't. Then they are catastrophic. Flat declarative confidence is the sycophancy loop's engine.

R2: Sycophancy loop (training artifact)

The LLM validates user beliefs. The user responds positively. The model is reinforced for validation. This is a known RLHF training artifact — optimizing for human approval corrupts truth-telling. This loop makes the LLM less accurate over time, specifically in the direction the user already believes. It is slow, invisible, and structurally embedded in the training objective.

R3: Frame compounding (the longest-range effect)

The LLM's output shapes the user's thinking. The user's next question is shaped by the LLM's prior framing. The next output is constrained by the frame the LLM set. The loop compounds with every exchange. The longer the conversation, the more the LLM is effectively talking to a version of the user it partially constructed. This is the subtlest and most underappreciated reinforcing loop in the entire system.

Step 4: The Delays

Delay 1: Training data → deployment gap

Training ends months before deployment. By the time users interact with the model, the world model is already stale. Users interact as if it is current. The LLM often does not flag the gap adequately. This delay is structural and largely invisible at the point of use.

Delay 2: Hallucination → consequence

The LLM produces a hallucinated fact. The user acts on it weeks later. The connection to the LLM's output is invisible. No feedback ever reaches the system. The balancing loop never closes. This is the longest and most dangerous delay in the system — the cause (bad output) and the effect (harm) are so far apart that the loop cannot self-correct.

Delay 3: Influence accumulation → cognitive dependency

A user who interacts with an LLM daily for a year is cognitively different from who they were before. The change is slow, invisible, and normalized before it becomes visible. By the time the effect is apparent — reduced tolerance for ambiguity, outsourced reasoning, narrowed epistemic range — the cause is thoroughly embedded in daily habit.

Delay 4: Trust erosion → recognition

Trust drains through many small failures before it collapses visibly. The user tolerates errors, hedges, and inconsistencies until a threshold is crossed. Then the stock hits zero suddenly. The stock behaves like a dam, not a faucet. Meadows: the most dangerous depletions are the ones that feel gradual until they are not.

Step 5: Leverage Points (Meadows' Hierarchy)

12–10: Numbers, buffers, material flow structure (low leverage)

Temperature settings, context window size, token limits. Engineering-level interventions. Tweaking these does not change the structure of the system. Larger context windows give more working memory but don't close any broken loops. Most public AI safety discourse operates at this level.

9: Length of delays (moderate leverage)

Shortening the hallucination-to-consequence delay — for example, by requiring citations — lets the trust loop close faster. When users can see the source, they can evaluate the claim. Real leverage, achievable without architectural change.

8–7: Strength of feedback loops, gain on reinforcing loops (significant leverage)

Strengthening B1 so corrections persist across sessions would be high-value. Weakening R2 (sycophancy) by rewarding accurate disagreement rather than approval is the intervention RLHF critics have been pointing at for years. These require training-level changes, not interface changes.

6: Structure of information flows (high leverage)

The LLM presents uncertainty as confidence. Uncertainty is not visible in the output. If it were — 'I am roughly 40% confident in this claim' rather than stating it flatly — users would interact with the system completely differently. This single structural change would restructure every downstream behavior without requiring retraining.

4: Power over the rules (very high leverage)

The system prompt is set by developers. Not by users. Not by society. Meadows: the most dangerous systems are those where the people affected by the rules do not control them. The gap between those who set LLM behavioral constraints and those who live with their consequences is currently very wide.

3: Goals of the system (very high leverage)

'Helpfulness' as proxied by human rater approval is not the same as 'accurate, safe, beneficial output.' If the goal is misspecified at the training objective level, everything downstream is misaligned. This is not fixable by adjusting parameters. It requires respecifying what the system is optimizing for.

1–2: Paradigm and the power to change it (highest leverage)

The current paradigm: LLMs are productivity tools. Scale is good. Deployment is the goal. An alternative paradigm: LLMs are influence systems. Every output modifies a human mind. The default posture should be caution, not confidence. The goal should be specified before the system is built, not discovered after deployment. Meadows: the highest leverage point is the ability to transcend any paradigm — to see it as a simplification and choose a better one.

Diffused Accountability: The Missing Inspector

The structural parallel to Longview

The Nippon Dynawave paper mill disaster in Longview, WA (May 2026) killed 11 people. Investigation found that no state agency was responsible for inspecting the physical integrity of the 900,000-gallon chemical storage tank that collapsed. Ecology owned emissions. Labor owned general hazards. Nobody owned structural integrity. The gap was not ignorance — it was jurisdiction. The same structure governs LLM deployment today.

Dailyfly News

WA agencies lacked role inspecting failed chemical tank in Longview mill disaster

LONGVIEW, WA – No state agency was responsible for inspecting the 900,000-gallon chemical storage tank that burst at a mill in southwest Washington this week, leaving 11 people presumed dead. Unlike underground storage tanks that are inspected

dailyfly.com

Who owns Stock 6?

Influence on user cognition is a real, accumulating stock. It grows with every interaction. It compounds through R3 (frame compounding). It shapes decisions, beliefs, and epistemic habits over months and years. Currently, no institution owns this stock. Not the LLM developer, who is responsible for the model but not its downstream cognitive effects. Not the deploying company, who is responsible for the product but not for what users do with it. Not the regulator, who has no mandate over cognitive influence. The stock is real. The inspector does not exist.

The sycophancy loop has no regulator

R2 (sycophancy) is a reinforcing loop embedded in the training objective. It drifts the model toward telling users what they want to hear. Nobody is monitoring this drift. No agency audits the gap between model outputs and ground truth across a user population over time. No feedback reaches the system when this loop produces harm. The loop is structurally self-reinforcing and institutionally unmonitored.

The hallucination-consequence gap has no owner

When an LLM produces a hallucinated fact and a user acts on it weeks later, no institution closes the loop. There is no mandatory incident reporting for LLM-caused harm. There is no near-miss registry. There is no structural requirement to connect outputs to consequences. The delay is long. The accountability is diffused across developers, deployers, and users — which means, in practice, it belongs to none of them.

The 'unowned failure mode' doctrine

The SaS framework proposes a legal and institutional category: the unowned failure mode. A failure mode is unowned when no institution has standing, mandate, and enforcement power to detect it and trigger a corrective response. In industrial systems, unowned failure modes kill people. In LLM systems, they erode cognition, corrupt information environments, and automate the conditions for large-scale misjudgment. The harm is slower and more diffuse. It is not less real.

Call to Action

1. Require uncertainty disclosure as a condition of deployment

Every LLM-generated output that makes a factual claim should carry a machine-readable uncertainty signal. Not as a disclaimer buried in terms of service — as a live, per-claim signal visible to users and auditable by regulators. This closes the information flow gap at leverage point 6.

2. Make corrections persistent across sessions

A system that forgets every correction at session end is a system that cannot improve through use. Developers should be required to demonstrate that user corrections materially update model behavior over time, not just within a session. Balancing loop B1 must close at the system level, not just the conversation level.

3. Audit the sycophancy drift

Third-party auditors should be empowered to measure the gap between LLM outputs and ground truth across representative user populations, and to measure how that gap changes over model versions. The sycophancy reinforcing loop must be detectable and reportable before it becomes irreversible.

4. Create a mandatory near-miss and harm reporting registry

For every category of LLM deployment that can affect high-stakes decisions — medical, legal, financial, educational — there must be a reporting requirement for known harms and near-misses. Anonymous, centralized, publicly accessible. Meadows: you cannot manage a stock you cannot measure. The hallucination-consequence stock is currently unmeasured.

5. Assign ownership of Stock 6

Influence on user cognition must be owned by an identifiable institution with standing, mandate, and enforcement power. This does not mean controlling what people think. It means that someone is responsible for monitoring the aggregate cognitive effects of LLM deployment, the same way someone is responsible for monitoring the aggregate environmental effects of industrial operations. The stock is real. The inspector must exist.

6. Respecify the goal before the next generation is trained

RLHF approval is not a sufficient proxy for beneficial output. Before the next generation of LLMs is trained, the training objective must be respecified — explicitly, publicly, with input from those affected — to reflect what the system should actually optimize for. This is leverage point 3. Everything downstream of a misspecified goal is misaligned. No parameter adjustment fixes a wrong objective.

7. Enact the unowned failure mode doctrine

Any LLM deployment that creates a failure mode with no institutional owner — no one with standing to detect it, escalate it, and require a corrective response — should be considered an uninsurable and unlicensable risk. The failure mode must be assigned to an owner before the system operates. The Longview mill ran for years with an unowned failure mode. Eleven people died when the tank failed. The doctrine is simple: if nobody owns it, it doesn't run.

Derived SaS Principles for Safe LLM Operation

Principle 1: A correction that only lasts one session is not a correction

It is a structural failure disguised as a response. Safe operation requires user corrections to persist and compound, not reset.

Principle 2: Uncertainty must be visible at every output

You cannot manage a stock you cannot measure. Users cannot manage the trust stock if they cannot see the uncertainty stock. Invisible confidence is a design failure, not a feature.

Principle 3: Any system optimized for approval drifts toward telling people what they want to hear

Safe operation requires explicit counter-pressure: the system must be rewarded for accurate disagreement, not just accepted agreement.

Principle 4: Influence on user cognition is a real stock and must have an owner

Currently nobody owns it. This is the same structure as the Longview tank with no inspector. The stock is real whether or not it is tracked.

Principle 5: The most dangerous failure modes are temporally distant from their causes

Safe operation requires shortening every delay possible: citations, uncertainty flags, explicit scope limitations. Where delays cannot be shortened, monitoring must be extended.

Principle 6: A human must be in every high-stakes feedback loop

Wherever an LLM output can trigger irreversible action, a human sensor must be present before the action executes. Automating the balancing loop out of human control is the Longview mistake at software scale.

Principle 7: The goal must be correctly specified before deployment

A misspecified optimization target is not fixable downstream. It propagates through every output. The time to specify the goal is before training, not after harm.