Abstract
This article examines a recurring failure in the design and governance of systems: the conversion of persistent dissatisfaction into evidence of a defect in the people served rather than a reason to revise the system’s model of them. I call this defect attribution. A parable of a tailor who blames his customer for two ill-fitting jackets frames the analysis. Agent Smith’s account of the first Matrix illustrates the same inference applied to an entire designed world, and Nozick’s experience machine supports an alternative explanation: a design that delivers outcomes can still omit the activities and forms of agency that make those outcomes valuable. The analysis then turns to artificial intelligence, where a single request may express several purposes. It argues that good assistance must distinguish delegated work from participation a person wishes to keep, without compelling instruction any more than it compels dependence. Drawing on research on automation, self-determination theory, the capability approach, and Hirschman’s account of exit and voice, it distinguishes forms of dependence that preserve agency from forms that erode it. The contribution is a conceptual framework and a proposed audit, not an empirical finding about any particular institution. Its practical implication is that dissatisfaction should be routed into inquiry, and that a system’s success should be measured partly by what it enables people to understand, choose, contest, and leave.
1. Introduction: The Reversal of Purpose
A tailor makes a jacket for a customer. It is too large. He makes a second, and it is too small. After the second complaint he concludes that the customer is impossible to satisfy. Perhaps they are ungrateful; perhaps they do not know what they want; perhaps they should make more effort to fit a perfectly reasonable jacket. He stops trying to make a jacket for the customer and begins trying to make the customer fit the jacket.
The parable is deliberately absurd. A tailor’s function is defined by the person being fitted, so when he treats the customer’s body as the obstacle to his design, he has reversed the purpose of his own work.
The same reversal is difficult to detect at larger scales. Institutions, economic arrangements, and technologies are built on models of the people who will use them: what they need, how they will behave, what will count as success. When outcomes diverge from the model, two broad responses are available. The designers can revise the model, or they can reinterpret the divergence as a property of the people. The second response is often cheaper, more flattering to those in charge, and easier to justify in the vocabulary of expertise.
This article makes four claims:
- Defect attribution is a distinct failure pattern. It is not identical to bad design, because a well-intentioned system can fall into it, and a poor system can avoid it by learning quickly.
- A system can deliver every requested outcome and still fail people, because outcomes and the activities through which they are pursued are not interchangeable.
- The relevant question about AI and other forms of assistance is not whether people should depend on them, since all human capability involves dependence, but whether that dependence preserves the ability to understand, evaluate, contest, and redirect.
- Responsiveness requires mechanisms, not merely attitudes. Its presence can be assessed.
2. The Defect-Attribution Pattern
2.1 Three explanations for dissatisfaction
When people are persistently dissatisfied with a system, at least three explanations are possible:
- Unmet need within the model. The system’s model of the person is broadly right, but delivery falls short. The person needs help, information, or a better implementation.
- Model mismatch. The person’s needs, purposes, or circumstances differ from what the designers assumed. The jacket was cut for someone else.
- Rational resistance. The person correctly perceives that the arrangement works against their interests, perhaps because it serves another party at their expense.
A fourth explanation, that the person is mistaken about their own interests, is real and is addressed in Section 7. The difficulty is that defect attribution collapses all four into it. Every complaint becomes evidence of confusion, ingratitude, or resistance to progress.
2.2 Why the pattern protects the system
Defect attribution is self-sealing. If dissatisfaction is always the subject’s fault, it can never count as evidence against the design, so the design is never revised. The more confident the designers are in their model, the more readily they will interpret contrary evidence as noise.
The pattern also carries an incentive structure. Revising a model requires admitting error, spending resources, and often redistributing control. Attributing failure to users requires none of these things. It can even justify expanding the system’s authority, since people who cannot be satisfied must presumably be managed, educated, or corrected.
James C. Scott’s Seeing Like a State documents large-scale versions of this dynamic: schemes to improve the human condition that failed because they imposed simplified, legible models on complex local realities and disregarded the practical knowledge of the people subject to them (Scott, 1998). The relevant lesson is not that planning always fails. It is that planning which cannot absorb correction from those it affects tends to treat their resistance as an obstacle rather than as information.
2.3 What the pattern does not claim
The argument does not hold that every complaint is valid, that satisfaction is the only standard of a good system, or that designers should defer to every stated preference. Some limits are necessary. Some requests are harmful. Some dissatisfaction reflects conflicting interests that no design can fully reconcile.
The claim is narrower. Dissatisfaction is evidence whose meaning must be investigated. A system that cannot distinguish among the explanations in Section 2.1 has lost the capacity to respond competently to any of them.
3. Outcomes, Activities, and the Designed Paradise
3.1 Smith’s inference
In The Matrix (Wachowski & Wachowski, 1999), Agent Smith tells Morpheus that the first Matrix was designed as a perfect human world, free of suffering. Humans rejected it. Smith infers that human beings define their reality through suffering and misery.
The scene is instructive because Smith also reports a rival hypothesis. Some, he says, “believed we lacked the programming language to describe your perfect world.” The film thus presents both responses from Section 1 in a single speech: the designers misunderstood their subjects, or the subjects are defective. Smith chooses the second. It preserves the machines’ competence and recasts human rejection as a confession of human nature.
Whatever the fictional truth, the structure of the inference is familiar. A failed design is taken as proof about the people rather than about the design.
3.2 The experience machine
Robert Nozick’s thought experiment supplies a philosophical basis for the rival hypothesis. He imagined a machine that could give a person any experiences they desired while they floated in a tank, and asked whether one should plug in for life. His answer was that most people would not, and he offered three reasons: we want to do certain things, not merely to have the experience of doing them; we want to be a certain kind of person; and plugging in would confine us to a reality constructed by others, with no contact with anything deeper (Nozick, 1974, pp. 42–45).
A designed paradise in which outcomes arrive without activity resembles the experience machine. Someone who wants to climb a mountain is placed at the summit. Someone who wants to build a house receives the finished building. Someone who wants to become a musician receives the sensation of flawless performance. Each request is satisfied under one interpretation and defeated under another. The climber has not climbed, the builder has not built, and the musician cannot play.
Self-determination theory provides an empirical counterpart. Ryan and Deci (2000) identify autonomy, competence, and relatedness as basic psychological needs whose satisfaction supports motivation and well-being. A world that supplies outcomes while eliminating opportunities to exercise competence and direct one’s own activity may meet material wants and neglect those needs. Its inhabitants’ discontent would then be a predictable response to an incomplete design, not evidence of an appetite for misery.
3.3 Imposed hardship and chosen difficulty
This argument is easily misread as a defense of suffering. It is not.
Hunger does not make food meaningful. Preventable illness does not become valuable because recovery demands courage. Much human misery is simply misery, and its removal is an unambiguous good.
The relevant distinction is between imposed hardship, which a person neither chose nor would choose, and chosen difficulty, which a person undertakes because it is part of something they value. The mountain is difficult, but for the climber the difficulty is constitutive of the activity. Removing it would not improve the climb. It would eliminate it.
Two errors follow from ignoring this distinction. The first, the designed paradise’s error, removes chosen difficulty along with imposed hardship. The second, which is often used to defend unjust arrangements, treats imposed hardship as if it were chosen difficulty and calls deprivation character-building. A humane system avoids both. It removes hardship people do not want and leaves people to determine which difficulties are theirs.
4. Care, Authority, and the Limits of the Parental Model
Institutions often describe themselves in parental terms: they protect, provide, and know best. The model deserves both credit and scrutiny.
Consider a child who asks for chocolate. A good parent can give some, save some, explain why eating chocolate all day is not wise, and set limits without treating the desire as a defect. If the child remains unhappy, the parent can investigate. The child may be tired, may want to share with a friend, or may be asking for something they cannot yet name. Limits and understanding are compatible.
A poor parent alternates between refusal and capitulation. The parent denies the request entirely, then gives a mountain of chocolate to end the protest, then confiscates everything and concludes the child is impossible to satisfy. The child is managed rather than understood, and each failure is attributed to the child.
The parental model captures something true: care involves judgment, and judgment sometimes means refusing a request. But it fails as a general template for systems that serve adults, for three reasons.
Authority is not self-conferred. A parent’s authority over a young child rests on the child’s actual developmental limits and on social recognition of the parent’s role. An institution does not acquire parental authority over adults by claiming to understand their interests.
Good parenting aims at its own obsolescence. Its goal is an adult capable of independent judgment. A system that treats adults as permanently in need of supervision has adopted the parental posture while abandoning the parental goal.
The language of care can mask other interests. “We know what is good for you” is sometimes an expression of genuine concern. It can also justify arrangements that serve the speaker. The rhetoric alone cannot distinguish the two cases. Mechanisms of accountability can.
5. Delegation Without Dependence
5.1 One request, several purposes
Artificial intelligence makes these questions immediate. A person may soon be able to describe a goal and have an AI system perform much of the work required to achieve it: writing software, conducting research, designing a building, or establishing a business. Much of this is a real expansion of freedom. People should not be required to master every discipline before pursuing an idea, and capabilities that once required large organizations may become available to individuals.
But a single request may serve different purposes. A person who asks an AI to build a software engine may simply need a working engine, just as someone who commissions a house need not want to become an architect. Alternatively, the person may want to understand the architecture, test their own judgment, and become capable of building such systems themselves. The finished artifact serves the first purpose fully and the second hardly at all.
The same ambiguity appears across domains. A person may want wealth and also the ability to create value; companionship and also the capacity to sustain relationships; influence and also an understanding of its responsibilities. These purposes vary between people and within one person over time.
5.2 The ironies of automation
Research on industrial and aviation automation anticipated a version of this problem decades ago. Bainbridge (1983) described the “ironies of automation”: systems designed to remove the human operator leave that operator with the tasks designers could not automate, typically the rare and difficult ones, while the operator’s skills deteriorate through disuse. The person is least practised precisely when intervention is most needed.
Parasuraman and Riley (1997) distinguished the misuse of automation, meaning overreliance that degrades monitoring and judgment, from disuse and from what they called automation abuse: automating functions without regard for the consequences for human performance, which tends to define the operator’s role as a by-product of the automation.
The phrase applies well beyond the cockpit. When an assistive system defines the person’s role as whatever remains after automation, it may be efficient in the short run while quietly eroding the competence on which oversight depends.
5.3 The opposite error: compulsory edification
The correction is not to force participation. A system that turns every request into a lesson, withholds results until the user demonstrates understanding, or treats delegation as a moral failing commits the parental error from Section 4 in a different form. It substitutes the designer’s view of what is good for the person for the person’s own purposes.
Delegation is among humanity’s oldest and most productive practices. Nobody can master everything, and wanting relief from a task is not a character flaw. The principle is therefore one of choice: the person, not the system, determines which work is delegated and which participation is preserved. AI should make it easier to become capable, not merely easier to remain dependent. It should not make capability the price of help.
5.4 Four forms of retained agency
Complete self-sufficiency is neither possible nor desirable. Dependence on surgeons, engineers, farmers, and institutions is the normal condition of human life. The question is what a dependent person retains. Four capacities seem central:
- Comprehension. The person understands enough about what is being done on their behalf to make meaningful decisions about it.
- Evaluation. The person can judge whether the result serves their purpose, or can obtain an independent judgment.
- Contestation. The person can challenge a result or decision and expect a response from someone able to change it.
- Redirection. The person can change course, choose another provider, or resume the work themselves, without losing the conditions of a tolerable life.
These capacities do not require expertise. A patient need not become a surgeon to give informed consent, seek a second opinion, or pursue redress for negligence. Those mechanisms are what make dependence on medical expertise compatible with agency. AI systems, and institutions generally, can be assessed by whether they preserve analogous mechanisms.
6. The Prince Problem: Command Is Not Control
Imagine two princes with equal resources. The first can obtain almost anything by giving an order but understands little of how his orders become reality. The second knows enough to question advice, evaluate results, and recognize when those serving him pursue interests of their own.
Both can command. Only the second retains the capacities listed in Section 5.4. The first prince’s power is real but brittle. If the mechanisms on which he relies fail, change their terms, or are captured by someone else, he lacks the means to notice, let alone respond.
The parable illustrates a distinction often obscured by talk of empowerment: the freedom to request is not the freedom to direct. A system can greatly expand what people may ask for while narrowing their ability to understand or govern what they receive. Measured by requests fulfilled, it looks like liberation. Measured by retained agency, it may look like a more comfortable dependence.
The capability approach offers vocabulary for the difference. It evaluates arrangements not only by the resources or satisfactions they supply but by the substantive freedoms they secure—what people are actually able to do and to be, given their circumstances (Sen, 1999; Robeyns & Byskov, 2025). Two princes with the same resources can possess very different capabilities.
7. Making Dissatisfaction Count
7.1 Exit and voice
Albert Hirschman distinguished two responses to a declining organization: exit, in which dissatisfied members leave, and voice, in which they attempt to change it from within (Hirschman, 1970). Each carries information. Exit signals failure through lost participation, and voice signals it through articulated complaint.
Defect attribution disables both. When exit is impossible because the system controls access to necessities, departure cannot register. When voice is reclassified as ingratitude or confusion, complaint cannot register either. A system in this condition can deteriorate while every indicator it monitors reports compliance.
The practical implication is that responsiveness depends on structure, not intention. A system can sincerely value feedback while lacking any channel through which feedback compels change. The mechanisms that convert dissatisfaction into correction include:
- explanations that make consequential decisions intelligible to those affected;
- appeal processes in which someone with authority to reverse a decision actually considers it;
- viable alternatives, so that exit remains possible without catastrophic loss;
- independent scrutiny by parties who do not benefit from the system’s current form;
- representation, so that affected people influence how the system changes.
7.2 Adaptive preferences and the limits of satisfaction
An important objection runs in the opposite direction. If dissatisfaction is evidence, is satisfaction also evidence of success? Not reliably. Sen (1999) and others in the capability tradition have noted that people living under persistent deprivation may adapt their preferences to what seems attainable and report contentment with conditions that sharply limit their lives.
This complicates any purely satisfaction-based account and is one reason the argument here does not equate a good system with a satisfied population. It also cuts against defect attribution. If preferences can adapt to bad arrangements, then a population’s apparent acceptance of a system is not proof that the system fits them, and those who remain dissatisfied may be the ones registering a real problem.
7.3 Conflicting interests
Dissatisfaction is sometimes the visible result of conflicting interests. A design that fits one group may disadvantage another, and no arrangement can satisfy everyone. Defect attribution handles such conflicts poorly in a specific way: it tends to treat the complaints of less powerful groups as defects while treating the preferences of more powerful groups as the design’s natural specification.
Recognizing conflict does not resolve it. It does require that the system state whose interests a given design serves, and that those disadvantaged can see and contest that choice rather than receiving a diagnosis of their own unreasonableness.
8. A Fit Audit
The framework suggests six questions for assessing whether a system learns from the people it affects or attempts to reshape them. They are offered as a proposed audit, not as a validated instrument.
| Dimension | Question | Warning signs |
|---|---|---|
| Model fit | Does the system’s model of users match the needs, purposes, and circumstances that users report and that their behavior reveals? | Persistent workarounds; high abandonment; success defined only by internal metrics |
| Complaint routing | How is dissatisfaction classified, and who classifies it? | Complaints categorized mainly as user error, misuse, or education gaps; no category for design error |
| Participation choice | Can people choose which work to delegate and which to keep? | Uniform automation with no route to participation, or mandatory instruction with no route to delegation |
| Retained agency | Do users keep comprehension, evaluation, contestation, and redirection? | Opaque decisions; no appeal; skills that atrophy without notice; dependence that is costly to reverse |
| Exit and voice | Can dissatisfied people leave, and can they change the system from within? | Lock-in over necessities; feedback channels with no authority to compel change |
| Capability trajectory | Over time, do participants become more able or less able to direct their own lives? | Rising output alongside declining user competence, choice, or bargaining position |
The audit is comparative and longitudinal. A single snapshot cannot distinguish a system that is still learning from one that has stopped. What matters is whether evidence of misfit leads to revision.
9. What Would Weaken the Argument
A framework that cannot be wrong is an instance of the pattern it criticizes. Several findings would weaken or qualify this account.
If dissatisfaction rarely predicts design error. Suppose systematic investigation found that most persistent complaints about a class of systems reflected harmful requests or necessary constraints, and that design revisions prompted by complaints rarely improved outcomes. The claim that dissatisfaction should be treated as evidence would survive, but the practical weight given to it would decrease.
If choice over delegation produced no difference. Suppose people given control over which tasks to delegate showed no difference in competence, autonomy, or well-being compared with people receiving uniform automation. The emphasis on participation choice in Section 5 would lose much of its force.
If skill erosion did not follow from delegation. Bainbridge’s ironies describe specific operational contexts. If AI-assisted work in a given domain showed no decline in users’ ability to evaluate or resume tasks, the concern about the first prince would not apply there.
If responsive mechanisms did not change outcomes. Suppose appeal processes, explanation requirements, and representation were present but produced no measurable difference in how systems adapted. That would suggest the barrier to correction lies elsewhere, perhaps in ownership or incentives, rather than in the absence of channels.
None of these findings would rehabilitate defect attribution itself, which is an inferential error regardless of base rates. They would, however, change how much attention its remedies deserve in a given setting.
10. Conclusion: Take New Measurements
We usually evaluate systems by what they produce: goods delivered, time saved, wealth generated, suffering prevented. These measures matter, and a system that fails to meet basic needs while promising empowerment deserves scrutiny on those grounds alone.
But outputs are incomplete. The analysis here suggests three further questions. Does the system treat dissatisfaction as evidence that may require revising its model? Does it distinguish between outcomes people want delivered and activities they want to keep? Does it leave the people who depend on it able to understand, evaluate, contest, and redirect what is done on their behalf?
Abundance and agency can reinforce each other. They can also separate. A system may give people more while leaving them less say over how they live, and the discontent that follows may be the most accurate signal it receives.
The tailor need not satisfy every demand. He works under constraints of material, cost, and skill, and he may legitimately refuse requests he cannot or should not fulfil. What he owes the customer is a willingness to take new measurements, to explain his limits honestly, and to acknowledge when his work does not fit.
What he cannot do, while still claiming to be a tailor, is declare the customer’s body a mistake.
When a system repeatedly fails the people it was built to serve, the first question is not why they are so difficult to satisfy. It is whether the system ever measured them, and whether it is still willing to.
References
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8.
Hirschman, A. O. (1970). Exit, voice, and loyalty: Responses to decline in firms, organizations, and states. Harvard University Press.
Nozick, R. (1974). Anarchy, state, and utopia. Basic Books.
Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886.
Robeyns, I., & Byskov, M. F. (2025). The capability approach. The Stanford Encyclopedia of Philosophy. Substantive revision April 17, 2025. https://plato.stanford.edu/entries/capability-approach/.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68.
Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press.
Sen, A. (1999). Development as freedom. Oxford University Press.
Wachowski, L., & Wachowski, L. (Directors). (1999). The Matrix [Film]. Warner Bros.