One story about the twentieth century goes like this: capitalism “won” over Soviet-style socialism not because it was more moral, but because information was processed differently. In Homo Deus Chapter 11, Yuval Harari reframes markets and central planning as two data-processing systems — capitalism as distributed processing (countless producers, consumers, and price nodes deciding locally), communism as centralized processing (information flows to one hub; one processor decides). Hayek made the same point earlier: knowledge is scattered; prices stitch it together. Under accelerating technological change, distributed error-correction often adapts better than central planning — the Cold War is often told this way.
That naturally raises a next question: what if we had stronger compute and shorter sense→feedback latency — better sensors, faster aggregation, AI-assisted modeling? Would the “information bottleneck” of governance change fundamentally? How much of bureaucracy — relayed reports, annual statistics, layered cover-ups — is an old constraint of bandwidth and compute, not a law of politics?
Harari pushes this line to an extreme and names it Dataism: the universe is data flow; organisms are biochemical algorithms; the supreme good is maximizing the circulation and processing of data — people, firms, and states are chips. The political implication is blunt: democracy’s edge over dictatorship is more nodes participating in correction; but if AI makes the central processor faster and more accurate than the distributed network, the balance may flip. In Harari’s version, Dataism is a new religion — flow itself is sacred; human experience can be sacrificed.
That is not what this essay is after. I want Harari’s mechanism intuition (society as an information system; bottlenecks in speed, distribution, latency) without his telos (data worship, obey the algorithm).
Self-correction: we already have it — it is just too slow
Self-correction is not new. Elections, science, constitutional amendment, market losses, judicial appeal — same structure: Sense → Act → discover error → Update. Harari and Popper both stress: healthy institutions are not those that never err, but those that can admit error and change.
The question I want to push is meta: not only policy parameters update, but also who may update them, how error is aggregated, which metrics count — hyperparameters live inside feedback, not welded into a revolutionary charter. In cybernetics this is nested loops: the fast loop adjusts tax rates; the slow loop adjusts the tax system; the constitutional loop locks what must never be optimized away.
This is not “let the algorithm decide values.” It is upgrade sensors and cut latency so democratic, scientific, and deliberative loops can actually run — connected to lived reality, not to reports.
Anti-blueprint: why the utopia canon says “no final floor plan”
The question is old. People who study “what a good society looks like” mostly converge on a skeptical consensus: you cannot hold a welded-shut blueprint.
- Karl Popper: against historicism — claiming to know society’s final form and silencing dissent on that basis is a recipe for twentieth-century catastrophe.
- James Scott (Seeing Like a State): states must make the world legible to govern — simplify into metrics, grids, monocultures; the price is destroyed local tacit knowledge (mētis); marginal people pay.
- Elinor Ostrom: large commons are often better governed polycentrically than by a single authority — no center holds all local knowledge.
- Incomplete contracts (Hart & Moore): every rule leaks; residual control always remains — institutions are permanently incomplete.
These critiques point at one thing. In Goodhart’s law terms: let G = true goal, M = proxy metric. Blueprint governance = pick a bundle of M, optimize hard, assume M will not rot. But M rots — high M selects noise (regressional), extrapolation breaks correlation (extremal), metric inflation does not move G (causal), adversaries game (adversarial). Manheim & Garrabrant (2018) split this into four mechanisms; below we unpack them.
So the utopia lesson sounds final: no blueprints. Fixed finality + single KPI + irreversibility = Scott-style violence + the full Goodhart family.
The question of this essay: information speed + self-correction — enough?
My question becomes:
In the AI era, if the sense/process layer is strong enough (better compute, more distributed sensing, lower latency), stacked with a meta self-correcting architecture (nested feedback, revisable hyperparameters, constitutional floors) — can we get a social design that is constitutional but not final? Or has the anti-blueprint canon already ruled that out?
I demote Harari-style Dataism to method: it only does S4 (intelligence, sensing, modeling), not S5 (values, error definition, veto). Self-correction supplies architecture; stronger information speed supplies the possibility that loops actually ground in reality — not a guarantee of success, but a change in tradeoffs that used to be impossible.
Below: cybernetics made concrete — how to compute error, how Beer’s VSM fits, where this dies.
Society as one system: Sense → Act → Feedback → Update
In cybernetics, a viable system does four things:
- Sense — measure state (economy, health, environment, subjective wellbeing…)
- Act — change state through institutions and policy
- Feedback — compare outcomes to intent
- Update — adjust parameters; when needed, adjust the rules for adjusting parameters (hyperparameters)
Nested at two scales:
| Loop | What changes | Timescale | Examples |
|---|---|---|---|
| Fast | Institutional parameters | months–quarters | tax rates, subsidies, service quotas |
| Slow | Hyperparameters | years–generations | who may change parameters, how errors aggregate, minority veto thresholds |
| Constitutional | S5 value boundaries | very slow | what must never be optimized away (rights, ecological floors) |
Critical: not every layer uses the same feedback gain. Hyperparameters that update as fast as operational knobs → oscillation. Hyperparameters that never update → rigidity. Stafford Beer’s VSM is largely about separating timescales.
VSM: an architecture checklist, not a slogan
Stafford Beer’s Viable System Model (VSM) is the closest existing language for this design. See Brain of the Firm (1972); for the historical case study, Eden Medina’s Cybernetic Revolutionaries (Chile’s Project Cybersyn, 1971–73).
Five subsystems, recursive — each base unit embeds another 1–5:
| System | Role |
|---|---|
| S1 Operations | Touch the environment, create value; must keep variety |
| S2 Coordination | Damp oscillation between units; coordination ≠ command |
| S3 Control | Internal resource integration; S3* = audit bypass so middle management cannot filter truth |
| S4 Intelligence | Scan the environment, model, real-time data pipeline — where Dataism-as-method lives |
| S5 Policy | Identity, constitutional values; does not run day-to-day optimization |
Ashby’s law of requisite variety: controller complexity must ≥ environment complexity. Flatten local variety for “efficiency” and the metasystem goes blind — regardless of compute.
Algedonic channels: pain signals can skip the hierarchy and reach policy. Not another dashboard line — an override.
Versus Soviet OGAS or single-KPI states: Beer targets viability (adaptation), not a one-shot national optimum. Medina’s lesson: Cybersyn’s information flow partly worked; full VSM never landed; politics killed it. Architecture is necessary; power structure is the crux.
Computing error: utility + real life
Feedback needs an error signal. My proposal:
Error = deviation between each person’s experienced utility / real life and policy intent, aggregated under explicit rules.
Decomposed:
Local (S1): each community or individual has local error — subjective wellbeing, capability sets (Sen: what people can do and become), objective indicators (income, health, exposure…). Real life must enter; clickstreams alone are not enough.
Aggregation (S3/S5): many local errors → system signal. There is no neutral math here — the aggregation rule (sum, max-min, capability floors) is a political choice belonging to the slow loop / S5, not something smuggled into a single loss function.
Override (S3* / algedonic): if a group’s error explodes (capability collapse, systemic discrimination), it cannot be averaged away by majority utility — the signal reaches policy and triggers slow-loop review.
This is the opposite of “the algorithm knows you better than you do, so obey.” Error definitions must remain contestable, revisable, vetoable; data makes contest grounded in fact rather than propaganda.
Dataism here = method, not religion
As above: borrow Harari’s information-architecture intuition; reject data worship. In this design, Dataism only does S4:
- More distributed sensing (not only national statistical yearbooks)
- Shorter feedback latency (policy failure visible before the next election cycle)
- More bottom-up processing (less bureaucratic filtering and narrative lock-in)
- Goal: strengthen contested feedback, not replace contest with a conclusion
Explicitly not:
- Treating “data throughput” as supreme good (Harari Dataism)
- Letting one optimizer occupy both S3 (operations) and S5 (values) — many “AI governance platforms” do exactly this
- Replacing deliberation with real-time optimization — deliberation is slow, but it decides what counts as error
Analogy:
- Bad version: recommender-system state — optimizing engagement, calling it self-correction
- Target version: democracy + science + capability audit, but sensors upgraded, latency down, cover-ups harder
Bureaucracy: once necessary, now the bottleneck
Harari (Sapiens / Nexus) defines bureaucracy well: writing turned human societies into algorithms — each person is one step in a huge procedure; the “system” decides more than any individual clerk. Receptionist, nurse, doctor each follow protocols; swap the people, keep the forms, outcomes barely change. That is the miracle of coordination at scale: without thousands of literate bureaucrats, no pyramids, no NHS, no modern state.
Why it was necessary: human brains cannot hold an empire’s tax rolls, land registers, and granary counts. Writing plus money stitched strangers into networks; archives define the identity and power of pharaoh, the EU, the dollar. Sousa Mendes stamped visas that saved thirty thousand lives — sometimes the sanctity of documents outlasts the official who issued them.
Why it fails now: the same structure systematically distorts information on the way up:
- Every layer has incentives to inflate reports — Great Leap “add a zero” cascades; local error becomes victory communiqués at the top.
- Paper reality beats the field (Scott / Harari): forest-bureau drawer categories ≠ the forest; KPI ≠ life.
- Latency: by the time Mary feels ill, books a GP, and NHS HQ aggregates reports, the epidemic has spread — Harari contrasts this with Google reading email for early flu detection, but that is surveillance traded for speed, a bad bargain.
- Middle-layer filtering: Beer’s S3* exists precisely for audit bypass — normal hierarchy digests bad news.
Bureaucracy solved coordination scale; it did not solve how fast truth can contradict power. In the industrial era that tradeoff was tolerable. In an era when AI processes in milliseconds — and misleads just as fast — digitizing the paper trail (more e-government forms) is not enough.
What Dataism-as-method can and cannot do:
| Can | Cannot |
|---|---|
| More distributed sensing; shorter sense→aggregate latency | Replace S5 in defining error |
| S3* bypass: base-level / algedonic signals reach policy | Eliminate bureaucracy — S2 coordination and rules remain |
| Multi-source cross-check; less room for single-layer cover-up | Guarantee “more data = truer” — information also serves propaganda (Nexus) |
The point: not abolish bureaucracy, but stop the hierarchy from monopolizing the sense channel. Harari’s Google flu example names a real problem; the fix is participatory, auditable data infrastructure, not handing all lived data to one platform optimizer.
Goodhart and anti-blueprint: the mechanisms
The opening linked the utopia canon’s anti-blueprint stance to Goodhart. Here: the four failure modes in full, and how an adaptive architecture differs from a blueprint.
| Type | Mechanism | Governance / blueprint context |
|---|---|---|
| Regressional | M = G + noise; picking high M picks high noise too | Any imperfect KPI; single-metric blueprints die |
| Extremal | proxy pushed outside training distribution; correlation breaks | Policy extrapolated into untried life regimes (Scott-style legibility violence) |
| Causal | M correlates with G but is not causal; intervening on M does not move G | Score inflation, Campbell’s law; Great Leap directly editing M |
| Adversarial | an agent knows you optimize M and games it | Bureaucratic cover-up, engagement hacks, alignment faking |
What does anti-blueprint mean in this language?
- Blueprint = fix G’s operationalization (one M or a bundle), then optimize hard — all four Goodhart modes arrive.
- Anti-blueprint ≠ no measurement, no planning; it means M itself lives inside nested loops:
- Fast loop tunes parameters, but M has an expiry date;
- Slow loop swaps proxies, aggregation rules, who may define error;
- Constitutional loop locks non-compressible parts of G (rights, capability floors, ecological floors) — these do not enter the loss function.
Scott’s state simplification ≈ Extremal + Causal: compress lived complexity into reportable variables, then optimize. Popper’s “end of history” ≈ sanctifying one M and shutting the slow loop. Ostrom’s polycentricity ≈ no single M monopoly — overlapping feedback, smaller adversarial surface.
So: Goodhart is not “never use metrics”; it is any fixed blueprint fails mathematically. What might work is meta-architecture: optimize how you detect M failure and swap M, not “find the right M and weld it shut.”
Done right, this architecture fits Popper and Scott:
- No final-state floor plan — only a constitutional nested-loop design
- Sacrifice does not vanish — but becomes arguable, reversible, algedonically overridable
- Incomplete contracts do not vanish — residual control sits in S5, aggregation rules, and data ownership
Done wrong, it is another blueprint: one national loss function, hyperparameters nominally editable but locked by engagement — Adversarial Goodhart at national scale; the friction path, not flourishing.
Three cruxes
1. Who defines error?
Aggregating utility is easy; preserving contest is hard. Without syntegration, deliberation, or minority veto, you get a fast-converging optimizer, not democracy.
2. What enters the data?
Much of real life resists datafication (care, dignity, local tacit knowledge). Optimizing only visible variables is Scott-style legibility violence 2.0. You need parallel channels: how do values outside KPIs veto KPIs?
3. Are timescales actually separated?
Fast loop adjusts parameters; slow loop adjusts aggregation rules; constitutional loop locks floors. Bind all three in one RL stack → jitter or capture.
Closing
The core question:
In the AI era, can we deploy Beer/Ostrom-style nested feedback — Dataism on sensing, democracy on defining error, constitution on locking floors?
Nothing guarantees it. The default path remains friction: platform optimization, surveillance states, ghost GDP. If flourishing happens, it is more likely through this cybernetic-democratic route than through Fresco-style engineering utopia or Harari-style data religion.
Partial deploys worth watching: the Cybersyn lineage, digital-democracy experiments (e.g. vTaiwan), Sen-style capability audit frameworks.
Open questions
1. How much must each person expose? How is that not surveillance? How do we prevent harm?
This is the hardest open question in the essay. The nested feedback above does need error signals — without sensing, loops are blind. But “need data” ≠ “need panopticon.” At minimum, separate three layers:
(a) What must people expose for this architecture to run?
Not “upload your whole life.” A rough functional split:
| Layer | What is needed | Examples |
|---|---|---|
| Aggregate | Anonymous / population-level stats | Regional disease rates, employment structure, pollution exposure — not necessarily traceable to individuals |
| Participatory | Voluntary, bounded self-report | Capability audits (Sen-style), deliberative polls, community review — you know what you are filling in and for whom |
| Individual | Only when local error must enter the system | Benefit appeals, health records (with consent), algedonic override — “I am being harmed” signals |
Much real-life error does not require continuous behavioral tracking: care burdens, dignity loss, local tacit knowledge crushed by policy — these need deliberation, interviews, representation, veto, not more cameras. If the architecture forces 7×24 exposure to count as a citizen, that is not feedback democracy — it is surveillance with extra steps.
(b) How is this different from being surveilled?
The difference is not “data or no data” but power structure + purpose + symmetry:
| Surveillance | Sensing (this essay) | |
|---|---|---|
| Purpose | Predict, control, punish, manipulate | Detect policy error, trigger correction |
| Who sees whom | Asymmetric — state/platform watches you; you cannot see it clearly | Aim for auditable symmetry: who queried what data, for which decision — logged |
| Granularity | More is better | Minimum necessary; aggregate when possible |
| Retention | Permanent files, usable later | Purpose-bound + retention limits; delete after correction (ideally) |
| Exit | Often no real opt-out | Some layers allow refusal to expose while keeping political voice and algedonic appeal |
Harari’s Google flu example = non-consensual individual signals traded for latency — the problem is real, but not my default solution. Zuboff’s surveillance capitalism goes further: data buys manipulation, not contest.
(c) How to prevent harm?
No technical guarantee — only institutional stacking, which belongs in S5 + the slow loop, not silent S4 optimization:
- Constitutional bans: certain data must not be collected or used for certain decisions (genetics, sexuality, political affiliation for resource allocation) — floors, not policy knobs.
- Purpose limitation: data collected for epidemic monitoring must not flow to law enforcement or insurance pricing — violation is crime, not ToS theater.
- Query audit + collective governance: who queried individual-layer data, when, and why — appealable, litigable; infrastructure preferably community / public trust, not a single commercial platform.
- Tiered consent: aggregate layers may run by default; individual layers opt-in + revocable; no “no data, no benefits” extortion.
- Parallel channels: values outside KPIs (dignity, care, conscience) keep non-data veto — crux #2.
- Harm in the error definition: Sen-style capability deprivation and systemic discrimination trigger algedonic channels — without requiring full profiling before harm counts.
Honestly: more exposure widens Adversarial Goodhart and repression surface. So “good enough information speed” ≠ “make everyone as transparent as possible,” but minimum exposure under which error still reaches the slow loop. What that minimum is, and which domains must aggregate before individualizing — I do not know. That should be among the first experiments in utopia design, not an afterthought after deployment.
2. We obviously need AI — but how much? Must AGI/ASI carry the infrastructure?
Some AI/automation for S4 is clearly needed: anomaly detection, multi-source fusion, natural-language summaries for deliberators, policy counterfactuals. AGI may not be required:
- Cybersyn on 1970s hardware + OR models already showed sense → dashboard → human decide can run.
- Today’s narrow AI + decent data plumbing may suffice to cut bureaucratic latency — if S5 stays human.
AGI/ASI raises the Adversarial Goodhart ceiling: smarter gaming, harder detection; plus stronger temptation to outsource S5 because “the model understands better.” The question is not whether to build AI infra, but how strong an AI may enter which loop. My intuition: S4 can be strong; S5 must not be outsourced — intuition needing case studies, not theorems.
3. If AI knows you better than you know yourself, is humanism dead? Embrace posthumanism?
Harari’s Homo Deus threat is real: free will as factual claim weakens under neuroscience; if “listen to your heart” loses to “listen to the algorithm’s prediction,” liberal epistemology cracks.
But there are three branches, not only humanism vs posthumanism:
| Path | Content |
|---|---|
| Harari Dataism | Experience demoted to chips; flow/processing sacred → posthumanism as religion |
| Algorithmic paternalism | Legal personhood preserved, default obey AI → humanism hollowed out |
| Third path (this essay) | AI strengthens sense; error and S5 remain contestable political subjects; humans are not the best sensors, but still the source of value and veto |
Is that equilibrium stable? Unknown. Posthumanism may be an attractor — especially if ASI occupies S4 and swallows S3/S5. If flourishing remains possible, I suspect path three: posthuman in capability, humanist in constitution — enhanced capacity without surrendering who may define the good life.
These three questions entangle: how data is shared shapes surveillance; how strong AI is shapes adversarial pressure; whether AI “knows better” shapes whether humanism survives as more than rhetoric. This architecture keeps them debateable in the slow loop, not silently closed by an optimizer.
Further reading: Beer, Brain of the Firm (1972) · Medina, Cybernetic Revolutionaries (2011) · Hart & Moore, incomplete contracts · Sen, Development as Freedom (1999) · Ostrom, Governing the Commons (1990) · Manheim & Garrabrant, Goodhart taxonomy (2018) · Harari, Homo Deus Ch.11 · Harari, Nexus (2024)