I’ve been feeling torn for a while. In Silicon Valley, I talk to all kinds of people about AI every day. I read papers predicting AGI by 2027. But outside of the Bay Area and a few Chinese cities, most of the world hasn’t really been touched by AI yet.
The implications of AI are enormous. Some say it will bring either utopia or dystopia. But our preparation has completely fallen behind — not just technically, more so institutionally. E.O. Wilson once said: “The real problem of humanity is: we have Paleolithic emotions, medieval institutions, and godlike technology.” AI is making that mismatch more dangerous.
Everyone in the Bay Area is talking about AI, but I think most people aren’t thinking about the broader consequences. They’re focused on what’s within their own interest — investments, layoffs, job opportunities. Very few are seriously thinking about how society should respond. That disconnect bothers me.
This article is my attempt, based on what I’ve found so far, to describe a path that might actually work. Not a prediction — a path. Step by step, what each step requires, how hard it is, and what questions we still don’t know how to answer.
Phase 1: Safety before capability
Goal: Investment in alignment should exceed investment in capability. Not “develop the economy first, then do safety” — safety comes first from the beginning.
Like raising a child. You don’t let them get strong first and then teach them right from wrong. You do both from the start, and while they’re still small, teaching right from wrong matters more than building strength. AI should be the same — while it’s still not too powerful, alignment work should outpace capability work.
What needs to happen:
Safety research is severely lagging. Anthropic estimates spending 30-40% of development cycles on alignment. Most companies spend far less. The Bengio report explicitly identifies a “mismatch” between capability development and governance development. The “alignment tax” — the extra cost of making AI safe — is estimated at $8-15M additional compute per major model release. This needs to become an industry norm, not a competitive disadvantage.
Some will say: you can only invest more in safety after you develop the economy first. I disagree. That’s like saying “let the child grow however they want, teach them morals once they’re strong.” By then you can’t teach them anymore.
Current state of the art in alignment research: mechanistic interpretability (understanding what’s happening inside models), scalable oversight (monitoring AI systems smarter than their monitors), constitutional AI (AI self-constraining against principles). None is close to “solved.” But these are the most promising directions.
Bostrom distinguished two approaches: capability control (boxing it in) and motivation selection (making AI want the right things). Capability control is theoretically impossible long-term. Motivation selection is the real solution, but also the harder one. Stuart Russell’s Human Compatible proposes AI should remain uncertain about human preferences — an AI that knows exactly what you want has no reason to let you correct it.
How hard is this step: Very. Alignment is an unsolved research problem. Difficulty scales with AI capability. But we have no choice — if alignment isn’t right, none of the following steps matter.
Phase 2: Build regulation and international coordination with real power
Goal: Make the regulatory system’s authority greater than any individual company’s. Make international coordination mechanisms capable of actually constraining the AI race.
This step is the prerequisite for everything else. Without regulation, alignment investment won’t become an industry norm (companies won’t voluntarily accept competitive disadvantage). Without international coordination, economic transition won’t happen (countries race to not be the first to tax). Without institutional push, HCI design won’t change (market incentives push toward full automation). Safety, economics, human wellbeing — each depends on someone setting and enforcing rules at a higher level.
This will not happen on its own.
Individuals and companies are naturally short-sighted — that’s not criticism, it’s human nature. Throughout history it has always been this way. You can’t expect every CEO to voluntarily choose safety over profit. Silicon Valley CEOs tell politicians AI will create jobs, boost productivity, keep America ahead — these may all be true, but it’s not the whole truth. Nobody goes to Capitol Hill to say “my product might put 30 million people out of work, please regulate me.”
What needs to happen domestically:
In the US, Silicon Valley CEO influence on politicians is too strong — through technology influencing politics, through money influencing politics. Most politicians don’t understand the technology. This information gap keeps regulation perpetually behind.
What’s specifically needed:
- An AI regulatory agency independent from industry — not letting the regulated set their own rules (Basel II let banks assess their own risk → 2008 financial crisis)
- Technically literate regulators — not 64-year-old senators asking Zuckerberg “how do you make money”
- Transparent lobbying disclosure — letting the public know which AI policies were bought
- An FDA-like approval process for AI deployment — at least for high-risk applications (medical, legal, financial)
What needs to happen internationally:
Three incompatible frameworks exist now: EU strong regulation, US hands-off development, China government approval + content control. But the two countries that actually determine AI’s trajectory are the US and China — they control roughly 90% of global AI compute. Every other country’s regulatory choices have limited effect when these two aren’t coordinating.
The Biden-Xi nuclear AI agreement — humans, not AI, should control nuclear weapons — is the only substantive bilateral AI safety agreement. Extremely narrow scope. We need to expand this dialogue from “nuclear weapons” to “AI development speed,” “safety standards,” “economic impact response.”
On moratorium:
Many say a moratorium is infeasible — whoever pauses falls behind in the race. This objection proves exactly why we need stronger international coordination. Not unilateral pause, but coordinated slowdown or safety thresholds.
The Nuclear Non-Proliferation Treaty achieved something similar: the US and USSR were mortal enemies during the Cold War, but both agreed not to proliferate nuclear weapons, not to test in the atmosphere, and eventually limited warhead counts. Imperfect — but 80 years without nuclear catastrophe. If Cold War adversaries could coordinate on nuclear weapons, why can’t competing US and China coordinate on AI safety?
A moratorium doesn’t have to mean “full stop.” It could be:
- International approval for training above specific capability thresholds (like nuclear enrichment requiring IAEA approval)
- Bilateral agreement not to deploy specific dangerous applications (like the Biological Weapons Convention)
- Shared safety benchmarks — models must pass specific safety evaluations before release (like pre-market clinical trials for drugs)
- Compute monitoring — GovAI research shows compute is the governance lever best suited (detectable, excludable, quantifiable)
Why social push is the most important thing:
Everything above — independent regulation, international coordination, moratorium — none of it will happen automatically. Major regulatory changes throughout history have almost always required social push:
- The environmental movement (1960s-70s) pushed the creation of the EPA and Clean Air Act
- Consumer rights movements pushed FDA food safety standards
- Anti-nuclear movements pushed partial test bans
- Privacy movements pushed GDPR
What AI governance currently lacks isn’t good policy proposals — there are many. What’s missing is enough social pressure to make politicians feel “not doing this will cost me votes.” AI governance is currently an elite conversation — safety researchers, policy scholars, a few politicians. The general public either doesn’t understand or is confused by extreme narratives oscillating between hype and fear.
In fall 2025, over 400 AI scientists and Nobel laureates signed an open letter calling for a global AI treaty, demanding “operational red lines with robust enforcement mechanisms” by end of 2026. That’s a beginning, but far from enough.
A deeper question: will the form of government itself change?
There’s a theoretical framework worth taking seriously. Harari’s “Dataism” in Homo Deus reframes political systems as information processing systems: capitalism is distributed data processing, communism is centralized. Capitalism won not because it was more ethical, but because distributed processing is more efficient. Similarly, democracy’s advantage over dictatorship lies in distributed information processing — more nodes participating in decisions, processing more information.
Jay Galbraith’s organizational information processing theory says something similar from management science: an organization’s structure depends on how much uncertainty it needs to process. Low uncertainty → rules and hierarchy suffice. High uncertainty → flatter structures, more lateral communication, stronger information systems.
If this logic holds, governance structures in the AI age need to evolve too. When information flow speed goes from “days by messenger” to “milliseconds globally” to “AI processes autonomously,” government structures that meet monthly and legislate annually may simply not keep up. This isn’t saying democracy is bad — it’s saying democracy’s specific implementation (representative government, four-year election cycles, committee deliberation) may need to adapt to new information processing speeds.
What exactly would this look like? Maybe real-time public feedback systems. Maybe AI-assisted policy simulation (like the USV economic model but at larger scale). Maybe more frequent, issue-based civic participation replacing once-every-four-years elections. I’m not sure. But I am sure that governing 21st century technology with 18th century institutional designs needs at least an upgrade.
Open questions: How do we move AI governance from elite conversation to public concern? How do we prevent regulatory capture? How do we push AI safety coordination amid US-China tensions? How must the organizational form of government itself evolve to keep pace with AI?
Phase 3: Economic transition — ensuring the gains are shared
Goal: AI-created wealth flows to everyone, not just a few. Prevent inequality from widening further.
What we know:
AI’s real economic advantage — after verification and error costs — is roughly 1-5x, not the 100,000x headlines suggest. But even 1-5x, rolled out across enough industries, still means massive work restructuring and some jobs disappearing.
The USV model proves: utopia and dystopia are both equilibria of the same system. The difference lies in only two policy variables: competition and redistribution. Neither alone works.
What needs to happen (in stages):
Short-term: Correct the tax code’s subsidy of automation. US labor tax rate ~25%, capital tax rate ~5% — the tax system itself encourages replacing people with machines. Correcting this imbalance doesn’t require inventing a new tax, just removing an existing unreasonable subsidy.
Medium-term: Tax AI economic activity. Take from those who benefit from AI and redistribute to those who are harmed — retraining, transition support, social safety nets. Options include token taxes, electricity taxes, equity taxes.
Long-term: If AI truly pushes most production costs toward near-zero, UBI or UBS may become necessary. Seven UBI experiments consistently show people don’t stop working when given money. But by year three, mental health improvements fade — money is necessary but not sufficient.
“Politics won’t cooperate” is not an endpoint — it’s a problem to solve:
The economic transition is most likely to fail not because the economics is wrong, but because politics won’t cooperate. OECD BEPS took 15 years for half an agreement. Prediction markets give only 37% probability of any country implementing a robot tax by end of 2027.
But “politics won’t cooperate” is not an immutable fact — it’s a problem that needs solving. Every major economic redistribution in history required social push:
- The eight-hour workday took 100 years of labor movement (Robert Owen 1817 to Fair Labor Standards Act 1940)
- Social security was established only after the Great Depression
- Minimum wage laws required decades of labor organizing and several economic crises
Political push for AI economic transition may require:
- A sufficiently large AI-caused economic crisis (like 2008 driving financial regulatory reform) — Chatham House argues binding governance may only be politically feasible after a crisis
- Enough Klarna-style failures to make industry realize pure cost optimization is unsustainable
- Public experience of AI job displacement reaching a critical mass (60% of advanced economy workers already worry about AI affecting their jobs)
- Politicians discovering “supporting AI redistribution” wins votes (when enough affected voters exist)
Open question: Can we prepare redistribution mechanisms before a crisis? Or are humans destined to act only after pain? If the latter, at least design the policy frameworks in advance so they can deploy rapidly when crisis arrives — what Chatham House calls “off-the-shelf” governance frameworks.
Phase 4: Designing AI interaction — not just for efficiency
Goal: Make AI improve productivity without making people worse at their jobs or less happy.
What we know:
I discussed this in detail in a separate article. The core finding: efficiency and happiness aren’t inherently contradictory, but they conflict under specific conditions. The key variable is interaction design — the same AI in passive mode (AI generates → human reviews) damages self-efficacy and meaning, while in active mode (human drafts first → AI refines) both are preserved.
I admit you can’t always use the optimal mode. Sometimes a deadline demands full AI output. But I think we’re currently too biased toward short-term. Not choosing the fastest option every time, but choosing the one that preserves human capability most of the time.
What needs to happen:
AI tools should have four modes: Generate (AI does it), Scaffold (AI hints, human does it), Challenge (AI challenges human reasoning), Step Back (AI does nothing, human practices). Which mode depends on task, skill level, and time pressure. The Wharton chess study shows system-regulated AI assistance outperforms user-selected by 2x (64% vs 30%) — users can’t self-regulate, the system must help them.
This isn’t just a UX issue — it directly affects economic outcomes. Verification cost is the bottleneck compressing AI’s 53,000x raw advantage to 1-5x. When humans make judgments and AI handles mechanics, verification is cheaper. The design that preserves human capability and the design that maximizes economic return are the same design.
Open question: Market incentives push toward full automation (lower cost per task). How do you make designs that preserve human participation commercially competitive? Tax incentives? Regulatory requirements? Or wait for enough Klarna-style failures to wake the industry up?
Phase 5: If everything goes right — what does utopia look like?
The first four phases are “how to get to a good outcome.” This section is the harder question: what does a good outcome actually look like?
UBI is a tool, not the destination
UBI is a distribution mechanism, but it doesn’t answer “what do people do with their time and freedom.” As we previously researched — Jahoda’s five latent functions (time structure, social contact, collective purpose, status, regular activity) aren’t things money can provide. 30-40% of FIRE community members go back to work after achieving financial independence — not for money, but for meaning.
Dario Amodei’s “Machines of Loving Grace” paints a maximally optimistic version: AI defeating most diseases within 5-10 years, lifting billions from poverty, democratic renaissance. But even if all that happens, what does each person do when they wake up? That’s not a technology problem.
What do people actually want?
Everyone wants different things. Some want to create, some want to rest, some want to explore, some want to be with family. Utopia shouldn’t mean “making everyone ecstatic” — because as we’ve researched, happiness itself is counterintuitive. Dopamine is “wanting,” not “liking.” Permanent satisfaction may not exist at all.
Should we let AI understand what each person wants and help them achieve it? That sounds right — but who defines “help”? What if what someone wants harms others? What if someone’s preferences are themselves manipulated?
The deeper question: truth vs happiness
Most people want to be happy and live without pressure. A few want to investigate the world’s truths — even when the process is painful, even when the discoveries are disturbing, even when pursuing truth might bring risk to society.
There is real tension between these goals. If most people vote “let’s stop AI research, we’re fine now,” while a handful of scientists believe continued research could unlock more of the universe’s secrets — who should be heard?
I don’t have an answer. But I think honestly surfacing this tension is more important than pretending it doesn’t exist.
The ethics of brain modification
An interesting question: returning to E.O. Wilson’s quote — Paleolithic emotions, medieval institutions, godlike technology. If our institutions catch up, what about our emotions, values, cognitive capacity?
If AI becomes advanced enough, directly modifying the human brain becomes possible — making people more rational, more cooperative, less selfish. This sounds like “problem solved,” but it raises a terrifying question: who decides what the “correct” brain should look like?
Modify brains for more happiness → how is that different from plugging everyone into an experience machine? Modify brains for more rationality → who defines “rational”? Every historical attempt to reshape human nature led to disaster. Modify brains for more cooperation → is that still free will?
My position: this area needs stronger regulation and deeper ethical discussion than any other AI application. Until those discussions are resolved, we should proceed with extreme caution.
Why human-centered?
There’s a critique — speciesism — that asks: why should humans be the center? Why not let intelligence itself develop without limit? If humanity is just one stage of evolution, isn’t insisting on human-centeredness a form of selfishness?
I understand the force of this argument. “Truth is held by the few” — perhaps pursuing higher intelligence really is the universe’s “purpose,” if the universe has one.
But I can’t find a strong enough reason to abandon human-centeredness. The development of intelligence sounds grand, but if that process makes most people suffer, collapses society, and benefits only a few (or machines) — I don’t think that’s a path worth taking. I’d rather believe: if a path is good, it should be good for most of the participants, not just the smartest ones.
This isn’t a proof. It’s a position. I know some will disagree.
What we don’t know
Honestly listed:
- Can alignment be solved before AI becomes dangerous? — Don’t know.
- Can international coordination happen without a crisis? — History says probably not.
- Where does the political will for economic transition come from? — Don’t know. Currently nonexistent.
- Are efficiency and happiness truly compatible long-term? — Evidence suggests conditionally yes, but no longitudinal study exceeds one year.
- If AI becomes much smarter than humans, does the “human-centered” position still hold? — This is a philosophical question, not a scientific one.
- What does utopia actually look like? — Nobody knows. There may not be a single answer.
- Should population keep growing? Should humans explore space? Should we use technology to modify human brains? — These questions existed before AI. AI makes them more urgent.
Listing these isn’t pessimism. It’s because pretending we know the answers is more dangerous than admitting we don’t.
The meaning of this path
This isn’t a “if we do it right everything will be fine” story. Every step could fail. Politics may not cooperate. Technology may move too fast. Human nature may not allow us to cooperate enough.
But if we don’t articulate the path, we won’t even know when we’ve deviated. Every step has measurable indicators: is safety spending growing as a share of total AI investment? Has any country started taxing AI? Are AI tools beginning to default to scaffold mode? Is alignment research making verifiable progress?
The happy path with AI is not the default path. It’s the one that asks the most of us — more foresight, more coordination, more willingness to say no to short-term advantage.
Whether we take it is not a question about AI. It is a question about us.
Sources