AI Doesn't Know What It's Optimizing For

The doomer and the accelerationist founder are both missing the same variable.

The Cosmos Institute and the Aspen Institute spent three days last year asking whether artificial intelligence will deliver a new Enlightenment or a new despotism. The recap, written by Ashley Kim, is genuinely good, and it ends on a small philological catch worth the price of admission.

Reflections from Aspen Institute's Socrates Seminar

Kant gave the Enlightenment its motto, sapere aude, dare to know, lifted from Horace. But Horace had written one more word that Kant left on the cutting-room floor: incipe. Begin. Kim is right that the missing word is a summons to act.

I want to press on the word she, and the seminar, and most of the people now building these systems, still leave unsaid. Begin, yes.

But toward what?

In the past month Pope Leo XIV gave his first encyclical, Magnifica Humanitas, to exactly this, and unlike most of the people now building these systems he refused to leave the end unnamed. He set the choice as one between two cities: Babel, a power that claims to dominate the heavens, or Jerusalem, a people who work together in the presence of God.

Technology, he insists, is never neutral; it takes on the character of whoever devises and finances and deploys it. He is answering the very question the seminar left open, and answering it seriously.

The trouble is naming the city you mean to build turns out to be the easy part.

Why good things fall back to earth

Here is the difficulty that secular framings of these capital-G goods cannot quite see, and that the Christian tradition has stared at for sixteen centuries. Human beings with good intentions reliably produce the very thing they swore to destroy. The revolution that rises against the tyrant becomes the terror. The movement that promises to free the worker builds the labor camp. The platform that sets out to connect the world ends up training it to despise more efficiently.

This is not bad luck, and it is not, for the most part, bad faith. It is gravity. Augustine's name for the human condition was that the self is curved in on itself, amor sui, a love that quietly installs the self as its own center of mass. Pride is the orbital velocity that guarantees the curve. You can launch any virtue you like, courage, honesty, liberty, justice, and watch it climb beautifully for a while, but absent something outside the self with enough pull to capture it, the virtue never reaches escape velocity. It arcs over and falls back to earth. The ideal burns up on re-entry, in the atmosphere of a broken world and a fallen will, and we are left standing in the debris insisting that we meant well. We did mean well. That was never the problem. The problem is that meaning well, with the self at the center of gravity, bends every good trajectory back into orbit around the one thing it most needed to escape.

Watch these three goods do exactly this. Autonomy, cut loose from any end beyond the autonomous self, does not stay autonomous; it decays into what Tocqueville saw coming, a soft despotism in which citizens, worn down by the burden of choosing, hand their choices to a gentle administrator that asks only for their passivity in return. Truth-seeking, with nothing to seek truth for, hardens into the very thing Mill feared, a dead dogma, perhaps true but held without the living friction that made it worth holding, or else it dissolves into the frictionless consensus of a feed that flatters whatever you already believed this morning. Decentralization, severed from the responsibility it was meant to serve, is merely exit, a thousand sovereign individuals optimizing away from one another until there is no commons left to decentralize. The goods are stable only in orbit around a center that is not the self. Strip out the telos and you have not preserved them. You have only postponed the hour they come down.

This is what makes the Pope's fork harder than it looks. Amor sui does not tempt you to choose Babel over Jerusalem; it lets you build Babel while you remain certain you are rebuilding Jerusalem. The curve hides inside the good intention. Every tower in history went up to general approval, raised by people who meant to support the good, divine, finally-better city—and got a monument to themselves and never once felt the trajectory bend.

The triad, re-aimed

So re-aim them. Autonomy was always pointing past itself toward relational freedom, the liberty the New Testament describes as the capacity for self-gift rather than the mere absence of constraint. "You were called to freedom," Paul writes, "only do not use your freedom as an opportunity for the flesh, but through love serve one another" (Gal 5:13). The design question for anyone building these tools, then, runs deeper than the thin one the industry likes to ask. Not ‘did the user get to choose’, but is the user being apprenticed into wise love, growing more able over time to deliberate, to commit, to sacrifice, to attend to someone other than himself. A nudge that builds that capacity is a kind of moral pedagogy. A nudge that erodes it is despotism with a friendly interface, however much choice it leaves glowing on the screen.

Truth-seeking, re-aimed, looks less like debate and more like discernment. Mill's collision of opinions is a real and precious thing, and it is also the thin secular shadow of an older practice: a community that tests the truth together, in council, with a memory, and is willing to be changed by what it finds. The first church did not convene a marketplace of ideas. It held a council at Jerusalem, argued in the presence of one another and the Spirit, and then bore the cost of its conclusion (Acts 15). Christian truth is at once personal, the Logos, a person rather than a proposition, and communal, tested in ultra-local church bodies that remember. So the thing worth building for is neither maximal engagement nor even maximal agreement. It is the slow goods Mill himself prized and could not finally ground: beliefs alive enough to move us, disagreements clarified instead of smoothed over, the discipline of stating your opponent's case before you presume to answer it.

And decentralization, re-aimed, turns out to be a discipline of stewardship rather than an ideology of exit. Hayek was right that knowledge is dispersed, that no central planner can gather into one mind what lies scattered across millions of local circumstances. The church had said something adjacent long before Hayek said it about prices: the gifts are distributed, "to each is given the manifestation of the Spirit for the common good." The point of pushing authority downward is not fragmentation for its own sake. It is to keep power proximate to responsibility, and responsibility proximate to the people it actually touches, the family, the congregation, the city. That is subsidiarity, and it stands against both the central planner and the sovereign individual who has exited every obligation he could find a door for.

A speed that can stop

The seminar staged the familiar two-sided fight about what comes next, and both sides continually, quietly, make the same mistake. The doom side, following Bostrom's vulnerable-world hypothesis, imagines that somewhere in the urn of possible inventions waits a black ball that ends us, and concludes that only something close to total surveillance can keep our hand from drawing it. The acceleration side, following Andreessen's techno-optimist manifesto, treats stagnation as the one unforgivable sin and growth as the only grace, and concludes that whatever slows the build is a form of killing. One would make an omniscient state into a savior. The other would make sheer velocity into one. Both are the orbital problem again, the self with its fears or the self with its appetites bolted to the center, and both are species of idolatry, which is only the precise theological word for orbiting the wrong sun.

The tradition's answer to both is older and stranger than either. Sabbath. Built into the calendar of creation is a commanded limit, a day that blesses precisely by interrupting, and its whole pedagogy is to puncture the lie that our production is the thing that saves us. Sabbath is not anti-work. It is anti-idolatrous work. Applied to what we are now building, it suggests a posture you might call sabbath-rhythmed acceleration: stretches of genuine, energetic building, deliberately broken by principled halts to audit the harms, to count who is paying for the gains, to ask whether the thing still serves love or has started to serve itself. A speed that can cease is a speed that can love. A speed that cannot stop is just falling, and calling the fall progress.

What this asks of the people building it

None of this stays at the altitude of theology unless it touches the code, so let me bring it down on purpose, since coming down on purpose is the entire point. Imagine what building better looks like:

• Picture an agent that, before it answers, now and then asks what you are actually aiming at, and that sometimes declines to answer in the same breath when the domain, money, a marriage, a civic act, plainly deserves a night's deliberation over a reflex.

• Picture a feed whose onboarding asks you to name your commitments, the poor, the truth from traditions not your own, one local act a week, and then ranks against them, scheduling the credible contrary you would never have clicked and the local action you keep deferring, and going dark one day in seven to hand you back to your own street.

• Picture a product review that asks, in plain words and before launch, who bears the cost of this, and is willing to put a feature back in the box when the answer is the people least able to refuse it.

None of these requires solving alignment or forecasting the singularity. All of them are simply love, given foresight and a budget.

Begin, as worship

So Kim and Horace are right. Begin. The word the church adds is not a different word so much as a deeper one: begin as worship, which is to say begin with the self knocked out of the center of the orbit and Someone else set there, with gravity enough to hold a good trajectory in place when pride would otherwise bend it home.

We do not build to seize the levers of history, which is the accelerationist's temptation, nor to hide from its risks, which is the doomer's.

We build to serve in love, to make tools that apprentice the heart toward self-gift, the mind toward living truth, and the city toward a solidarity that keeps power near the people it touches. Dare to know. Dare to love. Dare to build, at a pace that can stop, in a polity that can care, with a truth that is finally a person and can therefore change you.

That is the whole difference between digital despotism and a genuinely new light. The Enlightenment was afraid of being told what to think. Christian freedom is afraid of only one thing, the idol, the false center, the sun that is not the sun, and it is grateful for every limit that keeps it human.