Looked through the AI 2040 Plan A thing. It is bad, but I don’t think it is disingenuous. So I’m willing to give them the benefit of doubt and assume sincerity at least. I think the badness comes from this group being immersed in a subculture that has metabolized path-dependent regional subcultural success and local influence into unwarranted intellectual confidence in its ideas. With apologies to Thomas Sowell, this is “the vision of the anointed; self-congratulation as a basis for AI policy.” The result is a weird mix of a reasonable-toned and reassuring sounding document that appears to cover all the bases and address all obvious concerns and questions, and a document with a deep commitment to proselytizing a particular ideological-theological frame by exclusively and uncritically using its terms of reference. Smuggled in throughout are several deeply motivated speculations. Speculation (and risking being wrong) is of course legitimate and unavoidable in any such exercise, but the problem is the particular speculations are motivated not by the most interesting uncertainties that call for opinionated commitments, but by the validation needs of the theology. The result is a document that is uniquely and dangerously insular, unimaginative, and blind to all the uncertainties in the picture and the things nobody knows.

It reads like a mashup of a 101 history of nuclear treaties and a brochure for invitees to the Council of Trent. Rationalist hermeneutics pretending to be trend-based empiricism.

Writing this kind of document with an uncritical use of constructs like “superintelligence” and “alignment” like they’re self-evidently reasonable ontological primitives (like say “Sun” and “Moon” for astronomy) rather than deep theological commitments (closer to “Resurrection” and “Transubstantiation”) is either disingenuousness or an oblivious degree of religiosity. I think it’s the latter. These people can’t contemplate a world in which these are not in fact the right ontological primitives.

The Reformation analogy would be much stronger except the “Protestant” side (which would be e/acc here) is so deeply committed to simply being gleeful trolls, they have produced nothing resembling even the 95 theses, let alone a competing ecology of serious theologies with alternative concerns and priorities. I don’t know if that’s good or bad. We’ll avoid the bloodiness of the thirty-year war maybe. But it also means no good thinking can ultimately emerge because there’s no real dialectic here. Just uncontested theological momentum around bad metaphysics.

Of course, my own commitment is to a Spinoza-Hume type approach that discards theology altogether and begins the slow and painful longer process of separating AI church and AI state, discovering the right set of robust and conservative primitives, actually grappling with the limits of what can actually be known, and the vastness of what is either unknown or unknowable in the profoundly foggy fan of futures unfolding before us. My own top concern is imagination. How do you radically expand the philosophical inventiveness of AI discourse? How do you learn to see the emerging territories through the fog rather than through inherited texts that pretend to be maps of it? How do you develop a richer language for talking about what we’re seeing? We are in the position of Renaissance geographers, charged with building better maps of the world based on a flood of pre-theoretical data from an age of exploration, but with our project getting distorted by not-even-wrong prefigured geographic concerns like the location of “paradise.”

I’d be depressed about all this if this strain of thinking were as influential as it thinks it is and hopes to continue to be. Fortunately I think they’ve already lost the plot, but have retained a hold on the imagination of a large audience that is unaware there are other ways to think.

But I fear this crowd can probably remain loud and annoying longer than I can stay solvent.