As one of the team members who contributed to establishing the "non-proliferation" agreements in the 1970s and 80s, I agree with the higher-level objectives of this article. What's clearly missing, however, is the perspective of someone who actually knows what nuclear weapons are, how they're used, and what the drivers of the treaties really were. For example, in the section titled," How AI Differs from Nuclear Technology", it uses the criterion of similar social "potential" for both, and the threat to "human survival" they have. It then "claims" the comparison "breaks down for three reasons: AI’s far-reaching scope, its lack of excludable inputs, and its graduated strategic impact." The comparison, in fact, easily breaks down for very different, simple comparisons.
At the time the non-proliferation treaties were established, the nuclear "landscape" was very precisely defined. The U.S. had a stockpile of about 20,000 warheads. The U.S.S.R had about 70,000. We knew exactly how they were built. They knew how ours were built. We both knew generally: where both were stored, the command and control each had to launch them, and how they would be delivered. We both also knew, with high reliability, that no other country either had or was capable of rapidly building such stockpiles. This "landscape" is entirely absent from the AI universe.
The "less excludable" claim is also misleading because it is too focused on the exotic parts like "plutonium and uranium", which it views as able to be "restricted or controlled". That factor has long been lost. Right now, NINE countries have nuclear weapons. They have been sold or provided to others. Just a single use will radically change world politics. Beyond that, the creation of "dirty" nuclear bombs is within reach of any country that has nuclear power reactors. Past governments knew these issues. Each required a different type of overview and control "agency".
The third element, that nuclear weapons have "binary ... strategic" influence, overlooks the factors throughout human history that have led to the dominance of "nations" within their practical influence range. Egypt, Persia and Rome each could claim "world dominance" for their "time". But that was not based on some super weapon. It was a result of the level of "transportation technology". The "mast and sail" technology of the "tall ships" of Portugal, Spain, France and England was as impactful as their gun powder weapons.
