Hermeneutic Uncertainty Anyone?

The idea that we can just put error bars on the outputs of reasoning models, agentic workflows etc.. based on earlier eras of statistical thinking is breaking down. The data generating mechanisms underlying AI outputs and downstream decision making process are massively non-i.i.id and perpetually operating in a loop. I don’t know if “hermeneutic uncertainty” covers the new ground entirely, nor does performative predictions literature [1].

The top machine learning / statistical learning researchers who think about rigorously drawing inferences about the world have a CFP for new ways of statistical thinking in the agentic era.

High-dimensional models operating on massive non-i.i.d. datasets strain traditional techniques, generative objectives prioritize plausibility over correctness, and interactive and multi-agent systems compound uncertainty across sequential dialogues and multiple agents. Yet, the challenge is not solely mathematical. As AI integrates into high-stakes professional domains such as healthcare, law, and education, experts face forms of uncertainty that resist reduction to percentages. A legal professional assessing subtle aspects of human actions, or a teacher evaluating the cultural meaning of a literary interpretation, is navigating hermeneutic uncertainty, where the challenge is contested interpretation. When AI systems force these nuanced judgments into probability scores, they risk atrophying the very professional expertise they are meant to augment.

• New Taxonomies of Uncertainty: Developing new definitions of uncertainty adapted to the landscape of modern AI. These include definitions that move beyond the standard statistical distinctions to encompass ethical, cultural, and contextual uncertainties central to professional judgment.

• Epistemic vs. Hermeneutic Uncertainty: Distinguishing between "what we don't know" (epistemic) and "what is open to interpretation" (hermeneutic). How can AI systems signal the latter without falsely quantifying it?

• Methodological Innovations: Novel methods for quantifying uncertainty in generative models trained on near-universal datasets, including metrics for semantic uncertainty, and frameworks for tracking reliability across interactive and multi-agent systems.

• Visualization & Uncertainty Communication: Moving beyond standard confidence intervals through innovations in visual analytics. We seek designs that help users navigate high-dimensional spaces, signal not only statistical uncertainty but also when outputs are technically sound yet open to interpretation, and link uncertainty to downstream decisions.

• Professional Practice & Sense-Making: Participatory frameworks and empirical studies evaluating how uncertainty communication impacts the judgment, accuracy, and agency of human experts in collaborative workflows.

[1]. Performative predictions paperpile.com/shared/sn…