“…we would never have gotten to where we are on aviation safety by waiting for planes to crash before we fixed problems. Safety of large complex systems comes from being proactive, following industry safety standards, and independent checks. For edge cases that means fixing edge cases when you see one for the first time rather than waiting for an embarrassing news story.”

Exactly.

In aviation we typically find that minor incidents outnumber major incidents/accidents, often by several orders of magnitude. For every Waymo that got stuck in floodwaters (and publicized), there were probably several that drove through smaller amounts of water and did not get stuck or lose control. To use the wet/dry training data example, in those cases there would be a few dots in between the wet data and point B (Flood) where the road was under a manageable amount of water (but that fact may not have been known prior to entering it).

If companies can address those situations sooner through a proactive process then they can start to address some of the most severe risks instead of waiting for the next (hopefully only) embarrassing news story. Has there been any discussion of an aviation Safety Management System-type framework for AV manufacturers/operators?