Direct analogy of biology and software, when we talk about AI-enabled biology modeling/tweaking, breaks in at least two areas, according to Kenny Workman:

“First, biological ground truth is more difficult to define. Even apparently well-scoped tasks like clustering cells, annotating cell types or calling differentially expressed genes rarely have one canonical answer. This is an active area of research, with Benchmarks Bio to develop practical methods and tools to constrain ambiguity and model scientific intent verifiably. It remains very difficult.

Second, feedback is much weaker once we leave the analysis layer. For data analysis, the feedback loop still looks somewhat like software: run code, inspect outputs, loop. However, true feedback for higher-order scientific reasoning requires some control over the data-generating process. We will likely make a lot of progress using ground truths constructed and graded a lot like code without control over this loop. But true autonomous science requires direct feedback with the substrate being studied.”