Good essay! It's also hard to figure out what the right modules to move are, I've been experimenting with evolving transformer architectures (github.com/strangeloopc…) which is, albeit rudimentary, set up to search those types of knobs and score them under objectives