I still do not understand why would anyone run inference on such small servers. You still want same hardware as for training but do not need that much as for batch backpropagation and for reinforcemen learning for training.

Yet blackwell still has best watt power to processing power ratio.