During inference, any expert may be chosen, so these have to remain in memory ready to use for when they are called upon.