Right: sedentarism kills. A certain amount of physical exertion is certainly beneficial for physical and mental health, but it's a matter of degree and kind. 10-hour toilet cleaning shifts, day in day, day out, and all that for miserable pay, is far more likely to trigger than prevent depression. Bonus depression-inducing points if you do it alone rather than in a team. (It doesn't have to be toilet cleaning, either. Try picking fruits on a plantation day in, day out, for miserable pay and with little human contact.)

My main point, though, is that it's not idleness as such that got us where we are. As I said, many people are very busy. It's the meaninglessness of the work in question (cf. bullshit jobs), combined with social atomization.