Exactly. An Ayn Rand quote comes to mind.

"Do not hide behind such superficialities as whether you should or should not give a dime to a beggar. That is not the issue. The issue is whether you do or do not have the right to exist without giving him that dime. The issue is whether you must keep buying your life, dime by dime, from any beggar who might choose to approach you."