If you have 1,000 subscribers and 400 of them have never opened a single email from you — you don't have 1,000 subscribers.

You have 600 subscribers and 400 dead email addresses that are actively preventing Gmail from showing your work to the 600 who want to see it.

Deleting them isn't losing your audience.

It's finding it.