That's because they are genetically independent. Think of each cell in your body as a fishbowl, with a school of identical sibling mitochondria swimming around in it. Each time a cell divides, it divides the mitochondria into two separate fishbowls. Mom gives us the egg, which is just one of those fishbowls swimming with her mitochondria. Dad contributes the sperm, which squirts a half-copy of his nuclear DNA into the egg, where it finds the nucleus and joins up with Mom's half-copy. But he doesn't contribute any mitochondria, so all our mitochondria come from our mother.