It all depends on the shape of the distribution. An asymmetrical distribution can create big differences between the median and average even in very large samples. A good example is home prices - extremely long tail on the high end, which creates big differences between median price and average price.