Cost and the possibility to upgrade in future are about the only major considerations really.
In some cases, memory controllers may struggle to hit higher clock speeds with all slots populated due to the additional loading, but that's generally only a concern if you're using very high speed memory and/or overclocking.