Heh, I think I literally had a penny just drop on me. It's all down to the switches you use in ESXi, right?
For the Management, Storage and vMotion networks you connect them to the ESXi hosts as VMkernel ports and for the VM network you connect it as a port group. This way the hosts don't get IP's but provide the network to the VM's. Right?