There are four of you sharing the line? Is anyone hitting the cap of the download limit they imposed a few months back? If you hit more than 2gb in certain hours which I am assuming 4 of you will be they cap your line to deter you from downloading. They class it as their fair use policy due to those of us in the "high bandwidth users"
The 4mb line will cap out at about 580kb/s anyway. The 10MB lines were capping out at about 900 - 1100kb/s and the 20mb lines hit 2000+ so what's the problem with your 4mb line? Running within the limits of the line from your 400kb/s. If there are four on the line try bumping up to the higher lines
You will be very lucky to get out of this contract due to "poor speeds" as you put it as your line is running fine for a 4mb line.