Inssider polling the wireless device could increase latency, so it's possible it was just that.
If it's happening fair regularly, it sounds like it could be either someone getting home and starting their PC+torrent client for example, or possibly someone's using online backup which is scheduled to run at around that time every day?
Sharing an internet connection with a load of users using consumer WiFi is just asking for trouble TBH. Is there no way they'd run some CAT5 to your room or something?