well i took the fan out of mine and the panel didnt fall out, maybe you should have put the fan mount back in after dremeling off anything that would catch
i got the side panel for my original version with soundproofed side panel as i wanted more airflow after fitting a pa120.3 in the top and buggering up the airflow
trouble is i shut the side one day and a water pipe caught the fan and broke a fin.. not that it really needs the fan as the huge vent did the trick