As ik9000 said some designs cannot be over tightened, it'll depend on which cooler it is.
Vibration sounds are not good, it could more be the fan vibrating against the cooler, again down to which cooler your using.
@ik9000, it's very easy what you do is fit the heatsink and hold the motherboard by it like a handle.
Lower it in at an angle, get the IO lined up and in position, then lower down the other side so it's flat and you can screw it down.
Must admit the Asus Q-Shield makes fitting a motherboard so much easier, thoes are IO shields that, instead of the annoying tabs, have a foil covered foam backing it makes aligning them much simpler.