I've been experimenting with 24p on my HTPC too. I'm running as Asus M3N-HDMI board with a Geforce 8300 in built. I'm also running Windows 7 which could be part of the issue, but I've yet to lock down proper 24p playback.
You can select 23hz and 24hz from the Nvidia CP, and 24hz results in bang on 24.000 playback. 23hz seems to give the same, when what we really want is 23.976. Using the custom resulation selector I can create a res of 23.976 but uit never seems to be 100% accurate, and the true refresh rate reports back as 23.974 or thereabouts.
I did a clean install of Win 7 over the weekend, grabbed the latest drivers & set them to 23hz and watched some 1080i/30 mpeg2 content deinterlaced to 23.976 and it played silky smooth. I need to experiment a bit more with avc & vc-1 encoded stuff before I can say whether it's really working.
Do ATI offer a smilar custom resolution tool? And have you tried Powerstrip to lock your card to that refresh rate?