They are both among the top cards on the market, but I'd go for the X800GT, since it is a lot better suited toward higher end gaming needs. And, in regards to the PC as a whole, you'll probably find the following threads very helpful:
Building A Gaming Rig - A Living Document. (A HEXUS Project)
A great resource that explains all of the basics for those new to system building and to recap key points for the experienced. Has some recommendations on various components, but do be wary of the last update times on the posts, some of the recommendations may be outdated.
Draft Specs of Gaming / Multimedia PC [ Feedback Required ]
A thread created by myself that contains loads of advice and suggestions based upon the design of my high spec PC. I originally had a budget of £500 (excluding the case, monitor, PSU, hard drive, cooling system, speakers and so on; pretty much the same situation as yourself), but stretched it over a bit more to get a machine that'll last for a long time and deliver for my needs. Loads of useful stuff posted in here.
Cheapo gaming rig? Ideas?
It's an older thread, but it has been updated with more recent and relevant information.
PCI Express Graphics Card Review *external link
A recent roundup of the best PCI-E cards. A good start in terms of comparisons. You can ignore the SLI results on the benchmarks, since those are two cards running simultaneously and is beyond the budget you have set.
Hope that all helps you out there.