I love both games although i have been spending alot more time playing rock band as of late, there is so much high quality DLC available for the game compared to GH3 (although this now has the amazing "Guitar Virtuoso" pack
I prefer the Guitar mechanics on GH3 over Rock Band due to the lax nature of the HO and PO system compared to Rock Bands tight and accurate HO/PO system.
Guitar Hero is alot harder that Rock band for the Guitar though in my opinion as i didn't fail a single song on expert guitar on rock band where on Guitar Hero 3 it took me a while to pass "Raining Blood" and to beat the devil in the boss battle (and i've also recently beat "TtFatF" YAY!! lol)
I would for the time being recommend buying GH3 with the guitar then buying Rock Band separately as the GH3 guitar works on Rock Band so it will save you alot of money