Bruce Bartlett argues against the Fair Tax here: http://taxprof.typepad.com/taxprof_blog/files/bartlett_fair_tax.pdf
I generally don't bother discussing the Fair Tax, as it appears to be a dead horse both intellectually and politically. Even assuming one wants a flat rate consumption tax with no zero bracket, why use the retail sales tax model instead of a value-added tax (VAT)? The latter can lead to the same overall result but with better enforcement capabilities since the revenue authorities can cross-check rebates against taxes remitted on inter-business transactions, and since it can be embarrassing for a business to claim rebates on purchases without admitting to any sales on items that are no longer observable in inventory.
That said, I have learned more recently from people in VAT nations that the tradeoff is not quite as clearcut as I had thought. E.g., Europe has had fun lately with "carousel fraud," in which one side to an inter-business transaction claims a rebate, while the other side disappears before paying tax on the offsetting receipt. Also, U of Sydney law prof Graeme Cooper won, to my mind, the 2007 tax article title-of-the-year contest with his SSRN-posted piece, "The Discrete Charm of the VAT," available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1027512, in which he argues that the administrative tradeoffs are closer than various of us had been inclined to believe.