I think that a good reply to any Christian who has not lived a lifetime of celibacy and who accuses one of giving in to his desires is to point out that St. Paul recommended celibacy, but said it was "better to marry than to burn" - i.e. he gave a way out for those who didn't have the strength of character to keep it in their pants.

So if they don't have it, accusing you of not having it is the pot calling the kettle black.