Say Reuven eats his fill at a Kiddush, and has no appetite for the Shabbos Seudah. So, as to not make his wife feel bad, he eats anyway, but it has no Taam. He feels like a Tzaddik - man, what mesiras nefesh I put myself through, just so my wife wouldn't feel bad! But, does he have Hakaras Hatov for all the effort his wife put into preparing the Seudah? That far he doesn't go - after all, he was full anyway, and would have preferred to just hit the pillow! His wife, really, should have Hakaras Hatov to him for being so considerate! Look how unfair that is to his wife - she put in all the effort, and he has no appreciation whatsoever, simply because he ruined his appetite.
The same is true when a husband allows his eyes to stray. He doesn't feel all that attracted to his wife anymore, and any attention he gives her makes him feel like a Tzaddik. Certainly, he doesn't appreciate all her efforts, to look nice for him, etc. By ruining his appetite for a healthy attraction to his wife, he has done her, himself, and their marriage a tremendous disservice.