What really determines the Maiseh, is the machasavah that is behind it. When one is having yachasim with their wife, it is so important to control the thought process that is taking place at the same time. It is the thought process that determines what you are doing. The Baal Shem Tov says, wherever your thoughts are...that is exactly where you are at that moment. So, even if you are with your wife, and you are thinking at the same time of someone else's wife, you are with the other women, and NOT your wife. This is what is taking place in your brain, and this is what the Baal Shem Tov is referring to. It may look like your wife, but in reality, in the spiritual- (neuron world), you are destroying yourself.
The Baal Shomer Emunim says that our minds are filled with compartments, and compartments within compartments, millions and billions of nooks,cranies and boxes in our minds that our stuffed with ALL of our thoughts, and experiences. These compartments are filled with the experiences and feelings of the fullfilment of our taivos etc. Literally every thought, every bit of excitment and stimulation is not lost. It is stored here in our minds. As soon as we begin to re-think, and act in a way that is in harmony with one of these compartments, (even something that happened 20 years ago), the memory, it's pictures, and all of the neurological excitement will come out of this compartment and lay right in front of our eyes, and re-stimulate our nervous systems.
So it is not enough to be with one's wife physically, but one must work to make it a holy union-Tahor. How many times, when one is with their wife, and in the height of the experience, the menuval comes and starts to put thoughts of other women in your mind. So, it is not enough to be with one's wife physically but also on the Machshavah level.
BTW. In addition to what Mevakesh said, not to touch one's bris etc. The Baal Shomer Emunim says also not to even look at one's bris, let alone someone's else's, (for example at the mikvah etc). He says that there are klipot there, and when one looks, these klipot have permission to cling to you. Of course to see without having intention is much less severe. It is for this reason, even when relieving oneself I am very careful not see anything. We must fight him on all fronts!