This struggle is very normal, and everybody deals with it on some level or another. Having a desire for this doesn’t mean that there is anything negative about a person. Chazal tell us that , a person’s nefesh craves it. Chazal aren’t referring specifically to bad people; they are addressing every single person.
There is no kehillah that can be considered immune to this. Every group of Yidden across the entire world deals with this yetzer hora.
You need to realize that you have no control over when a nisayon comes your way. An improper thought may fall into your head suddenly, out of the blue. Since this is the case, you have no reason to feel bad that the thought came; it was out of your realm of bechirah.
Bechirah, however, does lie in whether the person will leave the thought alone and allow it to pass through,or whether he will dwell on it, which causes it to leave a stronger impact on him.