Am I doing all the right things?

I've been going out with these guy about 3 years, we are happy couples actually but myself will be having a mive mood swings for no reason, well that's my attitude. Recently I found out that my man is dating out with his pet sisters friend, (he told the gal that he broke off with me cause could not get along with my attitude), and they have been dating out about 3 months, and he SLEPT with the girl. So after all the conflicts, hu ha and so on we 3 had a talk (myself, the new gf and my man) and he claim that he was only flirting with her and im his true love, so I forgave him and we patch up again, but the problem is every moment when he touch me, talk to me and whatever he do to me my mind keep thinking on whatever that he does to that girl, I feel so guilty when he touches me, even I did told him this and it will end up in argument, the thing is im not happy from the deep inside my heart, so am I suppose to carry on with this relationship, or need to have a serious talk, or just leave him. I so hurt deep in here, please help.