
I dealt with the same thing a few years ago with my ex (we broke up for unrelated reasons). It got to the point where I was embarrassed to be in public with him because I was so worried about how people would perceive him and me for dating him. I would maybe talk to a therapist about it, otherwise if it's at a point that you don't like being with him you should probably end things to spare his feelings.