I have a co-worker who is married who came from a central office to a satellite office where I work. His co-workers from the central office made undermining jokes about him. I'm beginning to understand why. He boasts about an association he's in and tells stories about what he's doing.
He visits my cubicle often, too, and insists on going to lunch with me, but I stopped that. He was even told by management that he talks to me too much. I once mentioned the kind of wine I like and he bought me a bottle. I offered to reimburse, and he insisted on not receiving any money. I don't think that was appropriate, since he is married.
I don't feel comfortable going to lunch with him, and I feel uncomfortable whenever he is around me now. My intuition is telling me to stay away, and keep my distance. I want to know what is the best way to handle a person like that. I hate feeling like this in my office, but he's an oddball that pays too much attention to me.
You have a right to feel safe in the work environment without being harassed by a colleague. You need to talk to your supervisor both to obtain support and to determine the best way to handle this situation. (It sounds like your supervisor may already be aware of this problem, to some extent.) Someone in authority (perhaps the supervisor, someone from human resources, or both) needs to tell this guy clearly that he is making you uncomfortable and such behavior is unacceptable in the workplace.
If he approaches you, be firm and direct but remain calm. Tell him that he is disrupting your work and making you feel uncomfortable, and that you want your relationship to remain professional and work-related only. Try to avoid any situations where the two of you are alone. Document and report his behavior if it continues. Of course, do not initiate unnecessary conversation or accept any gifts from him.
This is a clear-cut situation, and you need to be clear where you stand.
Hope this helps.