Advice please

I have a coworker who has been hitting on me. I went along with it at first and flirted back. Then he called me one night and asked me out. I told him I can’t date cuz I’m not ready for that right now. He said that we could just hang out as friends. His idea of hanging out as friends was going to dinner and a movie just us. When he tried to call again a couple days later I ignored his call. I love the attention and validation it gives me. However I can see so many red flags. I know he’s not a good guy and I don’t like him as a person. He is not good at his job and is very selfish (we take care of people with disabilities). I guess I’m just looking for some encouragement or insight. I know that I need to set a hard boundary with him but I’ve always struggled with that.