AITA for smoking weed on my deck?

I (m24) live next to this couple with two kids. My deck is pretty close to their backyard and it’s started to become an issue.

I sometimes smoke weed after I get home from work to unwind and to deal with some anxiety. My neighbors have started to complain to me about it. A month ago the husband knocked on my door and told me to stop doing it on my deck. I told him it’s quite literally the only place I can do it. He then told me he doesn’t feel safe letting his kids play in the backyard until I stop smoking and that he doesn’t want his kids to play in the backyard if he catches a whiff of it.

I said sorry but I won’t stop and that I’m well within my rights to do it. I told him I’ll avoid smoking if I see his kids outside. I also said I don’t complain when him and his friends drink in their backyard and get loud at really late hours. He threatened to get the police involved and I invited him to do so because like I said I was well within my rights to.

He ended up filing a complaint with the HOA and they sided with me and told him that it’s perfectly legal to do.

His wife came up to my house yesterday and kindly asked me to stop and said it drives him nuts when he looks out the window and sees me do it. I told her I’m really sorry but it’s none of his business.

I feel bad that what I’m doing is driving him nuts but feel like I am doing nothing wrong.

AITA?