
NTA. You have free will, don't let yourself be pressured into going somewhere you don't want to go, especially if you've already talked to him about it. That is not your religion and you are not held responsible for attending any events associated with it, so it's completely valid that it's not something you're comfortable going to.