It almost always means you're dressed too casual in such a way that it stands out

4

I think tone and any other simultaneous comments from the same person are major here. I’ve told people I love their shirt/sweater and asked where they got it and said it looked really comfy. In that context, totally a compliment. If someone said “you look…comfy.” 100% insult

1