Question about certain symbols being seen as “bad”

Does it bother anyone else that most Catholics seem to have a Protestant mindset of seeing symbols such as an upside down pentagram as being inherently evil or unacceptable to wear? Even with things more in line with Catholicism such the upside-down (Petrine) cross, people still make the argument that you shouldn’t wear it because it might lead to scandal, even if you can actually know the history of certain symbols.

Am I the only one who thinks that is a dumb argument? It seems to me that it inadvertently gives more seriousness to Satanism and paganism than they really deserve.