The main point is that people who like traps, as far as I can tell, aren't attracted to the fact they are male, but the fact they are a literal trap. Because it's not the "maleness" that attracts them, it isn't gay. I think the best example of this "genderless" attribute would be the iconic Hideyoshi, who is even referred to in-universe as some mystical third gender. This same idea carries over to many other traps, including the infamous reverse-trap, AKA the "You thought I was a guy acting like a girl, but really I was a girl acting like a guy acting like a girl."
Furthermore, traps fit into that sexual gray area that futanari also goes in. They don't really belong to either gender because of their extraordinarily feminine appearance and the fact they have a dick. You can't call liking traps straight, because they're definitely not female, but you can't call it gay because they're really not all that male either.
To close it out, I understand the people who think it's gay. It's a reasonable stand-point. However, as someone who likes traps but firmly hates non-trap dicks, I have to disagree.
TLDR: Don't feel bad because you like girly-guys. It's (possibly) not a gay thing. I mean, it's possible to be gay and like traps, but you could also be straight and like them too.