I guess men who have been exposed liking such material are on spot because REAL criminals harming the peace and safety of society on the one part are mostly male - and no, I'm not being feministic. It's a commonly known FACT that men are most likely to become sexual predators if certain factors are given.
Also the presence of organizations lead or protected by feminists causes an unbalance on the issue. Newspapers, television, magazines pick up stories to favor the groups that have a noticable impact on the masses. Masses regardless of their single opinions support everything covered with the title 'noble cause' - also feminist extremism, since they view it as their duty.
Most ppl think supporting these 'noble causes' give others the impression they fit the image of a worthy and noble human - this cicle gave birth to the belief having personal opinions openly said is not appropiate if it doesn't match the thoughts of the mass.
Basically even if 50% would say lolicon 2D material is just 'cartoon fiction', they would still send a guy owning lolicon manga to jail.
Third reason why men are mostly the victims of punishment for deviant sexual interests is that society views men as potential predators. From ancient times on females have been given the features of weakness and powerless-ness. As such even in todays modern and more educated times ppl do wonder on female criminals committing murder or sexual abuse. Abusive and violent behavior are both features naturally matched to males. Especially sexual crimes are seen by females as less harmful as by males, since the natural image of violation is not there.
Society still didn't step out of the stereotypic view, that sex means a mens penis entering a womans vagina - at least one of her openings XD Male homosexuality is considered a fulfilled sexual act, because a penis is entering an opening. That is most likely not possible by female homosexuals.
It's the same by crime, womans do not enter any intim parts of men with an organ of their own. So basically in our underminds there is no sexaul act committed...
The issue of 'starge looks' is different from being labeled as a pervert. If you are too much into guns or anything other not really common as a hobby you'll get strange looks. It's more the matter what ppl around you, how much tolerance do they have. If you're lucky, ppl don't give a damn about your likings :)
Erm...Sorry for the wall of text XD