Fun fact, there is nothing from stopping liberty lovers from doing some version of voluntaryism right now, at least on the community building end of things.
People could join a network (mormons and LDS have something like this) in which they agree to contracts between themselves being arbitrated by mediators instead of govt agents ("judges"). A strong community based on respect for basic rights grows, and people want to stay in this grouping of quality people, who might perhaps see fit to sidestep some of the multitude of infantilizing rules and regulations dreamt up by gov't agents. Breaking the rules gets one kicked from the group, losing both community and business..
Seeing how easy an idea this would be to adopt, I think our critics have some basis in calling us hypocrites, even if their claims of "loving peace" while embracing a system of institutionalized violence (big government) is of course much more hypocritical.
Related, Rothbard was pretty solid, these purity tests & related hostilities are not really helping matters.