Slavery can be justified because we are not obliged to recognize the natural rights of those whom we do not expect or demand reciprocal right-recognition from. The state is illegitimate not because it uses violence but because its justifications for the use of violence do not hold up to logical scrutiny.
This is why it is acceptable to own (or enslave, per se) animals like cows and negroes. And why it should be admitted as justified to enslave those who aggress upon you, as the moment they have broken the ethical contract of non-aggression, they are likewise not protected by it.
Societies shall therefore be formed of groups of people who reciprocate the sentiment of non-aggression within the social group. Such social contract is akin to an alliance between macrocosmic states, wherein if one is attacked all alliance members attack the aggressor. If one member begins aggression upon others in the group, and is unable to be resisted, the group can no longer be said to be a society, but instead a pack of slaves owned by one slave-master. As stated, such organization is only 'illegitimate', logically illegitimate, if the slave-master does not recognize his status as an enslaver of men and likely oathbreaker. These titles are objective, of course, and are the metaphysical price of domination. If the enslaver fails to recognize his true nature, his life becomes the domain of lies, and thus he may pay even greater in a solipsistic denial of reality itself.