This question merits topic status. I don’t see obscurity and invitation-only helping to achieve critical mass. Invitation-only, if done by individuals, can still let in disagreeable elements.
What helped me rein in my worse impulses was the understanding my IP address would be blocked if I were banned, but I shared it with a regular. Other communities use invitation after a group discussion, with the guideline that the sponsoring inviter be responsible for the behaviour of the candidate invitee, who the inviter knows or trusts to be good, kind and loving and mature, calm and effective in handling misunderstandings and disagreements.
Incentives to look at the Community Guidelines could be promoted. I was shocked to learn that the percentage of accounts created on bbs.boingboing.net that had read the Community Guidelines was a little over 9% (as of this typing, 1048 have read Community Guidelines and 11357 accounts have the Basic badges).