How did Brahmanas ensure compliance with the prescribed norms?

By emphasizing the divine origin of the varna order

The Dharmasutras and Dharmashastras contained guidelines regarding the ideal "occupations" for each of the four varnas or categories. Brahmanas were expected to study and teach the Vedas, perform and oversee sacrifices, and engage in giving and receiving gifts. Kshatriyas were to involve themselves in warfare, protect people, administer justice, study the Vedas, participate in sacrifices, and make charitable offerings. Vaishyas, in addition to the duties assigned to Kshatriyas, were also expected to engage in agriculture, pastoralism, and trade. Shudras were restricted to one occupation, serving the three "higher" varnas.

To ensure compliance with these norms, Brahmanas adopted various strategies. Firstly, they emphasized the divine origin of the varna order. Secondly, they advised kings to enforce these norms within their realms. Lastly, they attempted to persuade people that their social status was predetermined by birth. However, convincing everyone of this idea was challenging. As a result, these prescriptions were often reinforced through stories narrated in texts like the Mahabharata and others.