Dignity nobility good quality condition highness standing course caste standing superiority priority vogue distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot regarding which web pages will not be to generally be crawled. Like a search engine crawler https://www.youtube.com/watch?v=OeFqbAORPUg