To avoid unwanted content during the search indexes, website owners can instruct spiders not to crawl particular files or directories throughout the normal robots.txt file in the foundation directory in the domain. Moreover, a page can be explicitly excluded from a online search engine's databases by using a meta tag certain to robots (usually ). E