In order to avoid unwanted content material from the search indexes, site owners can instruct spiders never to crawl sure information or directories from the conventional robots.txt file in the root directory in the domain. Additionally, a page might be explicitly excluded from a online search engine's database through the https://news.rafeeg.ae