Learn more about Search Engine Indexing!

The robot.txt file is used by search engine bots to analyze which content to index or deindex from your web server. If there is no explicit mention of de-indexing any folder or files then search engine bots will index everything by default. They crawl and index almost everything unless they’re advised not to in “disallow” variable. Learn more about controlling the idexing of your web sites!