Robots.txt is a text file that tells search engines which parts of the site to crawl or not to crawl. It helps so control what appears in search resultsand what will remain hidden. It is used, for example, to Preventing indexing of duplicate pages or technical parts of the site. It is designed mainly for search engine robotsnot for the general public.
Specific rules for individual AI bots, such as GPTBot or Google-Extended, are entered into robots.txt using the same syntax as for traditional search bots, but with a different user-agent. We discuss when to allow them access and when to block them in our article on AI Robots on the Web.
