robots.txt March 18, 2017 p1-2017 The robots.txt file gives instructions about your site to web robots that crawl your site regularily for indexing; this is called The Robots Exclusion Protocol.
