This means that the file does not exist or is not open to the public. In this case, Google will understand it as an empty file and will grant access to everything as if it did not contain any disallows.
Code “301” means “content mov permanently” and is accompany by a URL where the new content is. Google interprets for all purposes that the content of this.
To which robots.txt rirects is the robots.txt itself
Even if there is a folder change, the URL is on another domain or if the URL does not have the name “robots.txt”. to manage robots.txt programm in some systems. We can treat the URL robots.txt only as a rirection to where we actually manage our robots.txt file.
However, it can also be a problem
In poorly execut migrations from one domain to another or from http to https. In these cases we may find that when migrating a site we return a 301 to the new site with the robots.txt, which would mean applying the new robots.txt to the old viber database domain, no longer blocking the old URLs and potentially causing a cascade of error detection and loss of crawling time.
Generally, the recommendation
Should be that all sites have their own /robots.txt and that they never rirect, but in most cases this is not done.
Other codes (especially 503) . If there is an error serving the file, Google understands that the domain is down and in order not to interfere with a highly rank website is often the crawling of the site, it will only consult the 503 file until its error changes. Stopping crawling does not mean deindexing, unless we keep the server like this for so long that.
The links and content of
The page begin to lose strength, generally it is better not to do it for more than a few hours. We do not know exactly how Google acts if this error persists usa data over time, but for whatever reason it usually leads to loss of authority as it tries to reindex the website.