Instead, we have another more complicat but effective way at the server level that we can use. Google has document the use of robots directives with the use of “x-robots-tag” in the headers. According to this definition we only have to send a header of the type “x-robots-tag” with the same value that.
We would put in the robots meta-tag to pass it from the server and not from HTML
Apart from the system itself, this opens the door to creating a file that manages these headers on our server. language of our CMS (PHP, Java, .Net, etc.) or directly in the server configuration. In it, thanks to the .htaccess and .htconfig files, we can declare the sending of headers in a single file that defines what the robot can and cannot index.
Example to mark a “noindex,follow” in pagination of your website through the .htconfig file:
Or not index paginations through the modRewrite module with which we manage our home owner database friendly URLs and rirects: RewriteCond %{THE_REQUEST} “^(GET|POST) .*page/[0-9]+ HTTP”
RewriteRule .* – [ENV=NOINDEXFOLLOW:true]
Header set X-Robots-Tag “noindex,follow” env=NOINDEXFOLLOW
But I don’t want to go into too much detail about this system. If you want to read more about how to put it into practice, I invite you to visit collaboration with traditional chinese another post where I detail 10 other things you should know about meta-robots . The last one explains this system in detail.
I don’t know if you’ve experienc it like me as
I’ve discover over the years everything I’ve explain in this post, but the truth is that, like everything in SEO, you realise that you never know enough about everything. I thought it would be interesting to compile all this because I continue to see over time that the problems that exist on many sites due to not properly understanding robots.txt files are still usa data there year after year and no one talks about them too much.