Page 1 of 1

Robots.txt (overwrite)

Posted: Wed Oct 21, 2026 1:47 pm
by alextsampas
I set up robots.txt and WWB exports the lines below:

User-agent: *
Allow: /
Allow: /images/

I have set "allow all" but for some individual pages I set "Disallow" ,"noindex" etc
Why I do not see the "Disallow" lines in robots.txt but only in every page code?

Is it working like that ?

Re: Robots.txt (overwrite)

Posted: Thu Oct 22, 2026 5:39 am
by BaconFries
A robots.txt file only contains site-level rules that apply to crawlers before they fetch pages.
This means that when you set an individual page to "noindex", the do not add a Disallow rule to robots.txt won't be added to it instead they place a meta robots tag in the page HTML, such as
<meta name="robots" content="noindex,follow">
or
<meta name="robots" content="noindex,nofollow">
If it was added like in the following:
User-agent: *
Disallow: /private-page/
then Google may not process a page-level noindex tag this is why it is not added to the robots.txt and add in the meta tag as shown.

Re: Robots.txt (overwrite)

Posted: Thu Oct 22, 2026 8:14 am
by alextsampas
Thank you for your reply !