Robots.txt

All WYSIWYG Web Builder support issues that are not covered in the forums below.
Forum rules
IMPORTANT NOTE!!

DO YOU HAVE A QUESTION OR PROBLEM AND WANT QUICK HELP?
THEN PLEASE SHARE A "DEMO" PROJECT.



PLEASE READ THE FORUM RULES BEFORE YOU POST:
http://www.wysiwygwebbuilder.com/forum/viewtopic.php?f=12&t=1901

MUST READ:
http://www.wysiwygwebbuilder.com/getting_started.html
WYSIWYG Web Builder FAQ
Post Reply
lpug
 
 
Posts: 111
Joined: Thu Apr 05, 2018 10:15 am

Robots.txt

Post by lpug »

Ive looked through the forum and could not see answer to my question, in the tools menu I configured my robots.txt file and when I select go it shows the prompt about it will rewrite file if it exists, however nothing happens i.e. there is no file generated. In each page under SEO i have also set my no follow links etc. Am I missing something obvoius?
The sitemap file did generate and ftp to my site.

Thanks in advance.
User avatar
Pablo
 
Posts: 21574
Joined: Sun Mar 28, 2004 12:00 pm
Location: Europe
Contact:

Re: Robots.txt

Post by Pablo »

lpug
 
 
Posts: 111
Joined: Thu Apr 05, 2018 10:15 am

Re: Robots.txt

Post by lpug »

Ok got it, its published when you publish a page or site. However I have individually setup which pages I want indexed and the robot file still shows allow all to be indexed. Even in my pages I have set them up in the SEO tab....is there anywhere else I should be looking at

User-agent: *

Allow: /

Allow: /images/

Sitemap: https://mysite.com/sitemap.xml
Pablo wrote: Fri Jun 17, 2022 2:32 pm Maybe this will be helpful?
https://www.wysiwygwebbuilder.com/robots_txt.html
User avatar
Pablo
 
Posts: 21574
Joined: Sun Mar 28, 2004 12:00 pm
Location: Europe
Contact:

Re: Robots.txt

Post by Pablo »

If you need further assistance then please share a DEMO project
Also, describe exactly which settings you have used and what the expected result is.

See also:
https://www.wysiwygwebbuilder.com/forum ... 10&t=82134
lpug
 
 
Posts: 111
Joined: Thu Apr 05, 2018 10:15 am

Re: Robots.txt

Post by lpug »

Hi Pablo,

Ive created a new project and the Robot file is still not being generated as it should be. I have created a demp project with 4 pages, page 3 I have set as to not index or follow and yet the robot file is not correctly outputing.

https://1drv.ms/u/s!AkH9C6w0oL9tiw9Pzv6 ... L?e=Mn4bNH
User avatar
Pablo
 
Posts: 21574
Joined: Sun Mar 28, 2004 12:00 pm
Location: Europe
Contact:

Re: Robots.txt

Post by Pablo »

Everything seems to be correct.
What did you expect to happen?

Note that robots.txt only include folders rules
Rules for individual pages are set in the page itself via meta tags:

Code: Select all

<meta name="robots" content="noindex, nofollow">
lpug
 
 
Posts: 111
Joined: Thu Apr 05, 2018 10:15 am

Re: Robots.txt

Post by lpug »

The robots outputs this

User-agent: *

Allow: /

Allow: /images/

Shouldn't show that page3 is disallowed
Like this

User-agent: *

Allow: /

Disallow:/ page3

Allow: /images/
User avatar
Pablo
 
Posts: 21574
Joined: Sun Mar 28, 2004 12:00 pm
Location: Europe
Contact:

Re: Robots.txt

Post by Pablo »

Shouldn't show that page3 is disallowed
Like this

User-agent: *

Allow: /

Disallow:/ page3

Allow: /images/
No!
The robot rules for individual pages are set in the page itself via meta tags.
Otherwise the page may still be indexed if it was linked via another website.
User avatar
jerryco
 
 
Posts: 826
Joined: Fri Mar 27, 2009 2:42 pm
Location: Purmerend, Holland

Re: Robots.txt

Post by jerryco »

Otherwise the page may still be indexed if it was linked via another website.
I'm learning something here! Thanks Pablo!
// Love is the acceptance of nothing / Account age is no guarantee of efficiency ;-) ->

Above, Beyond, and @wwonderfull! <- Genuinely helps you with a powered up site that counts! Four Times Excellence!
Post Reply