robots.txt

WenWys · Post by **WenWys** » Tue Apr 21, 2009 11:12 pm

can anyone explain how to generate a robots.txt file ?

genieuk · Post by **genieuk** » Tue Apr 21, 2009 11:18 pm

Hi,

This is the wrong section.

Anyway to anwer your question

In Google type: robots.txt file generator

The first several results are links to robots.txt generators. All online and free of course.

If you feel you want to learn more about robots.txt file etc go here: http://www.robotstxt.org/

Hope this helps you.

Mathew

star57 · Post by **star57** » Tue Apr 21, 2009 11:18 pm

WenWys wrote:can anyone explain how to generate a robots.txt file ?

try this
http://googlewebmastercentral.blogspot. ... obots.html

Eddy · Post by **Eddy** » Wed Apr 22, 2009 7:21 am

I think you search for something so that spiders do not index your pages or images.

tutorial from wb how to ad html and javascript.
http://www.wysiwygwebbuilder.com/add_html.html

metatag are for your page text(script) and urls if you use a noindex,nofollow than google wil not index your site for the search engines!
Metatags always between <head> and </head>

Code: Select all

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

http://www.robotstxt.org/meta.html
==============================================

for only protect your images for searchbots you use a robots.text

http://www.gnc-web-creations.com/creati ... t_file.htm

you can read how to make a robots text,you upload it with ftp.
This example allows all robots to crawl all files except the images file.

Code: Select all

User-agent: * 
Disallow: /images/

But not all spiders read the robots.text file or metatag ,the best is if you also watermark your images.
=============================================

You can also make a htaccess file if you site is on a apache server to prevent hot linking.
Read the tutorial about Preventing hot linking of your images and other file types

http://www.javascriptkit.com/howto/htaccess.shtml
http://www.clockwatchers.com/htaccess_images.html

Prevent viewing of .htaccess fileIf you use htaccess for password protection

Code: Select all

<Files .htaccess>
order allow,deny
deny from all
</Files>

robots.txt

robots.txt

Re: robots.txt