Sunday, September 2, 2012


Robot.txt  is a file a text file which allows robots, search engines crawl your site better. It is not only for google bot but any search engine bots or even can be for any robot whose aim is different than searching.

You can simply write there

User-agent: *
Allow: /

What it means?" User-agent:*"   stands for any robot. Allow: / it is allowing robot search your website.

How to create a robot.txt for blogger

On dashboard, settings, search preferences find custom robot.txt. Usually default settings is disabled for it. But you can change it. Click to edit and Enable custom robots.txt content? choose yes. And underneath text area write what i wrote on top.

And top of it there is a message warn you " Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.". It is true it can cause search engine ignore your site for example if you write there; "Disallow: /" instead of "Allow:/" .

Lets say you want to write it only for google. It can be. Because some malware and malicious robots can enter site if you allow any robot such as email collectors, etc..

User-agent: Google

User-agent: *
Dissallow: /

But this will result only google bot can crawl your site but not other search engines.

How we can use it on website not in the blog. You can use it on html meta tags between head tags. For example <meta content=robots.....

<meta name="robots" content="all" />
<meta name="robots" content="index,follow" />

and if you want they not follow a link you should put inside href tag this  rel="nofollow"

1 comment:

  1. please share detail method of using robot.txt


Welcome to Bali Villa & Land

Welcome to Bali Land & Villa

This site is mostly for real estate business and Bali Property but little bit from everything content. If you have any questions please comment on entries or email me directly.

Bali is the one of the most beautiful island in the world. It has increasing number of visitors every year. It is also most profitable property place in the world. And you can find some of those properties and solutions here.

I hope you will enjoy.

Thanks for visiting

Some entries' contents are still weak but will be updated