Robots.txt file? How to invite search bots?

[yoda313]

So I've been reading a little about the robots.txt file that would be used to promote your own website or restrict which pages are searched.

Is there a simple way to say search everything? I read that having the file in your root directory will give an all clear for spiders to go through the website. I just want to know what the search all command is.

Thanks for any advice.

[thecoalman]

just don't make one.

If you must:

User-agent: *
Disallow:


This will ban bots from every page:

User-agent: *
Disallow: /

[yoda313]

Oh I don't want to ban bots I want to invite them. Hmm... I thought there was an allow all command.

Well I guess maybe I'll forget about it. Thanks coalman

[thecoalman]

Oh I don't want to ban bots I want to invite them. Hmm... I thought there was an allow all command.

Well I guess maybe I'll forget about it. Thanks coalman

No there's no allow command........ the first one doesn't disallow anything though.

The * is a wildcard which means every bot, there's no files or directories listed after the disallow so that means every bot can visit every page.

In the second example with the slash added it disallows every bot from every page. The / is equivalent to you HTML directory.

[yoda313]

Thanks coalman