I can remember when I first saw someone mention a robot.txt file on one of the internet marketing forums I immediately switched my mind off and thought about something else because I wrongly assumed that it was something which was far to technical for me to even understand.
Before I go any further I will tell you what a robot.txt file is, just in case you haven’t got a clue. It is basically a piece of code that you put on your website which tells the robots or spiders that come from the search engines what they can and what they can’t index. When I first started out in internet marketing I thought it must be something on the shady side, after all, why would you need to hide parts of your website from the search engines unless you were up to no good.
I couldn’t have been more wrong because being able to use a robot.txt can be of great importance to many webmasters and not in the way that you may have thought. To start with, it may be the case that certain pages of your website are under construction, you may be trying something out and the last thing that you want is other people seeing a half finished websites or some of your ideas which you would rather keep private for now. Just imagine that unfinished pages of your website came up when someone who wanted to business with you was carrying out a search, the half finished pages may give the impression that you are not an established company.
If you place the robot.txt file in the appropriate directory, the search engine robots, or spiders as they are often called will simply ignore those pages so that you will have nothing at all to worry about.
It is even possible to direct a robot.txt file to forbid certain robots from spidering your website, for example. There are robots out there which will search your website for email addresses, the robot.txt file will stop them from doing this, if you so wish.
So as you can see, it is well worth getting acquainted with the wonders of a robot.txt file as you never know when it will come in handy, I just wish I had not been so blinkered in my approach when I was first starting out and I would highly recommend that you use this txt file in your online business at every possible opportunity.