Until a website is ready for production, you must block the website from being found in search engines. We will do this by blocking 'bots.
HOW TO BLOCK 'BOTS':
Robot.txt is a file. When placed in the document root directory, this file prevents access by search engine robots.
STEPS TO MAKE A ROBOT.TXT FILE:
- Open text editor.
- Text edit.
- Type the proper approach from table below.
- Save as .txt file.
- In your Administration Panel, turn on setting allowing website to be viewed.
- Test to see if robot.txt is blocking robots.
HOW TO RUN A ROBOT.TXT VALIDATOR:
- Check if Robots.txt file is valid to the Robot Exclusion Standard.
- Type in full URL of site.
- Run validator. The validator will find syntax errors, 404 errors, poorly typed words, and suggest changes.
Block All Bots
Why: To keep sensitive information private, like databases of credit card numbers. To keep expensive research, when posted online, from being accessed.
User Agent: *
Why: To keep robots from accessing duplicate content, i.e., the original page and a printable version of the page A robots.txt file would prevent the print version from being accessed.
Use Forward Slash(/)
Why: To keep robots out of any folder containing important information like an administrative panel.
Use Two Slashes(//)