WEB Course

robots.txt

Go back

This is one of the files that robots are checking when they visit your website. You may block some robots, or tell them what URL then cannot see/index, or where is the sitemap.

Of course, robots may ignore this file, and you should not give critical information (for instance, you are forbidding access to /download/, but you are also telling hackers that you have a folder /download/).

You may create a robots.txt as explained on robots-txt.com. You can test your robots-txt using Google SC. And, you got a lot of examples on robots-txt. Here is mine

# allows everyone
User-agent: *
Disallow:

# link to the sitemap (optional)
Sitemap: https://memorize.be/sitemap.xml