Robots txt code

If it finds the robots. For a robots. Second, it gets placed in the root folder of the website, e. In brief, you define the bot user-agent the instructions apply to and then state the rules directives the bot should follow. A user-agent is the name used to define specific web crawlers — and other programs active on the internet. There are literally hundreds of user agents, including agents for device types and browsers.

Most are irrelevant in the context of a robots. On the other hand, these you should know:. For instance, if you wanted a certain page to show up in Google search results but not Baidu searches, you could include two sets of commands in your robots.

Sidenote: If there are contradictory commands in the robots. In short, a bot will follow the instruction that most accurately applies to them.

Directives are the code of conduct you want the user-agent to follow. In other words, directives define how the search bot should crawl your website. Here are directives GoogleBot currently supports, along with their use within a robots. Use this directive to disallow search bots from crawling certain files and pages on a specific URL path.

For example, if you wanted to block GoogleBot from accessing your wiki and all its pages, your robots. You can use the disallow directive to block the crawling of a precise URL, all files and pages within a certain directory, and even your entire website. The allow directive is useful if you want to permit search engines to crawl a specific subdirectory or page — in an otherwise disallowed section of your site. Since search bots always follow the most granular instruction given in a robots.

By including the sitemap directive in robots. Still, it is best practice to use the sitemap directive since it tells search engines like Ask, Bing, and Yahoo where your sitemap s can be found. Notice the placement of the sitemap directive in the robots. It can also be placed at the bottom. If you have multiple sitemaps, you should include all of them in your robots.

Either way, you only need to mention each XML sitemap once since all supported user agents will follow the directive. Note that, unlike other robots. You can add comments to remind you why certain directives exist or stop those with access to your robots. In short, comments are used to add notes to your robots. To add a comment, type. You can add a comment at the start of a line as shown above or after a directive on the same line as shown below :.

But what about other search engines? In the case of Bing , Yahoo, and Yandex , there is one more directive you can use:. The Crawl-delay directive is an unofficial directive used to prevent servers from overloading with too many crawl requests. Mind you, if search engines can overload your server by crawling your website frequently, adding the Crawl-delay directive to your robots. The crawl delay directive works by defining the time in seconds between which a Search Bot can crawl your website.

For example, if you set your crawl delay to 5, search bots will slice the day into five-second windows, crawling only one page or none in each window, for a maximum of around 17, URLs during the day. With that being so, be careful when setting this directive, especially if you have a large website. Just 17, URLs crawled per day is not very helpful if your site has millions of pages. The way each search engine handles the crawl-delay directive differs.

This means you can set a crawl-delay directive for the BingBot, Slurp, and YandexBot user-agents, and the search engine will throttle its crawling accordingly. Note that each search engine interprets crawl-delay in a slightly different way, though, so be sure to check their documentation:. That said, the format of the crawl-delay directive for each of these engines is the same. If you want to hide the page completely from Search, use another method.

Use a robots. This won't prevent other pages or users from linking to your image, video, or audio file. Before you create or edit a robots. Depending on your goals and situation, you might want to consider other mechanisms to ensure your URLs are not findable on the web. If you decided that you need one, learn how to create a robots. Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4. For details, see the Google Developers Site Policies.

Documentation Not much time? Beginner SEO Get started. Establish your business details with Google. Advanced SEO Get started. Documentation updates.

Go to Search Console. General guidelines. Search APIs. Create a robots. Here is a simple robots. All other user agents are allowed to crawl the entire site. This could have been omitted and the result would be the same; the default behavior is that user agents are allowed to crawl the entire site.

See the syntax section for more examples. Basic guidelines for creating a robots. Add rules to the robots. Upload the robots. Test the robots. Format and location rules: The file must be named robots. Your site can have only one robots. The robots. If you're unsure about how to access your website root, or need permissions to do so, contact your web hosting service provider. If you can't access your website root, use an alternative blocking method such as meta tags.

Google may ignore characters that are not part of the UTF-8 range, potentially rendering robots. Each group consists of multiple rules or directives instructions , one directive per line. Each group begins with a User-agent line that specifies the target of the groups. A group gives the following information: Who the group applies to the user agent.

Which directories or files that agent can access. Which directories or files that agent cannot access. Crawlers process groups from top to bottom. A user agent can match only one rule set, which is the first, most specific group that matches a given user agent. The default assumption is that a user agent can crawl any page or directory not blocked by a disallow rule.

Rules are case-sensitive. The character marks the beginning of a comment. Google is wonderful at crawling the internet to find websites, and a website under construction is no exception. Anyone can see what sections of your server you don't want robots to use. You will want to create an employee portal or secure login for confidential information. There are multiple tools you can run for diagnostic testing of your website.

There are other technical portions to the web, such as Schema markup, code structure, responsive design and image sizing, each of which are their own beast, but these are the first steps for beginners and intermediate website managers to improve their overall usability and site optimization.

How To Use robots.