Robots meta directive
Robots meta directives (sometimes called “meta tags”) are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file directives give bots suggestions for how to crawl a website's pages, robots meta directives provide more firm instructions on how to crawl and index a page's content.
Description
There are two types of robots meta directives: those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as “noindex” and “nofollow” in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.
Meta directives give crawlers instructions about how to crawl and index information they find on a specific webpage. If these directives are discovered by bots, their parameters serve as strong suggestions for crawler indexation behavior. But as with robots.txt files, crawlers don’t have to follow your meta directives, so it’s a safe bet that some malicious web robots will ignore your directives.
Below are the parameters that search engine crawlers understand and follow when they’re used in robots meta directives. The parameters are not case-sensitive, but do note that it is possible some search engines may only follow a subset of these parameters or may treat some directives slightly differently.
Indexation-controlling parameters:
- Noindex: Tells a search engine not to index a page.
- Index: Tells a search engine to index a page. Note that you don’t need to add this meta tag; it’s the default.
- Follow: Even if the page isn’t indexed, the crawler should follow all the links on a page and pass equity to the linked pages.
- Nofollow: Tells a crawler not to follow any links on a page or pass along any link equity.
- Noimageindex: Tells a crawler not to index any images on a page.
- None: Equivalent to using both the noindex and nofollow tags simultaneously.
- Noarchive: Search engines should not show a cached link to this page on a SERP.
- Nocache: Same as noarchive, but only used by Internet Explorer and Firefox.
- Nosnippet: Tells a search engine not to show a snippet of this page (i.e. meta description) of this page on a SERP.
- Noodyp/noydir [OBSOLETE]: Prevents search engines from using a page’s DMOZ description as the SERP snippet for this page. However, DMOZ was retired in early 2017, making this tag obsolete.
- Unavailable_after: Search engines should no longer index this page after a particular date.
Types of robots meta directives
There are two main types of robots meta directives: the meta robots tag and the x-robots-tag. Any parameter that can be used in a meta robots tag can also be specified in an x-robots-tag.
We’ll talk about both the meta robots and x-robots tag directives below.
See also
External links
- Robots Meta Directives @ moz.com
- noindex @ Wikipedia
- Using Noindex, Nofollow HTML Metatags: How to Tell Google Not to Index a Page in Search @ blog.hubspot.com
