Controlling how your content appears in Google Search is a core aspect of SEO. Google provides official tools and guidelines to manage how and whether your web pages are indexed. In this blog, we explore the various indexing settings that Google supports, based entirely on information from official Google sources.
What Is Indexing?
Indexing is the process where Google analyzes and stores web page content in its database. Once a page is indexed, it becomes eligible to appear in Google Search results.
How to Control Indexing
Google allows webmasters to control indexing behavior using the following mechanisms:
1. Meta Robots Tag
This tag is placed in the <head> section of your HTML and helps define indexing preferences per page.
Common values:
| Meta Tag | Meaning |
|---|---|
<meta name="robots" content="index"> | Allow indexing (default behavior) |
<meta name="robots" content="noindex"> | Prevent indexing of the page |
<meta name="robots" content="noindex, nofollow"> | Prevent indexing and link crawling |
Google states: “Googlebot obeys the robots
<meta>tag when it is valid and present in the HTML.”
2. X-Robots-Tag (HTTP Header)
This tag performs the same function as the meta robots tag but is set via the HTTP header. It’s especially useful for non-HTML content like PDFs.
Example:
makefileCopyEditX-Robots-Tag: noindex, nofollow
Google documentation confirms support for the X-Robots-Tag on both HTML and non-HTML resources.
3. robots.txt File
This file tells Googlebot which URLs should not be crawled. However, it does not guarantee deindexing.
⚠️ Google clarifies in its robots.txt documentation:
“Blocking a page withrobots.txtdoes not prevent it from being indexed if other pages link to it.”
Use it when you want to reduce crawl load—not for preventing indexing.
4. Canonical Tags
Although not a direct indexing block, the canonical tag helps signal the preferred version of a page when multiple similar URLs exist.
Example:
htmlCopyEdit<link rel="canonical" href="https://example.com/page">
Google states: “Google uses the canonical page as the one to index and show in Search.”
Best Practices from Google
- Use
noindexfor precise control over which pages you don’t want in Search. - Do not use
robots.txtalone if your goal is to remove a page from Search results. - Verify changes in Google Search Console under the “URL Inspection” tool.
- Avoid conflicting signals (e.g., blocking a page in robots.txt while also setting a
noindextag).
How to Check if a Page Is Indexed
- Use Google Search Console‘s URL Inspection Tool
- Run a site query in Google:
site:yourdomain.com/page-url - Check response headers for
X-Robots-Tag - Review meta tags in HTML
Indexing settings are essential for controlling your site’s visibility in Google Search. Using the appropriate methods—such as the noindex directive or canonical tags—you can signal to Google exactly how to handle your content. Always rely on Google Search Central and Search Console for guidance and verification.
