Robots.txt vs noindex vs password protection
Choose robots.txt, noindex, X-Robots-Tag, or password protection by separating crawl, search, file, and privacy goals.
Four URLs need four different outcomes.
A faceted filter should consume less crawl attention.
A staging page should not appear in search.
A PDF should remain accessible to customers but stay out of search results.
A private investor document should be unavailable to the public.
Using the same rule for all four is the mistake. The robots.txt vs noindex choice only makes sense after the real goal is clear. Crawl management, search exclusion, non-HTML control, and privacy are separate jobs.
One rule cannot solve four goals
Robots.txt primarily manages crawler access. A noindex directive tells a search engine not to show an accessible resource in results. An X-Robots-Tag can carry that instruction in an HTTP response, including for non-HTML files. Password protection controls access for people and crawlers that do not have credentials.
Those mechanisms can interact. They can also undermine each other.
Google's robots.txt introduction warns that a disallowed page can still appear as a URL in search results when other pages link to it. Robots.txt is not a reliable way to hide a web page from search and it is not a security control.
Route the crawl-efficiency case
The faceted filter creates many low-value URL combinations. The goal is to reduce crawling of unimportant or repetitive paths.
Robots.txt may fit this job when the team understands which patterns can be blocked without preventing Google from seeing essential page resources or discovering useful content. The rule belongs at the host level and affects compliant crawlers.
Before adding it, ask these questions.
Can important pages only be reached through the blocked paths?
Do blocked scripts or styles affect how the site is understood?
Are internal links still generating the unwanted combinations?
Would changing the navigation or parameter handling solve the root problem?
Robots.txt can reduce access to a pattern, but it does not clean up the content model. It also cannot guarantee that the URL disappears from search.
Route the search-exclusion case
The staging page should be publicly reachable for review but absent from search results.
A noindex directive can communicate that goal, provided the crawler can access the page and read it. For HTML, the directive is commonly delivered in a robots meta tag. Google's robots meta tag documentation explains the supported indexing and serving controls.
Do not disallow the same URL in robots.txt and expect Google to discover the noindex instruction. If crawling is blocked, the crawler cannot read the page-level directive. The result can be a disallowed URL that remains known through links.
For a staging environment with sensitive or unfinished material, public access plus noindex may still be too weak. The page remains available to anyone with the URL. Use access control when confidentiality matters.
Route the non-HTML case
The PDF must remain available to customers with the link but should not appear in search.
An HTML meta tag cannot be inserted into a PDF. Google documents the X-Robots-Tag HTTP response header for non-HTML resources such as PDFs, video files, and images. A server can return X-Robots-Tag: noindex with the file.
This approach depends on server or hosting configuration. Test the actual response header, not merely a rule in a deployment file. Also review where the PDF is linked. Excluding it from search does not make the URL private or prevent sharing.
If the document has an HTML landing page, decide which resource should be discoverable. Often the HTML page belongs in search while the file remains a supporting download.
Route the privacy case
The investor document should not be public.
Neither robots.txt nor noindex protects it. Both are instructions for crawler behavior, not authentication. Use password protection, authenticated access, or remove the file from a public host.
Google explicitly recommends password protection when information needs to be secure from web crawlers. This boundary is worth writing into the publishing checklist because teams often treat search exclusion as privacy.
A useful test is blunt. If someone posts the URL in a public chat, should an unauthenticated person be able to open it? If the answer is no, indexing controls are the wrong layer.
See the conflict before deployment
The most common conflict looks like this.
A page receives a noindex meta tag.
The same path is disallowed in robots.txt.
The crawler can no longer revisit the page to read the noindex rule.
The URL remains known from internal or external links.
The intent was exclusion. The implementation blocked the mechanism that communicates exclusion.
Another conflict occurs when a site-wide header adds noindex to files that should remain discoverable. The X-Robots-Tag can apply broadly through server configuration, so audit scope carefully.
When multiple robots rules conflict, Google says the more restrictive applicable rule is used. That does not rescue a directive that was never fetched because crawling was blocked.
Use a goal-first routing table
Goal | Primary control | Important limit |
|---|---|---|
Reduce crawling of unimportant paths | Robots.txt or a better URL design | A blocked URL can still be known or appear without a snippet |
Keep an HTML page out of search | Accessible | The crawler must be allowed to fetch it |
Keep a non-HTML resource out of search |
| The resource remains publicly accessible |
Protect confidential content | Authentication or removal from public hosting | Search directives are not security |
The table is a starting point. A real site may also need canonicalization, redirects, internal-link cleanup, removals, or server changes. Do not stack controls until the goal and current behavior are documented.
Check before the rule goes live
Use a short pre-deploy review.
Write the desired outcome for the exact URL pattern.
Classify the goal as crawl, indexing, presentation, or privacy.
Confirm the file type and response behavior.
Check for conflicting robots.txt, meta, and header rules.
Verify that important destinations and resources remain crawlable.
Test the live HTTP response after deployment.
Record an owner and a removal condition for temporary rules.
Temporary staging directives are especially easy to forget. Include them in launch checks and make sure the production release removes any noindex rules that were only intended for development.
The clearest request format is one sentence.
For these URLs, we want to control this specific outcome while preserving this access.
If the team cannot complete that sentence, it is too early to choose between robots.txt, noindex, a response header, or authentication.
Keep that sentence beside the deployed rule. When the page changes purpose, the team can reassess the outcome instead of inheriting a mysterious block. A rule without a recorded goal tends to outlive the problem it was meant to solve.
After a launch or migration, sample the affected URLs again. Confirm that public pages remain fetchable, excluded pages carry the intended directive, protected resources actually require credentials, and removed temporary rules did not leave new content hidden. The second check catches configuration drift that a correct pre-deploy review cannot predict.
Record the live response with the rule owner and review date.
