Meta Robots

The robots meta tag — a single line in a page's HTML head that tells search engines whether to index the page and whether to follow its links. The two core directives are index/noindex and follow/nofollow, combined into values like 'noindex, follow'. It's page-level control, which makes it more surgical than robots.txt: robots.txt stops crawlers at the door, while meta robots lets them in, lets them read the page, and then tells them what they're allowed to do with what they found. The distinction matters in practice — a page blocked in robots.txt can't have its meta robots tag read at all, which is how stores end up with 'indexed, though blocked by robots.txt' messes where Google lists a URL it was never allowed to understand. Used deliberately, meta robots keeps internal search results, filtered category views, cart pages, and thin tag archives out of the index while still letting link authority flow through them. Used accidentally, it's one of the most destructive tags in SEO: a 'noindex' left over from a staging site, a plugin setting flipped site-wide, a theme update that stamps the wrong directive onto every product template — and your catalog quietly falls out of Google over the following weeks. Nothing looks broken. The pages load fine for shoppers. Traffic just bleeds until someone thinks to view the source. We check meta robots directives early in any indexing investigation precisely because the tag is invisible in a visual review — it takes minutes to audit and can explain months of decline.