Robots.txt is a plain text file at the root of a domain that tells search engine crawlers which parts of a website they may or may not crawl. It lives at \/robots.txt, applies per host and protocol (www.example.nl and shop.example.nl each need their own) and uses four directives: User-agent, Disallow, Allow and Sitemap. The format, also known as the robots exclusion protocol, was formalised as RFC 9309 in 2022. The one thing to remember: a robots.txt file controls crawling, not indexing.<\/p>\n\n\n\n
That distinction is where most shop owners go wrong. They add a Disallow rule for a page they want out of Google and assume the job is done. In reality, a disallowed URL can still be indexed if other pages link to it; Google shows the bare URL with “No information is available for this page”. To remove a page from the index it needs a noindex meta tag, and Google must be allowed to crawl it to read that tag. Blocking and noindexing at once never works: the crawler never sees the noindex.<\/p>\n\n\n\n
Webshops generate far more URLs than they have products. Faceted navigation and filter parameters such as ?color= and ?orderby= can turn 2,000 products into hundreds of thousands of crawlable URLs. Add internal search results, cart, checkout and account pages, and a crawler spends most of its visit on pages that will never earn a click. A well-written robots txt steers that crawl budget back towards category and product pages, where the revenue is. On larger shops I regularly see new products take weeks to appear in Google for exactly this reason.<\/p>\n\n\n\n
The file also matters because it is so easy to get catastrophically wrong. A single line, Disallow: \/, tells every crawler to stay away from the entire site. That line is standard on staging environments and it is the classic shop migration mistake: the site goes live, the robots.txt comes with it, and organic traffic falls off a cliff within days. Blocking CSS or JavaScript is nearly as bad, because Google can no longer render your pages properly. These are the failures I check first in any technische SEO<\/a> audit.<\/p>\n\n\n\n Start simple: open yourdomain.nl\/robots.txt in a browser and read it. Then go to Google Search Console, where the robots.txt report shows which versions Google fetched per host, when, and whether it hit errors. URL inspection tells you for any single page whether crawling is allowed and which rule blocked it, and the Pages report lists everything under “Blocked by robots.txt” and “Indexed, though blocked by robots.txt”.<\/p>\n\n\n\n Next I run Screaming Frog, which respects robots.txt by default and lists every URL it could not fetch under its “Blocked by robots.txt” filter. I compare that list with the XML-sitemap<\/a> and with organic landing pages in GA4 to make sure nothing valuable is blocked. Server logs close the loop: they show what Googlebot actually requested, which is the only way to know whether the crawler is still hammering filter URLs.<\/p>\n\n\n\n Launching with the staging file.<\/strong> The site was built behind Disallow: \/, the developer copied everything to production, and nobody checked. I have seen it on WordPress, PrestaShop and custom builds, which is why a robots.txt check is on every migration<\/a> checklist I use.<\/p>\n\n\n\n Using Disallow to remove pages from Google.<\/strong> Blocking prevents crawling, not indexing. Discontinued products and old category pages with backlinks keep appearing in results, now without a title or description. Use noindex, or a 410 if the page is gone for good, and let Google crawl it.<\/p>\n\n\n\n Blocking CSS, JavaScript or image folders.<\/strong> Years ago it was common to disallow \/wp-includes\/ or \/themes\/ to “save” crawl budget. Today that breaks rendering and hides product images from Google Images. Only block what Google does not need.<\/p>\n\n\n\n Forgetting that each host needs its own file.<\/strong> A robots.txt on www.example.nl does nothing for the non-www version, the http version or a separate shop subdomain. On headless setups I often find an asset or API host wide open or fully blocked by accident.<\/p>\n\n\n\n I treat the file as a crawl budget decision, not a technical formality. I start with a Screaming Frog crawl and a sample of server logs to see which URL patterns eat the crawl, then map those against what actually earns clicks in Search Console. From that I write a proposed robots.txt with a short justification per rule: which filter parameters to block, which to leave open because they rank, and which paths (search, cart, checkout, account) should never be crawled. Every proposal is tested against a list of important URLs before it goes live.<\/p>\n\n\n\nHow to check your robots.txt<\/h2>\n\n\n\n
Common robots.txt mistakes<\/h2>\n\n\n\n
How I handle robots.txt for clients<\/h2>\n\n\n\n