DefinitieHow easily search engine bots can access and move through a website's pages by following links. Pages that are blocked, broken or buried cannot be crawled and therefore rarely rank.
Crawlability is how easily search engine bots can access and move through a website’s pages by following links. Googlebot discovers URLs through links and sitemaps and requests each page; a page that is blocked, broken or buried too deep cannot be crawled, and therefore rarely ranks. Crawlability (also called crawl accessibility) is the first of three gates: crawl, then index, then rank.
The common mistake is treating “it is live on the website” as the same thing as “Google can reach it”. Shop owners publish a new category, see it in the menu and assume the work is done. But if the menu renders its links with JavaScript after load, if the URL is disallowed in robots.txt, or if nothing links to it at all, the page is invisible to the crawler even though every visitor can see it.
Why crawlability matters for a webshop
Every product and category page that cannot be crawled is a page that earns nothing from organic search. For a shop with 300 products that is annoying; for a PrestaShop or Lightspeed shop with 20,000 URLs it is a structural revenue problem, because the pages Google skips are often the deep ones (page 4 of a category, a niche product) that carry the long-tail queries with the highest purchase intent. Poor crawlability also delays everything else: a price change or a fixed title tag only shows up in search after Google has recrawled the page.
For larger shops there is a second cost: crawl budget. Google spends a limited amount of time on each site, and if the crawler burns it on thousands of filter combinations, 404s from out-of-stock products and redirect chains, it has less left for the pages you want ranked. Above a few thousand URLs this is one of the first things I check within technische SEO.
How to check crawlability
Google Search Console is the first stop. The Crawl stats report shows how many requests Googlebot makes per day, the share of 5xx responses and whether response times are creeping up. The Pages report is more telling for a shop: “Discovered, currently not indexed” means Google knows the URL but has not crawled it yet, usually a sign of weak internal links or limited crawl budget; “Crawled, currently not indexed” means the bot got there and judged the page not worth indexing.
For the view from the outside I run Screaming Frog over the whole shop. It shows crawl depth, response codes, redirect chains, URLs blocked by robots.txt, and pages that only appear once JavaScript rendering is switched on. The Ahrefs site audit gives a similar picture on a schedule, useful for spotting regressions after a theme update. For large shops I add a server log analysis, because logs show what Googlebot really requested rather than what it could reach in theory.
Common crawlability mistakes
Blocking the wrong things in robots.txt. A disallow rule meant for a cart or search path is widened to a whole folder, and suddenly every product image or category page is off limits. Check your robots.txt after every migration and theme change.
Navigation that only exists in JavaScript. Mega menus, “load more” buttons and infinite scroll that inject links after the page loads leave the bot with an empty HTML skeleton. Google renders JavaScript later and less often, so those links are discovered late or not at all.
Infinite URL spaces. Faceted filters (colour, size, price, brand) and date-based calendars generate URL combinations without end. A Dutch fashion shop with ten filters can expose millions of URLs showing the same products in a different order, and the real category pages wait.
Orphan pages and dead ends. Products that were removed from a category but kept live, landing pages that only exist in an email campaign, and sold-out products that flip to a 404 and back again every week. Pages with no internal links are only found by accident.
How I handle crawlability for clients
I start with a Screaming Frog crawl set to behave like Googlebot, then compare it with the shop’s own product and category export. The difference between the two lists is the first deliverable: which pages exist but are unreachable, which are reachable but blocked, and which are allowed but buried five or more clicks from the homepage.
The output is a prioritised fix list, not a 40-page report: robots.txt rules to change, filter parameters to block or canonicalise, redirect chains to flatten, server errors to raise with the host, and internal linking changes that pull important pages closer to the homepage. I then regenerate the XML-sitemap so it only lists pages that return 200 and may be indexed, and track the Search Console crawl stats for six to eight weeks to confirm the crawler’s behaviour changes. This is usually the first phase of an e-commerce SEO engagement, because content and link building are wasted on pages the bot never sees.
Veelgestelde vragen
Is crawlability the same as indexability?
No. Crawlability is whether the bot can reach and read a page; indexability is whether Google is allowed and willing to store it in its index. A page can be perfectly crawlable and still carry a noindex tag, and vice versa. You need both before ranking is possible.
Does crawl budget matter for a small webshop?
Rarely. Google itself says crawl budget is a concern for sites with many thousands of URLs or very frequent changes. A shop with 500 products should focus on the basics: no accidental robots.txt blocks, links in the HTML, a clean sitemap and fast responses. Once filters and languages push the count into the tens of thousands, it becomes a real constraint.
Should I block filter URLs to improve crawlability?
Usually a mix. Filters nobody searches for (sort order, price ranges, three-way combinations) can be blocked in robots.txt or kept out of the HTML. Filters that match real queries, such as “zwarte sneakers maat 43” for a shoe shop, deserve a clean, crawlable URL with its own title and content. The goal is to let the bot reach the pages that can rank and keep it away from the ones that only duplicate them.
How quickly do crawlability fixes show results?
Googlebot picks up a robots.txt change within a day or two, but the effect on indexing follows the recrawl cycle of the affected pages: days for popular pages, weeks for deep ones. In practice the “Discovered, currently not indexed” count starts to fall within two to four weeks and rankings for the newly reachable pages follow after that.