Crawled, indexed, ranked
Three separate steps stand between a page and a search result. Crawling means a search engine’s software visits the page. Indexing means it decides to store the page and make it eligible to appear. Ranking is where it picks the order for a particular search. Problems can arise at each step, and the fix depends on which one has failed.
If a page is not crawled, the cause is often technical: it cannot be reached, or it is blocked. If it is crawled but not indexed, the engine has seen it and chosen not to keep it, which usually points to quality, duplication or an instruction on the page. If it is indexed but not shown for your target phrase, that is a ranking question rather than an indexing one.
Common reasons pages are not shown
The page is blocked
A site can tell crawlers to stay out of certain areas, using a small text file at the root of the site. A rule meant to hide the checkout or admin area can accidentally cover product pages too. Separately, a page can carry an instruction in its code asking engines not to index it. Stores sometimes launch with a “discourage search engines” setting left on from the build phase. Checking these is the quickest win.
The page has no way in
Crawlers mostly discover pages by following links. A product that is not linked from any category, or that is only reachable by typing a search into the store’s own search box, may never be found. Good site structure and internal linking prevent this. A sitemap file, listing the addresses you want found, also helps, but it is a hint and not a guarantee.
Errors and redirects
Pages that return errors cannot be indexed. Long chains of redirects, where one address leads to another and then another, waste effort and may be abandoned. Loops are worse. After a redesign or a move to a new platform, old addresses should lead directly to their new equivalents in a single step.
A related trap is a “soft” error: a page that tells shoppers the product is gone but still signals success to crawlers. Engines may treat these as errors anyway. If an item is permanently discontinued, either redirect to a suitable replacement or let the page clearly show it is removed.
Duplicate or near-identical pages
When many addresses show almost the same content, engines pick one and may ignore the rest. This is common with options and filters, and is covered in more depth under duplicate content. If the version they choose is not the one you want, indicating a preferred address can help.
Thin or low-value pages
Pages with little original content, such as a product with only a supplier’s one-line description, are sometimes crawled and then left out. Unique, informative text gives engines a reason to keep the page. Empty categories and placeholder pages should be removed or improved.
Content that crawlers cannot read
Some stores load product lists or text with scripts after the page opens. Modern engines can often process such pages, but it is slower and less reliable than plain content in the page’s code. If key text only appears after interaction, such as clicking a tab or a “load more” button, it may be missed.
Slow or unreliable servers
If the site frequently times out or responds very slowly, crawlers may visit less often. Good page speed and stable hosting help here as well as for shoppers.
New or low-authority sites
A brand-new store may simply be waiting. Discovery can take days or weeks, and engines are selective about how many pages they keep from a small site. Links from other sites, covered under backlink building, often speed discovery.
How to spot problems
You do not need expensive tools to begin.
- Search for your own site. Typing a site-restricted query for your domain gives a rough view of what is indexed. The count is approximate, not exact.
- Use the free webmaster tools that search engines provide. After verifying ownership, they report which pages are indexed, which are excluded, and often the stated reason. They also let you inspect one address to see how the engine fetched it.
- Check the page itself. View the page source or use your browser’s tools to look for an instruction not to index, and confirm the page returns a normal response.
- Review the blocking file. Read it line by line and confirm no rule covers pages you want found.
- Compare lists. Export your product list from the store and compare it to the indexed pages. The gaps show where to investigate.
- Look at server logs, if available. They show which pages crawlers actually request.
Fixing in a sensible order
Start with blocks and settings, because they are cheap to fix and can affect everything. Then repair errors and redirects, make sure every important page is linked from somewhere sensible, and improve thin pages. After a fix, ask the engine to re-inspect the key pages, then wait. Results are not instant, and requesting the same thing repeatedly does not speed it up.
Not every excluded page is a problem. Cart, account and internal search pages usually should stay out of results. The aim is to ensure that the pages you want found are available, not to index everything.