Indexing Manager
Indexing Manager tells search engines which low-value or duplicate pages should stay out of search results. Normal public catalog pages remain indexable by default.
- 1Leave normal catalog pages indexable; add a rule only for a real low-value or duplicate URL.
- 2Open the component's Rule Tester for the target route, store and applicable language.
- 3Check the target public page source for
name="robots"if a noindex rule should apply.
The rule tester matches your intention and the rendered page sends the expected robots directive. A missing robots meta tag usually means normal index/follow.
Remember: Robots directives do not immediately remove existing Google results. Frontend verification guide →
The most important thing to know
A normal page does not need an explicit <meta name="robots" content="index,follow">. If no restrictive robots tag is present, search engines already treat the page as indexable and follow its links.
Why is there no Store dropdown?
The standard indexing policies are global because merchants almost always want the same safety rules across every store: do not index checkout, account, search and duplicate sort/filter variants. HuntBee therefore keeps the normal policies global.
Custom rules can still target All Stores or an individual store when a real exception exists.
Built-in behaviour
| Page type | Default behaviour | Reason |
|---|---|---|
| Homepage | Index, Follow (no special tag required) | Main public page. |
| Products | Index, Follow | Core catalog pages. |
| Categories | Index, Follow | Important browsing/landing pages. |
| Manufacturers | Index, Follow | Useful public brand pages. |
| Information pages | Index, Follow | Normal public content. |
| Search/tag results | Noindex | Internal result pages can create many weak duplicates. |
| Account pages | Noindex | Private/utility pages are not search landing pages. |
| Cart and checkout | Noindex | Transactional utility pages. |
| Product compare | Noindex | Temporary/duplicate utility output. |
| Sort/order/filter/limit variants | Noindex where configured | Prevents many near-duplicate category/manufacturer URLs. |
Recommended settings
| Policy | Recommended |
|---|---|
| Noindex search pages | On |
| Noindex account/utility pages | On |
| Noindex checkout/cart pages | On |
| Noindex compare | On |
| Noindex sort/filter variants | On for normal ecommerce stores |
| Custom rules | None unless you have a specific exception |
What are Custom Rules for?
Custom Rules are exceptions. Use them when a custom extension or special route should behave differently from HuntBee's standard policies.
| Example | Possible custom rule |
|---|---|
| A custom internal results page that should never appear in search | Noindex that route. |
| A public campaign/content route that is incorrectly caught by another rule | Explicitly allow normal indexing if appropriate. |
| A rule needed only on one secondary store | Choose that store instead of All Stores. |
Rule Tester
The Rule Tester lets you enter a route/URL context and see which policy would apply. Store selection exists here because you may want to test a store-specific custom exception even though the standard policies are global.
Indexing vs Robots.txt
These are different tools:
- Indexing Manager / noindex tells a search engine “you may crawl this page, but do not keep it in search results.”
- Robots.txt can stop crawling. If a crawler cannot fetch the page, it may not see the noindex instruction.
Indexing vs Canonical
Canonical says “these URLs are similar; this is the preferred one.” Noindex says “do not include this page in search results.” They solve different problems. Do not add both randomly to the same page.
Troubleshooting
- A product has no robots tag: normal and correct; that means default index/follow behaviour.
- A search/filter page is appearing in Google: use Rule Tester, confirm the URL matches the intended built-in policy, and allow time for recrawling.
- One store needs a special rule: create a custom rule scoped only to that store.
- You changed a custom rule and results are unexpected: remove the custom rule first and confirm built-in behaviour before adding another override.