Crawler access control

Robots.txt Editor

Robots.txt tells compliant crawlers which URL areas they may or may not crawl. It is not the same as telling a search engine whether a page should appear in search results.

Try it on your storeThree steps to check this feature

1Open Robots.txt Editor and review existing Disallow directives before changing anything.
2Save a small safe edit or confirm the current file, then open your-store.com/robots.txt.
3Compare the public text with your saved rules and check the sitemap line if present.

What confirms it works

Your public robots.txt is reachable as plain text and contains the intended rules.

Remember: Do not block products or category URLs accidentally; robots.txt is not a substitute for noindex. Frontend verification guide →

Leave safe defaults

Recommended rule

Do not add robots rules just because a URL looks unimportant. Use Indexing Manager for “do not show this page in search”; use Robots.txt only when you intentionally want to control crawler access.

Robots.txt vs Noindex

Tool	Meaning	Typical use
Robots.txt Disallow	“Crawler, do not crawl this URL/path.”	Access control for crawl areas where blocking is appropriate.
Noindex	“You may crawl this page, but do not keep it in search results.”	Search, account, checkout and duplicate utility pages.

Important: if you block a URL in robots.txt, a crawler may not be able to fetch the page and see its noindex/canonical tag. Do not blindly use both.

Recommended setup

Keep HuntBee/OpenCart's safe default robots rules.
Do not block products, categories or sitemaps you want search engines to crawl.
Use Indexing Manager for normal noindex policies.
Only add custom Disallow rules for a known crawl-control reason.
After editing, open /robots.txt publicly and verify the output.

What should normally stay crawlable?

Homepage
Products
Categories
Manufacturers/brands
Public information/content pages
XML Sitemap URLs
Assets required for search engines to understand/render pages, where applicable

What should not be put in robots.txt just for SEO?

Every sort/filter URL — Indexing Manager handles common duplicate variants more appropriately.
Old broken URLs — use redirects/404 handling.
Duplicate pages that need a canonical signal.
A page you need search engines to crawl in order to see noindex.

Example

A robots rule is simple text:

User-agent: *
Disallow: /some-private-crawl-area/

Do not copy generic robots files from another website without understanding every blocked path.

How does Sitemap relate to robots.txt?

Many sites include the sitemap location in robots.txt so crawlers can discover it. The Sitemap should contain public URLs you want discovered; robots rules should not contradict that intent.

Common mistakes

Accidentally using Disallow: / on a live store.
Blocking CSS/JS/assets needed for rendering.
Blocking product/category paths that are in the sitemap.
Thinking robots.txt removes a URL from Google immediately.
Using robots.txt instead of redirecting a moved page.

Before saving a major robots change: copy the current content somewhere safe. One incorrect broad rule can affect the whole store.

Troubleshooting

Important pages stopped being crawled: remove the broad Disallow rule and recheck robots.txt.
Noindex page still appears: make sure it is not blocked from crawling before the noindex can be seen.
Sitemap cannot be reached: do not block the sitemap path and check XML Sitemap routing.