SEO Suite Component
Robots.txt Editor
Edit crawler access rules carefully and understand why robots.txt is different from noindex.
AdvancedComponent 4.0.0Difficulty: Advanced
Edit and validate robots.txt crawl rules without confusing crawl access with indexing.
What does robots.txt do?
It gives crawler instructions about which URL paths they may crawl. It is not the same as noindex and it is not a security/access-control file.
Recommended beginner setting
Advanced tool — change only with a reasonIf your existing robots.txt is working and you do not have a specific crawl problem, leave it alone. Use Indexing Manager for normal noindex decisions.
Before editing
- Copy/save your current robots.txt content.
- Write down the exact crawler/path problem you are trying to solve.
- Do not block product/category resources merely to reduce indexing; that can prevent crawlers from seeing page-level directives/content.
- Use the component validation tools.
- After publishing, fetch the public /robots.txt and verify it contains exactly what you intended.
Common mistakes
- Using robots.txt as a noindex replacement.
- Blocking CSS/JS or image resources without understanding the effect.
- Blocking the whole store with a broad Disallow rule.
- Copying a robots.txt file from another ecommerce platform/store.
- Assuming robots.txt protects private/customer data.
Explore HuntBee Suites