Take it step by step
- Choose the intended user-agent and test exact sample paths.
- Review broad rules such as Disallow: / before publishing.
- Keep required public page resources crawlable.
- Use appropriate indexing controls and authentication for their separate purposes.
A situation to try.
A robots rule can stop crawling a page without reliably removing a known URL from search. Do not put confidential content behind a robots rule alone.
Use the tool’s synthetic example first where available. The example here explains a decision; it is not a reported benchmark result.
Before you call it done.
- Test public and private example paths.
- Check accidental whole-site blocks.
- Review the deployed robots.txt, not only an editor preview.
Know the limits.
Crawler interpretations can vary. A local tester cannot certify every crawler’s behavior.
This tool’s current boundary: Review the output before using it. Your input remains on this device.
Evidence and scope
Technical source guidance and task-specific output checks. These do not establish that every file, browser or device passes.
Use the checks above on your own exported result. A source explains the format or mechanism; it is not proof that this export preserved every feature.
Technical reference: Google: robots.txt limitations ↗
Choose the right tool for this step.
Robots editor and tester
Write crawler rules and test supplied example paths.
On your deviceWhy use it: Test supplied rules against example paths.
Before you start: Not every crawler interprets rules identically.
Sitemap generator and validator
Generate sitemap XML from your canonical URL list.
On your deviceWhy use it: List the public canonical pages.
Before you start: Remove private and noindex pages first.
Try Robots editor and tester ↗
Preparation tools process locally. Official service links take you to the authority’s website. Inspect any exported copy before sharing it.