What it is
robots.txt is a small file that lives at yoursite.com/robots.txt. It lists rules that tell specific bots which paths they can crawl and which to leave alone. It is a request, not a lock: well-behaved crawlers respect it, but it does not physically block access or hide a page from someone who has the URL.
Why it matters
robots.txt has become an AI-era decision, not just an SEO housekeeping file. AI companies run crawlers like GPTBot, ClaudeBot, and PerplexityBot, and you can allow or disallow each one here. That puts a real choice in front of marketers: let AI systems read your content so they can cite you, or block them to protect it. Block a crawler and you also remove yourself from the answers that crawler helps generate.
What to do
Open your robots.txt and read it before changing anything, because an overly broad disallow can hide pages you want found. Decide which AI crawlers you want to allow, and keep your XML sitemap referenced in the file. If your goal is to be cited by AI tools, make sure you are not quietly blocking the bots that feed them. Run WAIO Engine to check how accessible your site is to machines.