Search and indexing

Edit robots.txt safely: check, test and undo

Use the robots.txt editor that reads your file the way Google does, tests it against your home page, sitemap and latest posts, and keeps the previous version.

robots.txt tells crawlers which parts of your site to skip. A wrong line can hide your whole site from Google, so the editor checks every change before saving it. Nothing changes until you press Check and save.

Open it

Go to OnlyAI SEO, Settings, robots.txt. You see the file WordPress serves today. The editor is off until you save your own text, and Use the WordPress default switches it off again.

Make a change

1. Edit the text. (Load the WordPress default gives you a starting point.)
2. Press Check only to see the findings without saving, or Check and save.
3. Read the list. Fix what it names.

What the check does

The file is read the way Google reads it (the most specific crawler group applies, the longest matching rule wins, Allow beats Disallow when they tie, and * and $ work in paths) and tested against your home page, your sitemap, your latest posts, your theme stylesheet, a WordPress script, your uploaded images and admin-ajax.php.

• Refused: a rule before any User-agent line, a misspelt directive such as Dissallow, a line that is not a rule, a Sitemap address that is not a full address, an empty file, or a file over 500 KB.
• Needs your confirmation: a file that would block your home page, sitemap or latest posts for Google or Bing. The list names each address and the rule. Tick the box only on a site you really want hidden, such as a staging copy.
• Warned: blocked CSS, scripts or images, a blocked admin-ajax.php, Crawl-delay and Noindex: (Google ignores both), and paths that do not start with a slash.

Test an address

Under Test an address, type a path or full address, pick Googlebot, Bingbot, GPTBot or *, and press Test. It says whether it is allowed and which line decided.

Undo

Restore the previous version undoes the last save.

Good to know

• The sitemap line and your AI-crawler rules (AI Visibility) are still added after your text, and the check includes them.
• A real `robots.txt` file in your site folder overrides the editor; the card tells you. Delete the file to use the editor.
• If Discourage search engines is ticked in Settings, Reading, WordPress serves a file that blocks everything whatever you write.