Your website went up years ago. Later, someone added a security plugin because the quote form kept filling with spam.
The site still loads and customers still call. Neither the site nor the plugin was built with ChatGPT in mind, and either could have turned its search crawler away. Nothing on your screen would tell you. For the bigger picture of how AI assistants choose local companies, see our guide to AI search for home service companies.
A crawler is a program that visits web pages automatically, so search engines and AI services can read them.[1] A small public file called robots.txt tells crawlers which pages they may visit.[2] Respectable crawlers obey it, and nothing forces the rest to.[2]
Which OpenAI crawlers matter, and which one decides search?
OpenAI's crawler page names four, and OAI-SearchBot is the one search depends on.[3] Three matter to you, and OpenAI says this about each:
- OAI-SearchBot is the search crawler. A site that opts out of it won't be shown in ChatGPT's search answers, though a plain link can still appear.
- GPTBot is the training crawler. It collects content that may be used to train OpenAI's models. Disallowing it signals you want yours left out.
- ChatGPT-User is the visit that happens when someone asks ChatGPT a question and it opens a page. Robots.txt rules may not apply to it, and it doesn't decide what appears in search.
The fourth, OAI-AdsBot, only checks pages submitted as ChatGPT ads. OpenAI says each bot's setting is independent of the others, and its own example allows OAI-SearchBot while disallowing GPTBot.[3] To appear in ChatGPT's search, its help page says to let OAI-SearchBot crawl your site.[4]
How do you check that your website allows ChatGPT to crawl it?
Type your address into a browser and add /robots.txt on the end. You'll see plain text.
Look for two things. One is a blanket block: "User-agent: *" with "Disallow: /" under it, which tells every crawler, Google's included, to keep out of the whole site.[5] The other is a group that names OAI-SearchBot and disallows it.
A healthy file is usually short and dull. If you get a "not found" page, there's no file and so no rules. Google's documentation says it treats that as no restrictions.[6]
A robots.txt with three groups. Read it, don't paste it.
- User-agent: *
- Disallow: /admin/
- User-agent: GPTBot
- Disallow: /
- User-agent: OAI-SearchBot
- Disallow: /
- Sitemap: https://www.example.com/sitemap.xml
Illustrative example
In that file, the first group is routine, GPTBot's is a decision about training, and OAI-SearchBot's is the one that turns away search. Copy the file's text into a note, then delete that group or ask whoever runs the site to. OpenAI says its search systems can take about a day to notice.[3]
Where else can a block hide?
In a CDN or firewall (services that sit in front of your site and filter who gets in), your host's or site builder's settings, or a security plugin. Google's advice for its AI features includes checking that crawling is allowed by any CDN or hosting setup, not only in robots.txt.[7] Look for a setting called something like "AI bots" or "AI crawlers", and read what it lists before changing it.
Take Cloudflare, a CDN. In July 2025 it said it would block AI crawlers by default and ask each new domain whether to allow them.[8] From September 15, 2026, new sites without ads are offered a preset with Search, Training and Agent all on Allow.[9]
Setting Training to Block now also stops Googlebot, which crawls for both search and training. To refuse training and keep search, Cloudflare points to Disallow AI Training.[9] It files OAI-SearchBot under Search, so check that Search says Allow.[10]
On Squarespace, Settings then Crawlers has a box called "Block known artificial intelligence crawlers". Squarespace says to leave it unticked if you want AI crawlers to scan your site.[11]
If someone else runs your site, ask them to look for OAI-SearchBot in the server logs. Then ask whether OpenAI's published IP addresses are allowed, as OpenAI recommends.[3]
Should you let the training crawler in?
That's your call, and either answer is reasonable. Let GPTBot in, and OpenAI may use your pages to train its models. Keep it out, and you're telling OpenAI you don't want them used that way.
Neither choice changes the search setting. Google has its own separate switch, Google-Extended. Google says it covers Gemini training and some Gemini answers, and doesn't affect your place in Google Search.[12]
Which robots.txt mistakes should you avoid?
- Opening the site to every bot because a blog said to. Decide by purpose, one crawler at a time.
- Blocking Googlebot. A page has to be indexed and eligible in Search to be linked in AI Overviews or AI Mode, Google says.[7] Our guide to showing up in Google AI Overviews covers the rest.
- Pasting a snippet you don't understand. One "Disallow: /" under the wrong name shuts out the crawler you wanted.
- Editing without a copy of the old file, or in a word processor, which can add curly quotes.[5]
Is unblocking enough to get you named?
No. It lets ChatGPT's search read your pages, which is the minimum. An llms.txt file doesn't change that. Whether it names you depends on what it finds: a site that says plainly what you do and where, a complete Google profile, matching listings, reviews and mentions on other sites. Our piece on why ChatGPT recommends competitors and not you goes through them, and our AI search visibility audit checks whether AI crawlers can reach your site. To see what the assistants say about you today, run an AI search audit yourself.
Where this check stops helping
Reading the file can't show whether a firewall turned a crawler away. Check again after a redesign, a new host or a new security plugin. And nobody outside OpenAI can tell you how it decides who to name.