OpenAI has printed details about its new net crawler named GPTBot. You’ll be able to learn the documentation on GPTBot over right here.
What’s GPTBot. GPTBot is OpenAI’s net crawler, utilized by OpenAI to crawl the net, devour information for its AI options, resembling ChatGPT, and use that to offer AI-generaterd solutions to your questions.
Useragent. GPTBot’s Person agent token is “GPTBot” and its full user-agent string: is “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; GPTBot/1.0; +https://openai.com/gptbot)”.
Robots.txt. You need to use your robots.txt to dam GPTBot from accessing all or components of your web site. To disallow GPTBot to entry your website you possibly can add the GPTBot to your website’s robots.txt:
Person-agent: GPTBot
Disallow: /To permit GPTBot to entry your solely components of your website you possibly can add the GPTBot token to your website’s robots.txt like this:
Person-agent: GPTBot
Permit: /directory-1/
Disallow: /directory-2/GPTBot IP ranges. OpenAI additionally printed the IP ranges that GPTBot makes use of over right here, it at the moment lists one, however I think they may add extra over time.
Why we care. If you don’t want GPTBot crawling your website and/or utilizing your content material for its functions, then you possibly can disallow GPTBot from crawling your website. This is similar protocol you’d use to dam GoogleBot, BingBot or different net crawlers.
