Dari SearchBot ke AdsBot: OpenAI Mulai Memisahkan Fungsi Crawler Lebih Jelas

Ada masa ketika tim web cukup punya satu pertanyaan:

“Bot OpenAI kita allow atau block?”

Pada 2026, pertanyaan itu sudah terlalu sederhana.

OpenAI sekarang mendokumentasikan beberapa user-agent dengan fungsi yang makin jelas dibedakan.

OAI-SearchBot untuk ChatGPT Search.

GPTBot untuk potential training foundation models.

ChatGPT-User untuk certain user-triggered actions.

OAI-AdsBot untuk validasi landing page ChatGPT Ads.

Perubahan ini penting bukan hanya buat engineer.

Ia mengubah cara Legal, Marketing, Security, Publisher, dan Ads team harus membicarakan crawler policy.

Satu perusahaan bisa ingin empat hal sekaligus:

muncul di ChatGPT Search,

tidak mengizinkan potential training crawl,

mendukung user-triggered action,

dan menjalankan ChatGPT Ads.

Dulu semua itu sering dibungkus dalam satu label “AI bot”.

Sekarang OpenAI sendiri membuat fungsi crawler lebih granular.

Itu justru kabar baik.

Granularity memberi kontrol.

Search bukan training

Ini distinction yang paling strategis.

OAI-SearchBot digunakan untuk membantu website tampil dalam ChatGPT Search.

GPTBot digunakan untuk crawling content yang dapat digunakan dalam potential foundation-model training.

OpenAI menyatakan setting keduanya independen.

Artinya publisher bisa:

Allow OAI-SearchBot.

Disallow GPTBot.

Website tetap bisa eligible untuk ChatGPT Search tanpa harus mengizinkan GPTBot.

Ini menyelesaikan salah satu konflik paling klasik antara tim Growth dan Legal.

Growth ingin Search visibility.

Legal tidak ingin training.

Mereka tidak perlu memilih salah satu secara absolut.

Crawler separation memungkinkan policy yang lebih presisi.

Tentu policy organisasi tetap harus mempertimbangkan kontrak, lisensi, dan jurisdiction.

Tetapi setidaknya level crawler tidak memaksa bundling.

Ads juga bukan Search

OAI-AdsBot memberi contoh yang lebih jelas lagi.

OpenAI menggunakan OAI-AdsBot untuk mengunjungi landing page yang diajukan sebagai iklan di ChatGPT, termasuk untuk validation, safety review, dan context yang membantu menentukan relevansi iklan.

Kalau brand menjalankan ChatGPT Ads, OAI-AdsBot perlu bisa mencapai landing page.

Tetapi allow AdsBot tidak berarti page otomatis menjadi source organic Search.

Organic Search memakai OAI-SearchBot.

Ads validation memakai OAI-AdsBot.

Satu page bisa membutuhkan keduanya.

Atau hanya satu.

Ini mirip perusahaan punya dua petugas berbeda.

Satu mengaudit katalog editorial.

Satu memeriksa landing page iklan.

Mereka mungkin melihat URL yang sama, tetapi objective berbeda.

Infrastructure harus berhenti membuat rule berdasarkan brand “OpenAI” dan mulai berdasarkan function.

ChatGPT-User menambah kategori ketiga

ChatGPT-User bukan crawler Search.

OpenAI menjelaskan user-agent ini digunakan untuk certain user-triggered actions di ChatGPT dan Custom GPTs.

Karena action dipicu user, robots.txt rules may not apply dengan cara yang sama seperti automatic crawling.

Ini membuat model crawler governance makin menarik.

Sekarang ada setidaknya:

automatic Search crawl,

training crawl,

ad validation crawl,

user-triggered fetch/action.

Website security tidak boleh menganggap semuanya satu traffic class.

Misalnya ChatGPT-User mencoba membuka public page atas permintaan user.

Itu bukan evidence SearchBot sedang indexing.

Dan allow OAI-SearchBot bukan alasan memberi ChatGPT-User bypass authentication.

User-triggered action tetap harus tunduk pada authorization, session, CSRF protection, rate limiting, dan normal application security.

Crawler identity menjelaskan fungsi.

Bukan permission universal.

Kenapa separation ini muncul?

Dari luar, kita bisa melihat OpenAI ecosystem makin punya banyak surfaces.

Search.

Browser/agent.

Ads.

Commerce.

Training.

Setiap surface punya requirement sendiri.

Secara engineering, masuk akal fungsi crawler tidak dicampur.

Advertiser membutuhkan crawler yang bisa diverifikasi untuk ad review.

Publisher membutuhkan SearchBot.

Training preferences perlu dipisah dari Search distribution.

Agent/user actions perlu jalur yang berbeda dari background crawl.

Ini bukan detail kecil.

Ini evolution dari “AI company crawls web” menjadi “AI platform punya beberapa produk yang masing-masing berinteraksi dengan web dengan tujuan berbeda.”

Untuk website owner, governance harus ikut naik kelas.

Robots.txt sekarang adalah policy matrix

Dulu robots file sering dikelola SEO sendirian.

Sekarang seharusnya ada beberapa stakeholder.

Marketing:

ingin Search visibility?

Publisher:

ingin citation/referral?

Legal:

training policy?

Ads:

ada ChatGPT campaign?

Security:

which bot identities can bypass challenge?

Infrastructure:

WAF/CDN implementation?

Product:

agent/user actions?

Satu file kecil merepresentasikan keputusan lintas fungsi.

Karena itu robots change sebaiknya punya change management.

Tidak perlu birokrasi berat.

Tetapi ada owner.

Reason.

Expected impact.

Rollback.

Review.

Contoh:

Change request:
Allow OAI-AdsBot to `/campaign/`.

Reason:
ChatGPT Ads landing review.

Do not change GPTBot.

Do not change admin paths.

Security verifies IP/verified bot.

Ini jauh lebih aman daripada:

“Please allow OpenAI bots.”

Kalimat kedua terlalu luas.

WAF policy juga harus mengikuti fungsi

Robots saja tidak cukup.

OpenAI advertiser guidance menjelaskan crawler bisa gagal karena WAF, CDN, bot mitigation, JavaScript challenges, CAPTCHA, auth, geo restriction, atau rate limit.

Jadi separation user-agent harus diterjemahkan ke infrastructure.

OAI-AdsBot:
allow public ad landing pages.

OAI-SearchBot:
allow intended public Search pages.

GPTBot:
block or allow sesuai training policy.

ChatGPT-User:
normal user-triggered security path.

Do not create one Cloudflare rule:

`if user-agent contains OAI -> skip all security`.

Berbahaya.

UA bisa dipalsukan.

OpenAI publishes crawler IP ranges and recommends verified mechanisms where supported.

Least privilege.

Function-specific.

This is security engineering, not SEO hack.

Logs juga harus dipisahkan

Dashboard:

OAI-SearchBot requests.

GPTBot.

OAI-AdsBot.

ChatGPT-User.

Jangan AI bots total.

Why?

SearchBot 403:
could hurt Search inclusion.

GPTBot 403:
might be intentional.

AdsBot 403:
could block landing review.

ChatGPT-User 403:
maybe normal authorization.

Same status code.

Different meaning.

Policy-aware log interpretation is mandatory.

Otherwise on-call sees “OpenAI crawler failure 30%” and raises incident, even though 30% is intentionally blocked GPTBot.

Observability needs expected state.

Training opt-out no longer harus mematikan distribution

Ini salah satu benefit besar.

Publisher historically worried:

“If we block GPTBot, do we disappear from ChatGPT?”

OpenAI current docs separate SearchBot.

So answer can be:

allow SearchBot, block GPTBot.

Search eligibility remains possible.

This gives publisher more bargaining space around rights.

They can pursue referral.

Citation.

Audience.

Without necessarily granting training crawl.

Again, individual licensing arrangements can differ.

But default crawler governance supports distinction.

This is healthy platform design.

Ads readiness tidak membutuhkan training permission

Same logic.

Advertiser wants campaign.

Need OAI-AdsBot.

May also benefit from OAI-SearchBot.

Does not need to allow GPTBot just because ads.

If agency says:

“Untuk ChatGPT Ads, semua OpenAI bots harus dibuka,”

ask for official basis.

OpenAI says OAI-AdsBot must be allowed and recommends SearchBot.

GPTBot is different.

Don't broaden data-use permission unnecessarily.

The correct question:

What bot is needed for this product function?

This should become procurement checklist.

Bot functions could continue to evolve

OpenAI docs can add/change agents and ranges.

Version numbers can change.

Ads ecosystem is evolving.

Atlas may evolve.

So crawler governance should not be written once and forgotten.

Monthly or quarterly check.

Subscribe official crawler docs update if available.

Compare:

new user-agent?

changed function?

new IP range?

new policy?

Then update.

This is increasingly similar to managing payment gateway IP allowlists or cloud service integrations.

Not a one-time SEO setting.

Third-party bot databases can lag

Security vendors maintain bot categories.

Sometimes current.

Sometimes not.

If official OpenAI docs conflict with old vendor taxonomy, official source should lead policy.

Provider verified bot status is useful, but verify current.

A bot can be new before your WAF recognizes it.

OAI-AdsBot is a good example of why systems need update cadence.

Brand with ads launch should test actual requests.

Not assume category.

A 200 response proves more than a checkbox.

Crawler separation also changes AI audit language

Old audit:

“AI crawlers blocked.”

Too vague.

New audit:

OAI-SearchBot:
Allowed, 98% 200.

GPTBot:
Blocked intentionally.

OAI-AdsBot:
Not required, no ads active.

ChatGPT-User:
Observed, normal auth.

Now executive can understand.

This is governance maturity.

Same for recommendation.

Don't say:

“OpenAI can crawl.”

Which function?

Search?

Training?

Ads?

User action?

AI Optimization should use exact terms.

Precision prevents bad decisions.

What about public pages that should not be Search sources?

Use SearchBot robots/noindex controls according OpenAI guidance.

What about training?

GPTBot.

What about ad landing?

AdsBot.

What about sensitive page?

Authentication.

Different tool.

This functional separation is better than trying to use robots as universal privacy/security mechanism.

Robots is not security.

Noindex is not confidentiality.

Crawler policy is one layer.

Agentic web forces teams to learn web architecture properly.

Could OpenAI reuse crawl results?

OpenAI notes that when a site allows both OAI-SearchBot and GPTBot, it may use the result from a single crawl for both use cases to avoid duplicate crawling.

This does not erase permission separation.

It means implementation can be efficient when both are allowed.

This nuance matters for logs.

If GPTBot traffic is low, don't infer training opt-out from log count.

Check robots policy.

Data-use permission and raw hit count are not the same.

Again, policy beats guess.

Commerce adds more reasons for clear separation

Product feed can provide structured product information.

OAI-SearchBot can access public product pages.

OAI-AdsBot validates ad destinations.

Agent/browser can help user shop or act.

One merchant may interact with OpenAI through four data paths.

If commerce team doesn't map them, debugging becomes impossible.

Feed accepted but ad rejected:
maybe AdsBot.

SearchBot crawling but product not surfaced:
selection/relevance.

Agent can't checkout:
UX/auth.

GPTBot blocked:
training policy.

Different failure domains.

Do not solve all with “allow bots.”

OpenAI's crawler evolution is a signal about the web itself

As AI platforms move from reading to searching, advertising, and acting, web access becomes purpose-specific.

The old binary:

human vs bot

isn't enough.

Now we have:

human,

search crawler,

training crawler,

ad verifier,

user-directed agent,

commerce feed,

API.

Each has identity, rights, and expected behavior.

This will likely become normal beyond OpenAI.

Organizations need a machine-access policy.

Which automated actors can read what?

For what purpose?

What can act?

What needs confirmation?

What data is public?

This is bigger than robots.txt.

Robots is just one enforcement mechanism.

A practical OpenAI crawler policy in 2026

Document:

OAI-SearchBot:
Purpose: ChatGPT Search.
Desired: allow public content.
Owner: SEO/Publisher + Infra.

GPTBot:
Purpose: potential training crawl.
Desired: company legal decision.
Owner: Legal/Data Governance.

OAI-AdsBot:
Purpose: ads landing validation.
Desired: allow submitted public destinations if ads active.
Owner: Ads + Infra.

ChatGPT-User:
Purpose: user-triggered actions.
Desired: normal application access/security.
Owner: Product/Security.

Review date.

Official docs source.

This one table can prevent dozens of misunderstandings.

In the opening, people asked:

“Bot OpenAI kita allow atau block?”

By end of 2026, that question should sound as incomplete as asking:

“Email kita allow atau block?”

Which email?

Transactional?

Marketing?

Support?

Password reset?

Different function.

OpenAI's move from SearchBot to AdsBot shows crawler identities are becoming product interfaces.

That's not extra complexity for the sake of it.

It gives website owners a better chance to say yes to the use cases they want and no to the ones they don't.

The mature response is not allow all.

Not block all.

It is to know exactly which machine is knocking on the door, why it is there, and whether the business actually wants to let it in.