Robots.txt untuk AI: Kesalahan yang Bisa Memblokir Search tapi Tidak Training

Satu baris bisa membuat tim merasa sudah mengambil keputusan besar.

`User-agent: GPTBot`
`Disallow: /`

“Done. OpenAI kita block.”

Belum tentu.

Yang diblok baru crawler dengan purpose tertentu.

Kalau Search masih memakai user-agent lain, Search bisa tetap berjalan.

Kalau user-triggered access memakai jalur lain, request bisa tetap datang.

Kalau URL sudah diketahui dari source lain, link tertentu bisa tetap muncul dalam konteks tertentu.

Dan kalau tujuan awal perusahaan sebenarnya hanya “jangan dipakai training”, rule yang terlalu luas justru bisa memblok Search yang masih diinginkan.

Inilah masalah utama robots.txt untuk AI pada 2026: orang sering menulis rule berdasarkan nama vendor, bukan berdasarkan purpose.

Search.

Training.

Ads.

User-triggered access.

Classic indexing.

Semua dianggap satu kategori “AI crawler”.

Padahal OpenAI sendiri secara resmi memisahkan OAI-SearchBot untuk Search dan GPTBot untuk content yang dapat digunakan dalam training foundation models.

Setting keduanya independen.

Jadi satu rule yang salah bisa menghilangkan Search visibility tanpa mencapai policy training yang sebenarnya dimaksud.

Atau kebalikannya.

Search Tetap Jalan walau GPTBot Diblok

Ini scenario paling penting.

Company policy:

“Kami tidak ingin content dipakai untuk training model.”

Developer menambahkan:

User-agent: GPTBot
Disallow: /

Lalu marketing melihat brand masih muncul di ChatGPT Search.

Panik:

“OpenAI nggak hormat robots?”

Tidak otomatis.

Karena GPTBot bukan OAI-SearchBot.

OpenAI mendokumentasikan GPTBot untuk training crawl.

OAI-SearchBot untuk Search.

Jadi jika SearchBot masih diizinkan, content tetap bisa eligible untuk Search.

Rule tersebut justru bekerja sesuai purpose.

Kesalahan terjadi di interpretasi manusia.

Training block tidak sama dengan Search block.

Search block tidak sama dengan training block.

Kalau policy ingin dua-duanya:
atur dua-duanya.

Kalau hanya salah satu:
jangan wildcard.

Satu `User-agent: *` Bisa Menghapus Nuance

Misalnya legal berkata:

“Block AI training.”

Developer menulis:

User-agent: *
Disallow: /

Itu bukan “block training.”

Itu “block semua crawler yang menghormati wildcard untuk seluruh site.”

Googlebot ikut.

OAI-SearchBot ikut.

Bingbot bisa ikut.

Crawler lain ikut.

Search dapat terdampak luas.

Ini contoh classic collateral damage.

Business intent:
training opt-out.

Technical implementation:
global crawl shutdown.

Tidak match.

Makanya crawler access matrix harus dibuat dulu.

Siapa crawler?

Apa purpose?

Apa business decision?

Baru syntax.

Jangan mulai dari file.

Kesalahan Kebalikan: Allow Semua karena Ingin Search

Marketing ingin ChatGPT Search citation.

Developer:

User-agent: *
Allow: /

Sekarang GPTBot juga tidak diblok oleh robots.

Kalau legal sebenarnya ingin training opt-out, policy bocor.

Again:
Search goal diterjemahkan menjadi blanket allow.

Lebih tepat:

OAI-SearchBot:
allow.

GPTBot:
disallow.

Lalu Googlebot dan crawler lain punya policy sendiri.

Precise.

No heroics.

Robots file seharusnya boring karena policy sudah jelas.

Google Mengajarkan Distinction yang Sama antara Crawl dan Index

Google Search Central sangat jelas:

Kalau ingin mencegah Googlebot crawling:
robots.txt.

Kalau ingin mencegah page diindex:
gunakan `noindex`.

Kalau page diblok robots.txt, Googlebot tidak bisa melihat meta `noindex`.

Ini principle penting.

Robots:
access to crawl.

Noindex:
indexing directive.

Jadi rule:

Disallow: /page/
plus meta noindex di page

bisa gagal mencapai tujuan noindex karena crawler tidak dapat membaca tag tersebut.

Google bahkan menjelaskan bahwa page yang diblok robots.txt masih mungkin muncul sebagai URL di Search dalam certain circumstances tanpa snippet, karena URL bisa diketahui dari links.

Robots bukan “delete from Search” control.

Itu sebabnya article 282 membahas kapan noindex lebih tepat.

Untuk AI Search, jangan otomatis menganggap semua platform identik, tetapi konsep crawl vs index/retrieval tetap harus dipisahkan berdasarkan official docs tiap platform.

OpenAI juga memiliki distinction antara SearchBot crawl controls dan noindex guidance untuk certain Search/Atlas contexts.

Jangan Memblok OAI-SearchBot Kalau Tujuan Anda Citation

Kalau business goal:

“Brand ingin bisa ditemukan dan cited di ChatGPT Search.”

Lalu robots:

OAI-SearchBot:
Disallow: /

Itu kontradiksi.

OpenAI publisher guidance mengatakan site yang opt out OAI-SearchBot tidak akan ditampilkan dalam ChatGPT Search answers, walaupun navigational link/title dapat muncul pada kondisi tertentu jika URL diketahui melalui sumber lain.

Jadi Search crawler access adalah prerequisite untuk normal Search inclusion.

Not guarantee of citation.

But if you explicitly block it, jangan heran Search answer tidak menggunakan page.

Ini bukan ranking problem.

Ini crawl policy problem.

Audit first.

Content optimization belakangan.

Jangan Memblok GPTBot kalau Tujuan Anda Justru Open Training

Ada organisasi open-source/open-knowledge yang mungkin ingin mengizinkan training.

Research repository.

Documentation.

Public-domain archive.

Mereka bisa decide:
allow GPTBot.

No moral default.

No universal recommendation.

But if policy says open:
robots should reflect.

If security inherited old template with GPTBot block:
intent mismatch.

Crawler policy is business choice.

Not SEO fashion.

Document.

Kesalahan User-Agent

Rule:
`User-agent: OpenAI`

Apakah itu documented token?

No.

Use exact official user-agent token:

OAI-SearchBot.

GPTBot.

OpenAI documentation publishes examples and notes version numbers in full user-agent strings may change.

So match token.

Don't build rule against complete version string that will age.

Also don't invent:
ChatGPTBot.
OpenAIBot.
AI-SearchBot.

Unless documented by specific platform.

Third-party blogs often use shorthand.

robots parser cares actual token.

Always check official docs.

Rule Order and Grouping Can Trip Teams

Robots Exclusion Protocol has grouping/matching semantics.

A line may not work like firewall order.

Google documents how it interprets user-agent groups and path matching.

Other compliant crawlers generally follow REP behavior, but implementation details can differ.

Don't assume:

“Last rule wins.”

Test.

Use simple groups.

Avoid enormous pattern soup.

Example:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Simple.

If specific folders:
make explicit.

Don't mix 100 agents in one giant block unless governance mature.

Complexity creates mistakes.

Robots.txt `noindex` Is Not a Valid Google Index Control

Some old robots templates use:

Noindex: /private/

Google stopped supporting unofficial `noindex` rules in robots.txt years ago.

Use meta robots `noindex` or X-Robots-Tag for Google indexing control.

This remains important because AI-era “robots generators” can resurrect old unsupported directives.

Do not copy.

Use official Search Central docs.

If platform has its own directive:
verify.

Do not assume unofficial syntax understood.

A robots file can look sophisticated while doing nothing.

Search loss and false confidence can happen together.

Assets Can Be Accidentally Blocked

Modern page:

HTML shell.

JavaScript fetches content.

CSS.

Images.

API.

If robots blocks `/assets/` or `/api/` broadly, crawler rendering can be incomplete.

Google Search Central troubleshooting notes blocked resources can prevent rendering and even contribute to soft-404-like interpretation if page renders blank/nearly blank or error.

For AI Search crawlers, exact rendering stack varies.

But general web principle:

Don't block resources required to understand public page unless you know impact.

A developer may think:
“API internal.”

Actually endpoint serves article body.

Robots rule accidentally removes content from crawler.

Test rendered output.

Not path name.

“assets” doesn't mean irrelevant.

SearchBot Allowed, WAF Blocked

Another false diagnosis.

robots:

OAI-SearchBot:
Allow: /

SEO:
“Allowed.”

But CDN:
Bot Fight Mode challenges.

WAF:
403.

IP deny.

Geo-block.

Result:
SearchBot cannot fetch.

Team blames robots.

No.

Robots is permission.

Network is transport.

Both must align.

Same for Googlebot.

Cloudflare docs explicitly provide mechanisms to allow verified bots through WAF rules.

If verified bot classification isn't enabled/recognized or custom rule terminates earlier, bot can still be challenged.

Audit status codes.

Don't assume `Allow` means 200.

Search but Not Training Requires Two-Layer Verification

Policy:

OAI-SearchBot allow.
GPTBot disallow.

Test:

1. Fetch robots.
2. Parse relevant groups.
3. Check CDN/WAF.
4. Verify logs.
5. Confirm SearchBot gets 2xx/3xx.
6. Confirm GPTBot respects disallow on target paths.
7. Check no wildcard conflicts.
8. Record date.

OpenAI says robots changes can take approximately 24 hours to be reflected for Search.

So test over appropriate window.

Don't panic in five minutes.

Logs plus time.

Training Block but Search Link Still Appears

As noted, URL can be discovered from other sources.

A block doesn't erase links from internet.

If page appears as a simple navigational link:
not automatically proof content was crawled contrary to policy.

Need inspect answer.

Was page cited with content?

Was only URL/title listed?

What was source path?

OpenAI docs include nuance around third-party search providers/known URLs in Atlas.

Don't make compliance allegation from screenshot alone.

For serious case:
collect logs.
robots version.
IP verification.
timestamps.
answer evidence.
Then investigate.

Evidence discipline.

Noindex vs Disallow

Preview article 282.

Goal:
Don't show page in Search.

Use noindex if platform supports and crawler can access.

Goal:
Don't let crawler fetch content.

Use robots disallow.

Goal:
Don't allow private access.

Authentication.

Goal:
Don't allow training crawler.

Use vendor-documented training crawler control.

Goal:
Don't allow Search crawler.

Search crawler robots.

One page can need several controls.

But don't combine reflexively.

Each control has consequence.

Misconfiguration happens when one tool is asked to do all jobs.

Robots Is Not Model Memory Control

A GPTBot disallow cannot be described as:

“AI will forget us.”

No.

It controls documented crawler access/training preference going forward.

Existing model knowledge.

Third-party sources.

User-provided content.

Search retrieval.

These are different.

Likewise OAI-SearchBot block doesn't delete brand mentions already known through model priors.

It controls Search crawler inclusion.

Marketing expectation should be calibrated.

Robots governs access.

Not cognition.

Publisher Policy Example

Illustrative.

Public articles:
Googlebot allow.
OAI-SearchBot allow.
GPTBot disallow.

Paid member:
auth required.
robots disallow as secondary.

Admin:
auth/firewall.

Ad landing:
OAI-AdsBot allow if campaign.

User-triggered:
public page normal access.

This policy could make sense for a publisher that wants Search visibility but not training.

Another publisher might choose differently.

No universal default.

The point:
each line maps to intention.

Don't copy example blindly.

Audit Common Errors

1. `User-agent: *` accidentally disallows everything.

2. GPTBot rule assumed to control OAI-SearchBot.

3. OAI-SearchBot blocked while business wants ChatGPT Search.

4. Full version user-agent matched instead of token.

5. Old unofficial `Noindex:` in robots.

6. Critical JS/API resources blocked.

7. WAF still blocks allowed bot.

8. Staging robots copied production.

9. Path patterns too broad.

10. Training policy documented but technical rule missing.

11. Search bot allowed but page `noindex`.

12. Robots change not versioned.

Each has different fix.

No single “AI robots template.”

Robots Comment Can Preserve Intent

Use comments:

# Allow ChatGPT Search
# Training opt-out

Comments don't control crawler.

They help humans.

Six months later:
developer sees why.

If policy changes:
comments updated.

Version control.

Pull request.

Review.

Small file deserves context.

Because consequence large.

Jangan Membuat Robots.txt dari Fear atau FOMO

Fear:
“Block all AI.”

FOMO:
“Allow all AI supaya citation.”

Both oversimplify.

Ask:

What content?

What crawler?

What purpose?

What business value?

What legal position?

What security risk?

What measurement?

Then decide.

Search may matter.

Training may not.

Ads may be irrelevant.

User-triggered access may be useful.

Unknown bots may be blocked.

This is strategy.

The biggest robots.txt mistake in AI era isn't syntax.

It's failing to distinguish why a machine is visiting.

Once purpose is clear, Search and training stop being one switch.

And a one-line rule stops accidentally making two different business decisions.