Cara Memisahkan Crawl Permission, Index Permission, dan Training Permission

Satu website bisa bilang tiga hal yang berbeda kepada machine.

“Boleh ambil halaman ini.”

“Boleh ambil, tapi jangan tampilkan di Search.”

“Boleh tampilkan di Search, tapi jangan gunakan crawler training tertentu.”

Masalahnya, banyak tim masih mencoba menyampaikan ketiga keputusan itu dengan satu file robots.txt.

Akhirnya policy jadi campur.

Training opt-out dianggap noindex.

Noindex dianggap crawl block.

Disallow dianggap privacy.

Search visibility hilang karena legal sebenarnya cuma ingin training dibatasi.

Ini bukan sekadar terminology.

Crawl permission, index permission, dan training permission adalah tiga layer yang punya tujuan berbeda.

Kalau layer-nya tidak dipisahkan, implementation bisa benar secara syntax tetapi salah secara business intent.

Crawl Permission Menjawab: Boleh Fetch atau Tidak?

Crawl permission adalah izin bagi crawler tertentu untuk mengambil URL.

Tool utamanya biasanya robots.txt untuk crawler yang menghormati Robots Exclusion Protocol.

Contoh:

Googlebot boleh crawl public articles.

OAI-SearchBot boleh crawl public articles.

GPTBot tidak boleh crawl.

Crawler unknown mungkin default policy.

Robots.txt mengatur crawler access ke path.

Ia tidak menjamin page akan diindex.

Tidak menjamin page tidak diketahui dari link luar.

Tidak melindungi private data.

Crawl permission adalah traffic/discovery control.

Kalau crawler tidak boleh masuk:
disallow.

Kalau boleh:
allow atau tidak ada rule yang melarang.

Simple.

Tetapi consequence-nya harus dipahami.

Index Permission Menjawab: Boleh Masuk Search Result atau Tidak?

Untuk Google, `noindex` adalah directive yang mengatakan page tidak boleh diindex dan ditampilkan di Search.

Google Search Central menjelaskan bahwa crawler harus dapat mengakses page agar bisa membaca noindex.

Jadi kombinasi:

robots disallow
+
meta noindex

bisa membuat noindex tidak terbaca.

Ini classic error.

Goal:
page tetap public tapi tidak Search.

Correct conceptual approach:
allow crawl.
serve noindex.

Contoh:

thank-you page.

internal search.

certain filter.

utility page.

PDF public yang tidak ingin muncul di Search, melalui X-Robots-Tag jika sesuai.

Index permission adalah Search-serving decision.

Bukan training decision.

Bukan privacy.

Page noindex masih public.

User dengan URL masih bisa buka.

Crawler yang tidak menghormati noindex masih bisa fetch jika public.

Jangan pakai noindex untuk secret.

Training Permission Menjawab: Boleh Digunakan untuk Training oleh Crawler Tertentu?

Ini layer yang paling baru menjadi mainstream.

OpenAI mendokumentasikan GPTBot untuk crawl content yang dapat digunakan dalam training generative AI foundation models.

OpenAI juga mendokumentasikan OAI-SearchBot terpisah untuk Search.

Controls independen.

Publisher dapat:

allow OAI-SearchBot.

disallow GPTBot.

Artinya:

Search discoverability tetap diinginkan.

Training crawl tidak.

Ini bukti jelas bahwa training permission bukan index permission.

A page bisa indexable/searchable tetapi training crawler diblok.

Likewise page bisa noindex namun GPTBot policy perlu ditentukan terpisah.

Jangan assume noindex otomatis training opt-out.

Use documented crawler control.

Tiga Pertanyaan sebelum Menulis Rule

Untuk setiap content class:

1. Boleh crawler fetch?
2. Boleh page tampil di Search?
3. Boleh training crawler mengakses?

Contoh public article:

Crawl:
yes.

Index:
yes.

Training:
no.

Implementation conceptual:

Googlebot allow.

OAI-SearchBot allow.

index allowed.

GPTBot disallow.

Example open dataset:

Crawl:
yes.

Index:
yes.

Training:
yes if licensing permits.

Example thank-you:

Crawl:
yes.

Index:
no.

Training:
maybe no.

Example member report:

Crawl:
no public.

Index:
no.

Training:
no.

Primary control:
authentication.

Robots secondary.

This decision table is far more useful than copying a “2026 AI robots.txt template.”

Privacy Adalah Layer Keempat

Need mention because teams confuse.

Private content:
salary.
customer report.
invoice.
member-only research.
admin.

Do not solve with:

robots disallow.

noindex.

training block.

Use authentication and authorization.

Why?

robots is public instruction.

noindex is Search directive.

training permission is crawler-purpose control.

None prevents direct HTTP access by unauthorized human/bot if URL public.

Security answer:
access control.

This creates four-layer model:

Access control.
Crawl.
Index.
Training.

Keep separate.

SearchBot Bisa Crawl tetapi Page Noindex

Suppose OAI-SearchBot allowed.

Page contains noindex.

OpenAI publisher documentation includes noindex guidance for preventing certain URL surfacing when crawler can read directive.

Exact platform behavior should follow current documentation, but conceptual:

Search crawler access doesn't mean Search inclusion permission.

Same Google:
Googlebot can crawl noindex page but not index.

Crawler presence in logs isn't proof of Search visibility.

Noindex is not bot block.

This matters for log analysis.

Don't alert:
“OAI-SearchBot accessed noindex page, violation.”

Maybe crawler needed to read directive.

Check docs.

Policy.

GPTBot Bisa Diblok walau Page Indexable

This is the publisher use case.

SEO:
wants Search.

Legal:
training opt-out.

No conflict if crawler controls separated.

OpenAI:
GPTBot disallow.

Google Search:
Googlebot allow/index.

ChatGPT Search:
OAI-SearchBot allow.

Now public article remains searchable.

Training preference documented.

This isn't guarantee against every AI training system in world.

It only addresses documented bot.

Other vendors need policy.

Crawler matrix.

Do not say:
“Training disabled globally.”

Precise.

Training Permission Bisa Berbeda per Vendor

Vendor A:
training crawler separate.

Vendor B:
combined crawler.

Vendor C:
no official crawler.

Vendor D:
license/API.

So don't invent universal directive:

`NoAI: true`.

There is no single universal standard that all AI companies must follow unless such standard becomes officially adopted and documented.

At present, policy is vendor-specific.

Use official docs.

Record last checked.

If crawler purpose unclear:
monitor/block per risk.

No speculation.

Google-Extended Menunjukkan Layer yang Juga Berbeda

Google has documented Google-Extended as a control token related to use of site content for certain generative AI systems, while it does not affect inclusion or ranking in Google Search.

This is another useful conceptual example.

Search crawl/index:
Googlebot.

Generative-use control:
Google-Extended policy.

Exact scope can evolve; always check current Google documentation before implementation.

The important structural lesson:

Search permission and AI model-use permission can be separate even within one vendor.

Do not treat “Google” as one crawler policy.

Likewise OpenAI.

Business purpose first.

Technical user-agent second.

Crawl Permission Bisa Diizinkan tetapi WAF Tetap Memblok

robots:
allow.

WAF:
403.

Effective crawl:
no.

So policy needs two views:

Declared permission.

Actual transport.

Crawler access matrix should record:

robots.

WAF.

status code.

verification.

Logs.

A company may think:
“OAI Search allowed.”

But Cloudflare blocks AI bots.

Policy intent and actual outcome differ.

This is why crawling must be tested.

Permission isn't only file.

Infrastructure participates.

Index Permission Bisa Bertabrakan dengan Canonical

Page:
noindex.

Canonical points to indexable page.

This can happen on duplicates.

Need careful architecture.

Noindex says:
don't index this page.

Canonical says:
represent with another URL.

Sometimes both can coexist but mixed signals make management harder.

If true duplicate:
canonical often enough.

If page shouldn't Search at all:
noindex.

Don't stack every directive “for safety.”

Search directives should express one coherent intent.

Index permission isn't crawler-block backup.

Use right tool.

Training Permission Bisa Bertabrakan dengan Licensing

Content licensed CC BY.

Publisher policy says training block.

Can they?

Legal question depends rights/license/contract.

Technical rule can be written, but legal effectiveness/appropriateness isn't a GEO assumption.

In other case:
company owns content and chooses opt-out.

Fine.

Crawler governance should connect licensing metadata.

No universal training decision.

Legal owns rights.

Engineering implements.

GEO explains consequences.

This separation matters.

A public dataset may intentionally allow reuse.

Premium report may not.

One site can have mixed policy.

Path architecture can help.

Content Class Matrix

Create internal table.

Class:
PUBLIC_EDITORIAL.

Crawl:
Google yes.
OAI Search yes.
GPTBot no.

Index:
yes.

Training:
no.

Class:
OPEN_DATA.

Crawl:
yes.

Index:
yes.

Training:
yes if policy.

Class:
THANK_YOU.

Crawl:
yes.

Index:
no.

Training:
no/irrelevant.

Class:
MEMBER.

Access:
auth.
Crawl:
no public.
Index:
no.
Training:
no.

This is governance.

Editor doesn't touch robots.

Template/class controls.

At scale:
policy generated.

Manual rules fewer.

No conflicting intentions.

Do Not Use noindex as Training Opt-Out

This needs repetition because easy mistake.

Noindex says:
do not index in Search engine supporting directive.

Training crawler may not treat it as training permission unless vendor explicitly says.

Use vendor-documented mechanism.

For OpenAI:
GPTBot robots.

For other:
docs.

Likewise `nofollow` isn't training control.

`nosnippet` isn't training control.

`max-snippet` isn't training control.

Each Search directive has documented purpose.

Do not reinterpret because AI era.

Technical semantics remain.

Do Not Use Training Disallow as Deindex

Block GPTBot.

Google still indexes.

OAI-SearchBot still Search.

No deindex.

Block OAI-SearchBot.

Google may still index.

GPTBot might still be allowed.

Different.

If goal:
remove from Google Search:
noindex/removal/canonical/status as appropriate.

If goal:
remove from ChatGPT Search:
OpenAI Search controls/noindex guidance current.

If goal:
training opt-out:
training crawler.

Write objective before mechanism.

This one discipline avoids most policy bugs.

Human Scene: Empat Tim, Satu Page

Legal:
“Kita nggak mau training.”

SEO:
“Tapi Search harus tetap.”

Security:
“Member content nggak boleh public.”

Developer:
“Page mana?”

Editorial:
“Free article, premium report, open dataset beda.”

Good.

Now policy becomes granular.

Free article:
Search yes, training no.

Premium:
auth.

Open dataset:
Search yes, training yes.

No one has to compromise because controls separated.

The conflict was created only by pretending “AI access” is one switch.

The internet stack already has different levers.

Use them.

Measurement Harus Mengikuti Layer

Crawl:
server logs.

Index:
Search Console/index status where available.

Training permission:
robots/config + verified crawler logs.

Citation:
AI observation.

Referral:
analytics.

Don't infer.

If GPTBot absent:
could be policy or no crawl demand.

If brand cited:
doesn't prove training.

If SearchBot crawled:
doesn't prove citation.

If page indexed:
doesn't prove ranking.

Each metric answers one question.

This is essential for audit.

One dashboard can show lanes.

Not one score.

Policy Changes Need Timestamp

August 9:
GPTBot disallowed.

August 10:
SearchBot allowed.

August 12:
noindex removed.

Now future analyst can understand.

Without timestamps:
AI visibility change cannot be contextualized.

Robots cached.

Crawlers recrawl.

Index updates.

Training controls process.

Set change log.

No causal claim.

But timeline.

Infrastructure events belong measurement dataset.

This improves interpretation.

Final Mental Model

Think door, catalog, library, training room.

Access control:
Can you enter building?

Crawl:
Can crawler enter this room and read page?

Index:
Can page be listed in Search catalog?

Training:
Can specified training crawler use content for model-development purpose?

Citation:
Will librarian choose this source for answer?

Different.

Trying to control all with one sign on front door creates confusion.

In 2026, crawler governance becomes much easier once we stop asking:

“Apakah AI boleh akses?”

And start asking three precise questions:

“Boleh crawl?”

“Boleh index?”

“Boleh training?”

Then add the fourth if sensitive:

“Boleh access at all?”

Once those answers are separate, technical implementation becomes much less dramatic.