Cara Membuat Crawler Access Matrix untuk Google, OpenAI, dan Bot AI Lain

Robots.txt yang baik seharusnya datang setelah policy.

Bukan sebelum.

Banyak organisasi melakukan kebalikannya.

Mereka buka file.

Lihat bot list dari internet.

Lalu menulis:

Allow.

Disallow.

Allow.

Disallow.

Tanpa menjawab:

“Kenapa?”

Hasilnya crawler policy menjadi koleksi keputusan teknis yang tidak punya owner bisnis.

Search team ingin discovery.

Legal ingin training opt-out.

Marketing ingin ad validation.

Security ingin bot mitigation.

Infrastructure ingin server stabil.

Product ingin user-triggered agent access.

Semua bisa benar.

Crawler access matrix adalah cara sederhana menyatukan semua kebutuhan itu sebelum satu baris robots.txt diubah.

Matrix bukan standard resmi.

Ini governance tool.

Tujuannya:

Untuk setiap bot atau access mode, tulis siapa, purpose, desired access, scope, verification, dan owner.

Dengan begitu “block AI bots” berubah menjadi keputusan yang bisa diaudit.

Mulai dari Kolom yang Tepat

Minimal:

Vendor.

Crawler/User Agent.

Purpose.

Automatic atau user-triggered.

Robots control.

IP verification source.

Allowed paths.

Disallowed paths.

WAF action.

Rate limit.

Business owner.

Legal status/decision.

Last verified.

Notes.

Contoh:

OpenAI.
OAI-SearchBot.
Search.
Automatic.
Allow public editorial.
Published IP JSON.
WAF allow verified.
Owner SEO.

OpenAI.
GPTBot.
Training.
Automatic.
Disallow all.
Published IP.
WAF policy according risk.
Owner Legal + SEO.

OpenAI.
OAI-AdsBot.
Ads validation.
Platform-triggered on submitted landing pages.
Allow campaign landing pages if advertising.
Published IP.
Owner Paid Media.

OpenAI.
ChatGPT-User.
User-triggered.
Robots may not apply.
Allow normal public access, no security bypass.
Published IP.
Owner Product/Security.

Now file has reason.

Google Row Tidak Cukup “Googlebot”

Google Search uses multiple crawlers/user agents.

Googlebot Smartphone.

Desktop.

Googlebot-Image.

Googlebot-News in relevant contexts.

Special-case crawlers.

User-triggered fetchers.

Exact list changes.

Use Google official crawler documentation.

Don't copy one static blog list.

For a normal site:

Googlebot Search:
allow indexable public pages.

Googlebot-Image:
depends image strategy.

AdsBot-Google:
if ads/landing validation relevant.

Google-InspectionTool:
Search Console testing.

Other fetchers:
business-specific.

The matrix can be broad at category level but should reference official docs for current user agents.

Don't create rules for bots you don't use without reason.

OpenAI Shows Why Purpose Column Matters

OAI-SearchBot:
Search.

GPTBot:
training.

OAI-AdsBot:
ads.

ChatGPT-User:
user-triggered.

Same vendor.

Different policy.

If matrix only has:
“OpenAI = blocked.”

you lose nuance.

Business intent should be expressed at purpose level.

This design also works for other AI companies if they document separate crawlers.

Don't assume names/functions.

Research official sources.

If unclear:
mark Unknown.

Unknown is legitimate matrix value.

Do not guess.

Unknown Bot Should Not Automatically Be Called AI

Logs show:
`SuperAI-Crawler/1.0`.

No official docs.

Could be real startup.

Could spoof.

Classify:

Vendor:
unverified.

Purpose:
unknown.

Verification:
none.

Policy:
default bot/WAF policy.

Don't create `robots.txt` allow because name sounds AI.

Don't create article claiming function.

Verify domain.

Documentation.

IP.

Reverse DNS.

Public contact.

If still unknown:
treat as unknown automated traffic.

Security first.

Crawler matrix is also misinformation prevention.

Google Robots Semantics Need Correct Use

Google docs say robots.txt tells crawlers which URLs they can access.

It is mainly a crawl-control mechanism.

If you want noindex, crawler must be able to fetch page to see robots meta/X-Robots-Tag.

Therefore matrix should separate columns:

Crawl.

Index.

Snippet/serving.

Training.

Because one checkbox “access” hides differences.

Example:

Public legal page:
crawl yes.
index no.
training no.

Could express:
Google crawl allow + noindex.
OAI-SearchBot allow if platform noindex handling desired.
GPTBot disallow.

Exact platform support must be verified.

Matrix gives conceptual clarity.

Implementation is vendor-specific.

Crawl Permission vs Network Permission

Column:
robots.
Column:
WAF/CDN.

Why separate?

robots:
crawler policy.

WAF:
transport.

If robots allow but WAF 403:
crawler fails.

If robots disallow but WAF 200:
compliant crawler self-stops.

Matrix should capture both.

Example:

Googlebot:
robots allow.
WAF allow verified Googlebot.

OAI-SearchBot:
robots allow.
WAF allow OpenAI published IP.

Unknown bot:
robots default.
WAF challenge/rate limit.

This makes debugging fast.

No more:
“Robots is fine, why no crawl?”

Check network column.

Verification Method Must Be Vendor-Specific

Google documents ways to verify Googlebot using reverse DNS and published IP ranges.

OpenAI publishes JSON IP ranges per crawler/user agent.

Other vendors may provide:
IP lists.
DNS.
ASN.
No method.

Matrix should store link/reference internally.

Never trust user-agent only.

Crawler spoofing is trivial.

If security wants special access:
require verification.

If no verification:
don't whitelist based solely UA.

This principle protects.

Rate Limit Policy

Official bots can still create load.

Allow doesn't mean unlimited.

Reasonable rate controls may be needed.

But aggressive 429 could hurt crawl.

Matrix:

Expected frequency.

Rate limit class.

Burst.

Contact/escalation.

Logs.

Search engine crawlers usually manage crawl rates dynamically.

Don't throttle randomly without observing.

For AI bots, behavior may differ.

Use server capacity.

Cache.

Robots path scope.

If overload:
identify verified bot.

Check vendor docs/contact.

Don't block by panic.

Rate limits should be purpose-aware.

Path Classes Make Matrix Manageable

Instead of every URL:

PUBLIC_EDITORIAL.

PUBLIC_PRODUCT.

PUBLIC_DATASET.

PUBLIC_DOCS.

LOGIN.

MEMBER.

ADMIN.

API_PUBLIC.

API_PRIVATE.

STAGING.

SEARCH_INTERNAL.

Then policy.

Example:

Googlebot:
public classes allow.
login/admin disallow/no access.

OAI-SearchBot:
public editorial/docs allow.
login/admin no.

GPTBot:
all disallow.

ChatGPT-User:
public via normal HTTP.
private only authenticated integration.

OAI-AdsBot:
campaign landing page class.

Now 10 rules become manageable.

Architecture should label content classes.

Matrix maps purpose to class.

No 500 path exceptions.

Authentication Always Wins over Robots

Private member report.

Matrix should say:

Network/app auth:
required.

Crawler:
not relevant.

Never rely on disallow.

Same for staging:
HTTP auth/VPN.

`Disallow: /staging/` not enough.

Crawler matrix should include:
security classification.

Public.

Restricted.

Confidential.

If confidential:
crawler matrix is secondary.

Access control first.

This prevents SEO file becoming security policy.

Legal/Training Column

For AI crawlers, legal/business policy often matters.

Fields:

Training permitted?
yes/no/conditional.

Licensing?
open/proprietary.

Search permitted?
yes/no.

Archive?
maybe.

User-triggered?
public-only.

Don't encode legal reasoning in robots comments fully.

Matrix stores decision.

Legal owner approves.

Robots implements.

If policy changes:
update matrix then file.

This gives audit trail.

“No GPTBot” now has documented reason/date.

Not one engineer's preference.

Ads Column

Marketing can add:

Google Ads landing crawlers.

OpenAI AdsBot.

Others.

If company doesn't advertise:
mark N/A.

Don't over-allow.

When campaign starts:
enable relevant verified access.

Marketing submits ticket.

Security rule.

Test.

This makes ad crawler management event-driven.

Not permanent blanket.

Search Console / OpenAI Docs Monitoring

Crawler definitions change.

OpenAI provides RSS updates for crawler documentation.

Google maintains crawler docs.

Matrix should have:
last checked date.

Owner monthly/quarterly review.

Automate alerts if docs change, when practical.

Don't rely on memory.

Bot user-agent version may change.

IP ranges change.

New crawler added.

Old deprecated.

Currentness is part of governance.

A matrix from 2024 may be worse than none if everyone trusts it.

Version it.

Example Matrix in Plain Text

Vendor | Agent | Purpose | Crawl Policy | WAF | Verification | Owner

Google | Googlebot | Search | Allow public indexable | Allow verified | Google docs/IP/DNS | SEO/Infra

OpenAI | OAI-SearchBot | ChatGPT Search | Allow public editorial | Allow verified | OpenAI searchbot IP JSON | GEO/Infra

OpenAI | GPTBot | Training | Disallow | Default/observe | OpenAI GPTBot IP JSON | Legal

OpenAI | OAI-AdsBot | Ads landing validation | Allow submitted landing pages | Allow verified | AdsBot IP JSON | Paid Media

OpenAI | ChatGPT-User | User-triggered | Normal public access; robots may not apply | No security bypass | ChatGPT-User IP JSON | Product/Security

Unknown | Unknown AI UA | Unknown | No special allow | Default bot mitigation | Unverified | Security

That's enough to start.

No fancy software.

Spreadsheet.

YAML.

Git file.

Whatever.

Important:
owner and reason.

Crawler Access Matrix Harus Terhubung ke Logs

Policy says:
OAI-SearchBot allow.

Logs:
403.

Mismatch.

Policy says:
GPTBot disallow.

Logs:
200 requests? Could be robots fetch or spoof/noncompliance; investigate according paths and verification.

Policy says:
AdsBot only campaign page.

Logs:
thousands unrelated URLs.
investigate verification/pattern.

Build compliance report.

Expected vs observed.

This turns matrix from document into monitoring tool.

No need real-time for small site.

Monthly.

Large:
daily alerts.

What Does “Allow” Mean?

Define.

robots allow.

WAF allow.

Page public.

No CAPTCHA.

Resources needed load.

200 response.

Index directive maybe.

A simple “allow” cell can be ambiguous.

Use separate fields or note.

Likewise “block”:

robots disallow?
403?
auth?
noindex?

Each has different effect.

Matrix should encode mechanism.

This is why crawler governance is more than robots.txt.

Noindex Is Not a Training Control by Default

Do not assume.

`noindex` is indexing/serving directive for search crawlers that support it.

Training crawler policy should use vendor-documented control such as GPTBot disallow for OpenAI.

Different purpose.

Similarly robots disallow SearchBot controls crawl, not same as noindex.

Matrix keeps columns separate so engineers don't reuse one mechanism across purposes.

This prevents subtle mistakes.

User-Triggered Agents Need Security Column

ChatGPT-User may access public content based on user request.

Other browser agents exist.

Robots may not govern.

Matrix row should point to:

Public content:
normal access.

Authenticated:
must authenticate.

Actions:
API scopes.

Forms:
CSRF/abuse controls.

Rate limit.

No UA bypass.

Agent traffic is client traffic.

Security architecture.

Not crawler-only.

This will matter more as agents act.

But don't invent universal bot rules.

Per platform.

AI Bot Lists from Third Parties Are Starting Point, Not Authority

Community lists useful for discovery.

Not source of truth.

Bot name can be outdated.

Purpose wrong.

User agent changed.

IP stale.

For policy:
official docs first.

If no official:
mark unverified.

Web search can find announcements.

But don't whitelist based on random GitHub list without verification.

For GEO editorial, distinguish:

Official platform fact.

Third-party observation.

Internal log observation.

Hypothesis.

Same factual discipline.

Google, OpenAI, dan Bot Lain Tidak Harus Punya Policy yang Sama

Business may want:

Google Search:
yes.

ChatGPT Search:
yes.

Training:
no.

Other AI Search:
yes/no.

Unknown scrapers:
no.

Ads:
selective.

That's valid.

No moral requirement for uniform.

Crawler access is strategic choice.

What matters:
consistent with business/legal goals.

Document.

Don't let a developer's personal view of AI become company policy.

Governance protects both sides.

Testing Before Production

Staging:

robots test.

curl with UA.

WAF simulation.

IP validation.

Public path.

Blocked path.

Check status.

After deploy:

Fetch robots public.

Test Google tools where available.

Observe OpenAI logs over appropriate window.

Check 403/429.

SearchBot adjustment may take time; OpenAI says about 24 hours for robots Search changes.

Document timestamp.

No instant conclusion.

Rollback if widespread Search impact.

Config as code.

Crawler Policy Changes Need Incident Plan

If accidental `Disallow: /`:

Detect.

Fix.

Deploy.

Purge robots cache if platform/CDN.

Request recrawl where supported.

Monitor.

Record incident.

If WAF blocks official bot:
correct verified rule.

If unknown bot overload:
mitigate.

Matrix gives expected policy.

Incident response compares.

No debate during outage.

“What should GPTBot be?”
Matrix says.

This is why governance saves time.

Metrics

Crawler success rate by agent.

2xx/3xx/4xx/5xx.

Robots changes.

Verified/unverified.

Top paths.

Bytes.

Latency.

AI referral traffic separately.

Citation separately.

Training policy compliance separately.

Do not combine into:
AI Visibility Score.

Crawler access is infrastructure metric.

Citation is Search outcome.

Referral is traffic.

Different.

A Good Matrix Makes robots.txt Boring

That's ideal.

By the time developer edits robots, decisions are already made.

OAI-SearchBot:
this.

GPTBot:
that.

Googlebot:
this.

AdsBot:
that.

Unknown:
default.

No debate.

No guessing.

Robots file becomes implementation of policy.

WAF mirrors.

Logs verify.

Owners review.

In 2026, that is much safer than downloading “ultimate AI robots.txt” from internet.

There is no universal ultimate file because companies want different things.

The reusable asset isn't the file.

It's the matrix.

Once you know who the crawler is, what it does, what your business wants, and how to verify it, the actual robots rule becomes the easy part.