Checklist AI Crawler Governance untuk 2026

Crawler governance 2026 bukan soal punya robots.txt paling panjang.

Bukan juga soal memblok semua bot yang namanya terdengar “AI”.

Yang dibutuhkan justru lebih sederhana:

Tahu siapa yang datang.

Tahu mereka datang untuk apa.

Tahu content mana yang boleh diakses.

Tahu control mana yang tepat.

Tahu apakah infrastructure benar-benar menjalankan policy itu.

Dan tahu apa yang masih belum bisa kita simpulkan dari log.

Setelah puluhan pembahasan tentang OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot, Googlebot, noindex, WAF, geo-blocking, rate limiting, canonicalization, member area, dan multi-domain, ada satu pattern yang terus berulang:

Kesalahan terjadi ketika layer yang berbeda dicampur.

Search dianggap training.

robots dianggap security.

noindex dianggap access control.

User-Agent dianggap authentication.

Crawler hit dianggap citation.

Verified bot dianggap boleh ke semua endpoint.

Semua itu terlalu kasar.

Checklist ini dibuat sebagai operational review untuk website institusional, publisher, corporate, ecommerce, research site, atau brand multi-domain yang ingin crawler policy tetap bisa diaudit sepanjang 2026.

Bukan universal legal policy.

Bukan jaminan AI citation.

Ini governance baseline.

1. Buat Inventory Machine Access yang Penting

Mulai dari crawler yang observed, documented, dan business-relevant: Googlebot, OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot bila relevan, crawler vendor lain yang benar-benar penting, serta unknown automation.

Untuk setiap agent, catat vendor, purpose, official docs, verification method, business owner, dan last review. Kalau purpose belum jelas, tandai Unknown. Jangan menebak.

2. Pisahkan Search, Training, Ads, dan User-Triggered Access

OpenAI current documentation memberi contoh yang sangat jelas:

OAI-SearchBot:
Search.

GPTBot:
training crawler.

OAI-AdsBot:
landing-page ad validation/review.

ChatGPT-User:
certain user-triggered actions, bukan automatic Search crawler.

Policy jangan:

“OpenAI allow.”

Tulis:

Search:
yes.

Training:
no.

Ads:
only campaign URLs.

User-triggered:
normal public access, auth tetap berlaku.

Ini satu perubahan mental yang mencegah banyak incident.

3. Tentukan Content Classes

Jangan manage per URL satu-satu.

Class:

PUBLIC_INDEXABLE.

PUBLIC_NOINDEX.

PUBLIC_NO_TRAINING.

OPEN_DATA.

PRODUCT.

SUPPORT.

ADS_LANDING.

PREMIUM.

MEMBER.

STAGING.

ADMIN.

API_PUBLIC.

API_PRIVATE.

Lalu map crawler policy ke class.

Contoh:

PUBLIC_INDEXABLE:
Google yes.
OAI Search yes.
GPTBot according training policy.

MEMBER:
authentication required.

STAGING:
private.

ADS_LANDING:
AdsBot access if campaign.

Ini lebih scalable.

4. Jangan Gunakan robots.txt sebagai Security

Private:
auth.

Staging:
auth/VPN/Access.

Member:
authorization.

Admin:
MFA/network controls.

robots.txt hanya instruction bagi compliant crawler.

Google sendiri menjelaskan robots bukan mekanisme untuk menjaga page keluar dari Search secara penuh, apalagi untuk menjaga rahasia.

Kalau URL tidak boleh dibuka anonymous:
server harus menolak.

Jangan cuma:
Disallow.

5. Pisahkan Crawl Permission dari Index Permission

Crawl:
robots.

Index:
noindex/X-Robots-Tag untuk supported Search crawler.

Google menegaskan crawler harus dapat mengakses page untuk membaca noindex.

Jadi jangan:

Disallow
+
noindex

lalu mengira keduanya “lebih kuat.”

Crawler mungkin tidak membaca noindex.

Use correct tool.

Public but not Search:
allow crawl + noindex.

Don't fetch:
disallow.

Private:
auth.

6. Pisahkan Training Permission

Noindex bukan universal training opt-out.

Untuk OpenAI:
GPTBot adalah documented training crawler control.

Untuk vendor lain:
check docs.

Google has its own controls for some generative-use contexts.

Don't invent:

`NoAI: true`

unless documented by target consumer.

Training policy is vendor-specific.

Record.

Review.

7. Verify Official Crawlers

Never trust User-Agent alone.

Google:
reverse DNS/published IP ranges.

OpenAI:
published IP JSON per crawler.

Security vendor:
verified-bot classification.

Classify:

Verified.

Unverified.

Unknown.

Do not grant WAF bypass solely because string contains:
Googlebot.
GPTBot.
OAI-SearchBot.

UA is self-declared.

It is not credential.

8. Keep Authentication Independent from Bot Identity

Even verified ChatGPT-User should not bypass:

login.

member entitlement.

private API.

checkout authorization.

admin.

If agent needs access on behalf of user:
authenticated integration/session/token.

Service identity ≠ user authorization.

This rule becomes increasingly important in agentic web.

9. Audit WAF/CDN after robots

robots Allow doesn't mean crawler receives 200.

Check:

WAF.

Bot Management.

Geo rules.

Rate limit.

CAPTCHA.

JS challenge.

Managed rules.

Cloudflare AI Crawl Control.

Custom firewall.

Origin protection.

Edge logs.

If crawler gets 403:
robots is not the problem.

Find rule ID.

Transport and policy must align.

10. Treat Geo-Blocking as a High-Risk Crawler Rule

Website Indonesia does not imply crawler IP Indonesia.

Verified Search/AI crawlers can originate elsewhere.

If public content needs global discovery:
create narrow verified-crawler exception where policy allows.

Do not remove geo-security globally.

Do not block all foreign IP and assume Search unaffected.

Test.

11. Treat Rate Limiting as Capacity Control, Not Crawl Policy

429:
too many requests.

403:
forbidden.

Use correct semantics.

Desired crawler:
reasonable verified rate.

Unknown bot:
stricter.

Training bot disallowed:
robots policy.

Search crawler:
don't throttle into permanent failure.

Cache public content.

Optimize origin.

No universal request/minute number.

Use load data.

12. Separate Read and Write Access

Search crawlers need GET/HEAD public content.

They do not need:

POST form.

login.

upload.

admin.

API write.

Even verified bot:
least privilege.

User-triggered agents that need actions should use proper authenticated tool/API path.

Never broaden WAF exception to all methods.

13. Keep robots.txt Small Enough to Understand

Google processes robots files up to 500 KiB; content after limit is ignored.

But practical auditability fails far earlier.

Avoid bot zoo.

Avoid duplicate groups.

Avoid unsupported directives.

Google does not support `crawl-delay`.

Use comments.

One purpose block.

Version.

Tests.

No secrets in comments.

A human should answer:
“Can SearchBot crawl `/research/`?”
quickly.

14. Version robots and WAF Policy

Store:

Date.

Diff.

Reason.

Owner.

Expected impact.

Affected agent.

Affected path.

Rollback.

If AI visibility changes:
timeline.

Not causal proof.

But essential context.

Config history should survive longer than raw logs.

Treat robots like code.

15. Monitor `/robots.txt` Itself

External fetch:

200?

Plain text?

Correct host?

Correct content?

No challenge?

No 5xx?

Hash expected?

CDN cache current?

After change:
did crawler refetch?

OpenAI notes Search robots changes may take roughly 24 hours to be reflected.

Don't judge instantly.

16. Audit Sitemaps

Sitemap should represent URLs you want discovered/indexed according Search architecture.

Check:

noncanonical URLs.

redirects.

404.

noindex.

staging.

old domain.

parameters.

A sitemap can expose URLs.

It doesn't guarantee crawl/index.

Google says sitemaps aid discovery, especially for large/new sites.

Keep clean.

17. Audit Internal Links

Crawler finds URLs through links.

A retired page still linked from old article remains discoverable.

Check:

navigation.

footer.

related posts.

breadcrumbs.

author pages.

old PDFs.

RSS.

If URL moved:
redirect.

If gone:
404/410.

If duplicate:
canonical.

If no Search:
noindex.

Don't use robots to hide lifecycle mistakes.

18. Understand Third-Party Discovery

OpenAI states a disallowed URL may still be known from third-party search provider or crawling other pages and can surface as link/title in Atlas under documented conditions.

Google also explains disallowed URLs can be discovered from external links.

Therefore:

URL known ≠ content crawled.

Link visible ≠ robots ignored.

Investigate logs.

Classify:
link.
snippet.
quote.
citation.

Evidence before accusation.

19. Monitor Canonicalization

For Google:

declared canonical.

Google-selected canonical.

redirect.

sitemap.

internal links.

hreflang.

structured data URLs.

Ensure critical pages align.

Google AI Overviews/AI Mode require page to be indexed and Search-eligible with snippet.

So canonical/indexing health is a prerequisite there.

No special “AI canonical.”

Normal Search architecture.

20. Don't Assume OpenAI Has Google-Style Index Visibility

We can observe:

OAI-SearchBot logs.

robots policy.

citations.

referrals.

URL surfacing.

But don't call a page “indexed in ChatGPT” unless a platform tool/documentation actually provides that state.

Use precise words:

crawled.

surfaced.

cited.

referred.

known URL.

Unknown stays unknown.

21. Audit Source Response, Not Only Status

200 can be:

real content.

challenge page.

empty shell.

maintenance message.

login.

bot-specific content.

Sample body.

Content length.

Title.

Hash.

Required resources.

Crawler must receive actual public content.

Don't celebrate 200 blindly.

22. Keep Structured Data in Its Lane

Schema describes entities and content. It is not crawler permission, training opt-out, or authentication.

Google says no special schema is required for AI features in Search. Use supported structured data and keep it aligned with visible content.

23. Maintain Crawler Access Matrix

Catat agent, purpose, verification, robots/WAF policy, owner, dan last review. Matrix adalah source-of-truth; logs membuktikan implementasinya.

24. Separate Crawler Logs by Purpose

Pisahkan Search, training, user-triggered, ads, dan unknown. Ukur requests, status codes, top paths, latency, serta verification.

25. Retain Logs with a Lifecycle

There is no universal retention number. Keep raw logs only as long as diagnostic needs justify, retain aggregates longer for trends, preserve config history, and handle incident evidence under security/legal policy.

Separate crawler telemetry where possible and redact unnecessary sensitive data.

26. Store Verification Provenance

IP ranges change.

Record:

source.

fetch date.

classifier version.

If old logs classified with old ranges:
context preserved.

Otherwise future audit can mislabel historical bot.

This is small detail that makes long-term telemetry trustworthy.

27. Monitor Unknown Bots

If a new AI-sounding UA has no official docs or verification, keep default security, monitor it, and only create an exception after business value and identity are established. Abuse can be blocked; uncertainty is not proof of legitimacy or misconduct.

28. Track Official Changes

Review crawler docs ketika ada bot baru, ads launch, migration, policy/legal change, atau security incident.

29. Include Paid Media

If running ChatGPT Ads:

OAI-AdsBot must reach landing page for validation/review according OpenAI advertiser guidance.

Check:

robots.

WAF.

geo.

CAPTCHA.

auth.

redirect.

429.

Don't let SEO block `/promo/` without paid media knowing.

Ads crawler is part of crawler governance now.

30. Include User-Triggered Agent Access

ChatGPT-User is not OAI-SearchBot.

OpenAI says it isn't automatic Search crawl and robots may not apply because user-triggered.

Public page:
normal.

Private:
auth.

Do not use its presence as Search indexing evidence.

Do not count as human pageview.

Do not bypass security.

Segment logs.

31. Include Multi-Domain Coverage

Corporate, country, shop, support, docs, investor, legacy, and campaign hosts can all have separate crawler behavior.

Google robots rules are host-specific. One root robots does not govern every subdomain. Audit each host under one central policy.

32. Retire Old Domains Properly

If moved:
redirect.

Do not simply robots-block old host.

Crawlers need to see redirect.

Update:

sitemap.

links.

canonical.

profiles.

WAF.

Search Console.

AI citations may lag through third-party links.

Monitor.

No instant assumption.

33. Protect Staging Properly

Use auth, VPN, or an access proxy. Robots/noindex are secondary safeguards, not isolation. Draft information should not be publicly reachable.

34. Protect Member Areas Properly

Use server-side authorization. Do not hide full content with CSS or bypass access by User-Agent. If Search needs a teaser, expose only the public portion; agent access to member content requires delegated authentication.

35. Separate Search Visibility from Training Compliance

Brand cited after GPTBot blocked?
Does not prove training violation.

Brand absent after SearchBot allowed?
Does not prove crawler broken.

Logs measure crawl, policy measures permission, citation measures answer output, dan referral measures traffic. Model memory tidak dapat dibaca dari robots.

36. Separate Crawlability from Citation

Crawler 200:
door open.

Citation:
source chosen.

Between:
indexing.
relevance.
freshness.
evidence.
query fit.
source competition.

No crawler configuration guarantees citation.

PASS bukan visibility PASS.

37. Prefer Primary Sources for Platform Facts

Gunakan OpenAI docs untuk OpenAI crawler, Google Search Central untuk Google crawling/indexing, dan dokumentasi security vendor untuk rule behavior. Third-party research boleh melengkapi, tetapi jangan mengubah inference menjadi fakta platform.

38. Test from the Outside

Fetch the real public robots, page, headers, canonical/noindex state, status, and WAF outcome, then compare with crawler logs. Production behavior is the evidence; configuration screenshots are only intention.

39. Build Incident Triggers

Alert on events such as wildcard `Disallow: /`, Googlebot/OAI Search 403 spikes, AdsBot blocks during campaigns, robots 5xx, mass canonical/noindex changes, unexpected GPTBot policy changes, or unknown-bot overload.

Assign an owner and rollback path. The matrix defines expected state.

40. End with One Question: Does Reality Match Policy?

Policy dan production harus sama.

Jika SearchBot seharusnya allowed tetapi mendapat 403, policy gagal. Jika content disebut private tetapi anonymous user mendapat 200, security gagal. Jika page noindex tetapi crawler diblok sebelum membaca directive, implementation gagal. Jika AdsBot wajib mencapai landing page tetapi WAF menolak, campaign path gagal.

Prinsipnya tetap stabil:

Crawl bukan index.
Index bukan training.
Training bukan user-triggered access.
Verification bukan authorization.
Allow bukan citation.
Robots bukan security.

Governance yang baik tidak mencoba memprediksi setiap bot. Ia membuat sistem yang cukup presisi untuk mengklasifikasi actor baru, memberi policy yang benar, dan menguji apakah production benar-benar menjalankannya.