Crawler Verification: Jangan Percaya User-Agent Saja

Log server bilang:

`OAI-SearchBot`.

Apakah itu pasti OpenAI?

Belum tentu.

Log lain bilang:

`Googlebot`.

Apakah pasti Google?

Juga belum tentu.

User-Agent adalah header HTTP yang dikirim client untuk menjelaskan dirinya. Ia berguna untuk classification, tetapi bukan authentication.

Client bisa menulis apa saja.

Bot scraper bisa mengaku Googlebot.

Crawler abusive bisa mengaku GPTBot.

Script buatan sendiri bisa mengirim `ChatGPT-User`.

Karena itu crawler verification tidak boleh berhenti di:

“Nama bot ada di log.”

Yang harus ditanya:

Apakah source network cocok dengan dokumentasi resmi?

Apakah IP masuk published range?

Apakah reverse DNS sesuai, jika vendor mendokumentasikan metode itu?

Apakah request pattern masuk akal?

Apakah status/path sesuai expected behavior?

Di 2026, distinction ini semakin penting karena infrastructure team mulai membuat exception khusus untuk Search crawler dan AI bot. Kalau exception hanya berbasis User-Agent, attacker bisa copy header yang sama dan ikut mendapat akses.

Itu bukan lagi masalah SEO.

Itu security bug.

Google Sendiri Mengingatkan User-Agent Sering Dipalsukan

Google Search Central secara eksplisit mengatakan HTTP User-Agent yang digunakan Googlebot sering dipalsukan crawler lain.

Google merekomendasikan dua cara untuk memverifikasi request Googlebot:

Reverse DNS lookup pada source IP.

Atau mencocokkan source IP dengan published Googlebot IP ranges.

Ini platform fact.

Jadi rule:

“If UA contains Googlebot → Allow.”

terlalu lemah.

Lebih baik:

UA matches expected Googlebot class.

Dan network verification passes.

Untuk managed platform, provider seperti Cloudflare dapat membantu dengan verified-bot classification.

Untuk custom infra, gunakan official ranges/DNS procedure.

The point:
identity must be verified beyond self-declared string.

OpenAI Menyediakan Published IP Ranges

OpenAI crawler documentation current juga mempublikasikan IP address lists terpisah untuk:

OAI-SearchBot.

GPTBot.

ChatGPT-User.

OAI-AdsBot.

Ini memberi infrastructure team basis untuk verification.

Kalau request mengaku OAI-SearchBot tetapi source IP tidak berada di published SearchBot range:

Jangan langsung whitelist.

Classify as unverified.

Investigate.

Bisa spoof.

Bisa range berubah dan local copy stale.

Bisa request lewat architecture yang belum kita pahami.

Bisa documentation update.

Jadi response bukan:
“Pasti fake.”

Response:
“Belum terverifikasi.”

Kemudian refresh official list.

Check date.

Check JSON source.

Compare.

Evidence first.

User-Agent Adalah Claim, IP Range Adalah Corroboration

Cara mental paling sederhana:

User-Agent:
“Aku OAI-SearchBot.”

IP:
“Request berasal dari network range yang OpenAI publikasikan untuk SearchBot.”

Dua signal lebih kuat daripada satu.

Tetap bukan cryptographic identity.

Tetapi untuk crawler verification operasional, jauh lebih baik.

Google reverse DNS adds another verification path.

Some vendors may publish ASN or DNS.

Others mungkin tidak punya official method.

Jangan universalize.

Per vendor:
official docs.

Kalau vendor tidak punya verification mechanism:
policy should be more conservative.

No secret allow based on name.

Skenario yang Bisa Menjadi Incident

Security membuat rule:

If User-Agent contains `OAI-SearchBot`
→ Skip WAF.

Attacker melihat.

Mengubah User-Agent scraper mereka.

Sekarang bot abusive melewati challenge.

Fetch:

Login.

Search endpoint.

Expensive API.

Maybe forms.

Even if only GET:
server load.

This is why UA-based WAF bypass is risky.

Search crawler exception should be narrow:

Verified source.

Public GET/HEAD paths.

No admin.

No private API.

Normal application auth remains.

No bypass sensitive route.

Least privilege.

SEO should never ask security:

“Whitelist this UA everywhere.”

Wrong abstraction.

Crawler Verification dan Robots.txt Beda Layer

robots.txt uses User-Agent token by design.

It tells compliant crawlers policy.

Example:

User-agent: OAI-SearchBot
Allow: /

This doesn't authenticate bot.

A fake bot may ignore robots anyway.

Or may read.

WAF verification is separate.

So:

robots:
policy.

network verification:
identity confidence.

WAF:
enforcement.

logs:
observation.

Don't confuse.

A robots group matching GPTBot is not proof the request actually came from OpenAI.

When analyzing logs:
verify separately.

This matters especially when claiming:

“GPTBot ignored robots.”

Maybe request was spoofed.

First question:
Was IP official?

Only then investigate compliance.

Full User-Agent Version Jangan Dijadikan Hard Match

OpenAI documentation gives example user-agent strings and notes version number may change.

So don't write firewall rule requiring:

`OAI-SearchBot/1.0`

forever.

Better match documented token plus official source range.

If version becomes 1.1:
rule still works.

Googlebot strings also vary by crawler/device/context.

Use vendor-documented tokens.

Do not overfit.

Hardcoding entire UA is brittle.

Security precision comes from source verification, not exact marketing string.

Reverse DNS Googlebot: Jangan Berhenti di PTR Saja

Google's documented reverse DNS verification process conceptually requires verifying that source hostname belongs to Google domain and then resolving hostname back to IP to ensure it matches original source.

Why two steps?

PTR records alone can be manipulated under control of IP owner.

Forward-confirmation reduces spoof risk.

If using published IP ranges instead:
compare ranges.

Choose method.

Automate carefully.

Do not perform huge reverse DNS lookup synchronously per request in app.

That's inefficient.

Use WAF/provider verified-bot feature or cached verification.

Operational design.

No need PHP call DNS for every pageview.

Crawler verification belongs infrastructure layer.

IPv6 Harus Masuk

Don't build verification only for IPv4 if official ranges can include IPv6.

IP list parser should support:

CIDR v4.

CIDR v6.

Normalize.

Test.

A rule that accepts official IPv4 but challenges official IPv6 can create random crawl failures.

Likewise logs:

X-Forwarded-For may contain IPv6.

Use trusted proxy chain.

Extract real client IP correctly.

If app reads wrong header:
verification fails or attacker can spoof.

Security team must define trusted proxy sources.

Crawler audit depends on correct client IP.

Cloudflare `CF-Connecting-IP`, load balancer headers, etc. need correct trust configuration.

Don't blindly accept user-supplied X-Forwarded-For.

Source IP at Edge vs Origin

Behind CDN:

Origin sees Cloudflare IP.

Not crawler IP unless forwarded header.

If you compare origin socket IP to OpenAI range:
all fail.

Need edge logs or trusted connecting-IP header.

Example:

Crawler IP → Cloudflare → origin.

Origin remote_addr:
Cloudflare edge.

Header:
original client.

So verification ideally at CDN where source is known.

Cloudflare verified bot classification lives there.

If custom origin audit:
use trusted header.

Document.

Many “OpenAI IP mismatch” bugs are proxy confusion.

Map architecture first.

Crawler Verification Database Harus Punya Timestamp

Official ranges can change.

Store:

Vendor.

Agent.

CIDR.

Fetched_at.

Source URL.

Checksum.

Last successful refresh.

If fetch fails:
don't replace list with empty.

Keep previous and alert.

If list changes dramatically:
review.

Automation itself can create outage.

Example update job downloads HTML error page and clears allowlist.

Now Search crawler blocked.

Safe configuration update:

Fetch.

Validate JSON.

Validate CIDR syntax.

Compare diff.

Deploy.

Rollback.

This is small but real infrastructure.

User-Agent Spoof Detection in Logs

Classify:

verified_googlebot.

spoof_googlebot.

verified_oai_searchbot.

unverified_oai_searchbot.

verified_gptbot.

unverified_gptbot.

unknown_bot.

Don't label unverified as “fake” automatically.

Use neutral.

Then security can see:

Which spoof labels most common?

Which paths?

Any abuse?

You can block/challenge unverified claims.

No impact to verified.

This makes dashboards more reliable.

GEO team can then report:

“Verified OAI-SearchBot crawl activity.”

Not:
“OpenAI visits.”

Precision.

ChatGPT-User Requires Same Verification Discipline

Because ChatGPT-User is user-triggered context, teams may be tempted to trust it more.

Don't.

UA can be spoofed.

Even verified IP isn't authorization to private endpoint.

It only identifies network class.

If action needs authentication:
OAuth/API credential/session.

Never:
If ChatGPT-User → bypass login.

Public GET:
fine.

Private data:
auth.

Source verification and user authorization solve different questions.

Who is requesting?

Are they allowed?

Don't combine.

OAI-AdsBot juga Bisa Dispoof

Marketing campaign landing page.

Security makes bypass:

UA = OAI-AdsBot.

Attackers can use same string.

Better verify published AdsBot IP range.

And narrow access to public landing pages.

No reason AdsBot needs wp-admin.

No reason to bypass WAF on upload endpoint.

Purpose-specific allow.

The crawler matrix tells expected paths.

Identity + purpose + route.

This is more secure.

What About Google-InspectionTool and Other Official Bots?

Google has different crawlers/fetchers beyond Googlebot.

Use Google's official list and verification docs.

Don't assume everything from Google uses Googlebot token.

If Search Console URL Inspection fetcher differs:
policy may need allow.

Again:
vendor crawler inventory.

OpenAI also has several.

One provider is not one UA.

Verification system should map classes.

No giant regex:
`google|openai|anthropic|ai`.

That's not verification.

That's wishful allowlisting.

Third-Party Bot Databases

Cloudflare, Fastly, AWS WAF partners, bot-management vendors may classify known bots.

Useful.

But if business-critical:
understand source/update.

Cloudflare `cf.client.bot` verified classification can be practical.

Still, custom rules may conflict.

And newly launched crawler may not immediately be verified.

So maintain fallback:

official IP list.

Vendor classification.

Logs.

Don't treat third-party database as infallible.

Verification is operational confidence, not metaphysical truth.

Request Pattern Is Supporting Evidence, Not Identity Proof

OAI-SearchBot hits:

robots.txt.

sitemap.

articles.

GET.

Reasonable.

But malicious scraper can imitate.

So behavior is not sufficient.

Similarly:

High request rate doesn't prove fake.

Official crawler can crawl fast.

Use behavior for anomaly detection after source verification.

If official crawler suddenly hits 10,000 expensive search URLs:
maybe site architecture leak.

Not fake.

If unverified bot claims SearchBot and probes login:
likely suspicious.

Classification uses multiple signals.

Don't let one pattern decide.

Crawler Verification Before Whitelisting

The safest workflow:

1. Identify business need.

2. Find official crawler docs.

3. Record User-Agent token.

4. Record official IP verification method.

5. Implement verified classifier.

6. Limit scope to required public paths/methods.

7. Keep authentication unaffected.

8. Monitor logs.

9. Refresh ranges/docs.

10. Review anomalies.

Only then:
special WAF exception.

No need whitelist if default security already allows crawler.

Don't add complexity without problem.

If crawler already gets 200:
monitor.

A verification project exists to prevent false blocks and spoof bypass.

Not to create unnecessary rules.

Crawler Verification Before Accusing Platform

This is equally important.

Publisher sees:
UA GPTBot.

robots says Disallow.

Bot keeps fetching.

Before public allegation:
verify source IP.

Check robots fetch timestamp.

Check CDN cache.

Check path allowed?

Check wildcard syntax.

Check fake bot.

Check official docs.

Then document.

This protects credibility.

There are real crawler bugs in world.

But UA spoofing is common.

Evidence first.

GEO.or.id should model that discipline.

No sensational “AI ignores robots” from one log line.

The Best Crawler Policy Starts with “Who Are You?”

robots answers:
“What are you allowed to crawl?”

WAF answers:
“Do we let this request through?”

But before either special-case access, verify:

“Are you actually who you claim?”

User-Agent alone cannot answer.

It is a name badge anyone can print at home.

Official IP ranges, DNS verification, verified-bot systems, and careful logging give us stronger evidence.

Not perfect cryptographic identity.

Strong enough for practical crawler governance.

And much safer than turning the string `Googlebot` or `OAI-SearchBot` into a master key.