WAF, CDN, dan Bot Protection: Kenapa Crawler Resmi Bisa Tetap Terblokir

Robots.txt bilang Allow.

Server sehat.

Sitemap ada.

Page public.

Tetapi crawler resmi tetap menerima 403.

Ini salah satu incident paling membingungkan buat tim SEO karena semua pemeriksaan “Search” terlihat benar.

Masalahnya bukan di robots.

Masalahnya di layer sebelum origin.

CDN.

WAF.

Bot Management.

Geo-block.

Rate limit.

JavaScript challenge.

Managed challenge.

Firewall rule.

ASN block.

Country deny.

Crawler resmi bisa dihentikan sebelum WordPress atau application server sempat melihat request.

Di 2026, problem ini makin common karena security platform semakin agresif membedakan human, known bot, AI crawler, scraper, and suspicious automation.

Cloudflare sendiri mendokumentasikan konsep Verified Bots dan menyediakan contoh bagaimana custom WAF rules dapat memberi pengecualian untuk search engine bots dan verified bots.

Tapi verified status bukan guarantee bahwa setiap custom rule otomatis melewati bot.

Rule order, terminating action, geo policy, and product configuration masih bisa menyebabkan legitimate crawler terblokir.

Jadi “crawler sudah verified” bukan akhir audit.

Network path tetap harus diuji.

Request Tidak Langsung ke WordPress

Typical path:

Crawler
→ DNS
→ CDN edge
→ WAF
→ Bot management
→ Rate limiter
→ reverse proxy
→ load balancer
→ origin web server
→ app/WordPress

Kalau WAF block:
origin log mungkin kosong.

SEO membuka Nginx log:
“Bot nggak pernah datang.”

Actually request stopped at edge.

Need CDN security events.

Different logs.

This is why infrastructure and SEO must collaborate.

Crawler audit yang cuma melihat WordPress access log bisa missing edge blocks.

Map request path.

Know where telemetry exists.

Robots Allow Is Not Network Allow

Robots:
crawler permission signal.

WAF:
security enforcement.

Example:

robots:
OAI-SearchBot Allow.

Cloudflare:
block AI bots.

Outcome:
403.

Or:

Googlebot allow.

WAF country rule blocks US data center IP.

Outcome:
403.

Or:

verified bots allowed in Bot Management but custom firewall rule above blocks all non-Indonesia IP.

Outcome:
403.

The crawler never sees content.

No contradiction.

Different layers.

Fix requires changing WAF/CDN policy, not robots.

Verified Bots Help, but Rule Design Matters

Cloudflare defines Verified Bots as bots/agents confirmed to transparently represent who they are and not abuse the access.

Examples include search engine crawlers, monitoring services, user-driven agents.

Cloudflare provides fields like `cf.client.bot` for rules.

A site can create logic:

challenge selected countries,
except verified bots.

Useful.

But if another higher-priority custom rule blocks IP before exception, bot can still stop.

Cloudflare docs note rule execution order/interoperability matters; terminating actions can prevent later products/rules from evaluating.

So audit security stack.

Not one toggle.

Also, Cloudflare verified list and bot categorizations evolve.

Check current.

Don't assume every new AI crawler already classified.

Official Vendor IP Range Is Separate Verification

OpenAI publishes IP ranges for:

OAI-SearchBot.

GPTBot.

ChatGPT-User.

OAI-AdsBot.

Google publishes crawler verification methods/IP ranges and reverse DNS guidance.

Security team can use those.

For official crawler allowlist:
verify source.

Don't trust user-agent.

Potential strategy:

UA matches OAI-SearchBot
AND source IP in official list
→ skip selected bot challenges.

But exact WAF syntax depends provider.

Don't copy blindly.

Keep ranges current.

Automate update safely if possible.

Log.

Review.

Security exception should be narrow.

Geo-Blocking Is a Common False Block for Indonesia Sites

A site targets Indonesia.

Security rule:

Block all countries except ID.

Looks efficient.

But search/AI crawler infrastructure may originate outside Indonesia.

Googlebot locations vary.

OpenAI crawler IP ranges may be US/cloud regions.

Result:
public Indonesian page inaccessible to crawler.

User in Jakarta:
200.

Crawler:
403.

Team tests locally:
“Works.”

Search system:
cannot fetch.

Geo-block should be based on actual threat/business need, not audience geography alone.

If public content wants global indexing:
allow verified crawlers regardless source country.

Article 287 goes deeper.

For now:
country block is high-priority audit item.

JavaScript Challenge and CAPTCHA

Crawler hits:

“Checking your browser…”

Needs JS.

Cookie.

Challenge.

Official crawler may not solve.

Even if crawler can execute JavaScript for page rendering, anti-bot challenge is different.

CAPTCHA intentionally demands interactive verification.

Searchbot may fail.

WAF returns challenge status/page rather than original content.

Result:
crawler indexes challenge page?
fails?
retries?
Behavior varies.

Don't serve bot challenges to verified crawlers you intentionally allow.

Security can skip challenge for verified sources.

Keep normal abuse controls.

Public content should be directly fetchable where possible.

Rate Limiting Can Be the Hidden Cause

Crawler starts refresh.

100 requests.

Rate rule:
20 requests/minute/IP.

429.

Crawler slows or fails.

Google documentation says 429 is treated as server overload signal and crawl rate reduces.

For other bots, exact backoff behavior depends.

OpenAI doesn't publish universal rate behavior guarantees for every bot in simple crawler docs.

Don't assume.

If rate limit is too aggressive:
false block.

If no rate:
server overload.

Need balance.

Article 285 deep dive.

In WAF audit:
check 429 counts by verified crawler.

Not only 403.

403 and 429 tell different story.

403:
forbidden/security.

429:
too many requests/rate.

5xx:
origin/server.

Each remediation different.

Bot Fight Mode Can Override SEO Intent

Many small sites enable one-click bot protection.

Good for abuse.

But product may classify certain crawler categories.

Some AI crawlers may be considered content bots.

Settings evolve.

Cloudflare now has controls around AI crawlers/content bots and verified bots.

Don't assume “Bot Fight Mode” means same behavior forever.

After enabling security feature:
test Search bots and AI bots important to business.

Release checklist:

Googlebot verified fetch.

OAI-SearchBot if desired.

Ads crawlers if campaigns.

Monitoring.

Security improvement shouldn't silently remove distribution.

Tradeoff.

No universal allow-all.

Just test desired.

CDN Cache Can Serve Different Content to Bot

Sometimes no block.

But bot gets stale/error page from edge.

Cache key differs by UA/country/cookie.

Human:
200 fresh.

Crawler:
200 but challenge HTML.

Or 503 cached.

Status code alone not enough.

Compare response body.

Title.

Content length.

Hash.

Headers.

Cache status.

Request ID.

If crawler sees “Access denied” with HTTP 200:
soft failure.

Search might interpret poorly.

Audit response content.

Not only code.

This is important for bot protection vendors that return branded challenge pages.

Origin log may never show.

WAF Rules Can Match User-Agent Strings Accidentally

Rule:
if user-agent contains "bot", block.

OAI-SearchBot contains Bot.

Googlebot contains bot.

Blocked.

Simple but destructive.

Rule:
if UA contains "GPT", block.
Maybe blocks GPTBot only, maybe future agents.

Don't use crude string matching for allowed official bots.

Use verified source where possible.

Unknown bots:
string can inform, not authenticate.

Security rules should have explicit exceptions.

Unit test.

No production regex adventures.

IP Allowlist Alone Can Also Be Fragile

Vendor ranges change.

If static:
stale.

If too broad:
security risk.

Best:

Official published source.

Periodic sync.

Change review.

Monitor when IP outside list appears.

Do not scrape random community list.

Google offers JSON IP ranges for crawlers.

OpenAI has official JSON links.

Use primary.

If vendor supports reverse DNS verification:
apply.

Don't assume ASN alone enough.

Cloud providers host many actors.

Purpose-specific IP list is stronger.

Staging vs Production WAF

Staging:
protected by Cloudflare Access/auth.

Good.

Production copied WAF zone rules?
Maybe public crawler gets auth page.

Ensure environment policies separated.

Production indexable:
public.

Staging:
private.

Do not loosen staging to bots just for test if sensitive.

Use production URL or controlled crawler simulation.

Security boundary first.

SEO should not request public staging.

Test through verified user-agent from controlled environment cannot replicate source IP identity exactly, so use vendor tools/logs.

Google URL Inspection.

Server logs.

OpenAI crawl observation.

Evidence.

Googlebot Verification

Google warns fake crawlers can spoof user-agent.

Official verification can use reverse DNS or published IP ranges.

If WAF simply allows all `Googlebot` UA:
attacker can bypass.

Same with OAI-SearchBot.

This is why security team resists SEO requests:
“Whitelist UA.”

They're right.

Solution:
verified bot mechanism.

Cloudflare Verified Bots.

IP ranges.

Reverse DNS.

Not blanket UA.

Security and Search goals can coexist.

OpenAI Bot Verification

OpenAI crawler page publishes IP lists separately for SearchBot, GPTBot, ChatGPT-User, AdsBot.

If WAF wants allow Search:
use SearchBot IP list, not all OpenAI ranges.

If training blocked:
no need allow GPTBot.

This preserves purpose.

Do not whitelist all OpenAI because one bot needed.

Principle of least privilege.

Crawler access matrix guides.

Edge Logs Are Gold

Fields:

timestamp.

source IP.

country.

ASN.

user-agent.

host.

path.

method.

status.

WAF rule ID.

action:
allow/block/challenge/skip.

bot score/verified flag if provider.

cache status.

latency.

request ID.

Now incident:
OAI-SearchBot 403.

Look rule ID:
geo-block.

Fix targeted.

No guess.

Without edge logs:
team might edit robots for days.

Infrastructure observability reduces SEO folklore.

A Reproducible Audit

1. Confirm official crawler UA/IP docs.

2. Check robots intended policy.

3. Inspect CDN/WAF rules.

4. Inspect bot product settings.

5. Check geo rules.

6. Check rate limits.

7. Check edge logs for verified crawler.

8. Compare origin logs.

9. Verify response status/body.

10. Test important resources.

11. Document exception.

12. Monitor.

If crawler appears edge but not origin:
edge/cache may serve or block.

If not edge:
DNS/network/vendor crawl timing.

If origin 200:
transport good.

Then Search indexing/retrieval separate.

Layer-by-layer.

Do Not Disable WAF Globally to “Fix AI”

Bad advice.

Security protects:

login.

API.

forms.

checkout.

admin.

abuse.

DDoS.

scrapers.

Credential attacks.

Don't turn off.

Create scoped exception for verified crawler and public GET/HEAD routes.

Keep:

rate limits where reasonable.

sensitive path blocks.

auth.

method restrictions.

No POST access unless integration requires.

Search crawler usually needs public GET.

Least privilege.

Security team should be comfortable.

Search team should be specific.

WAF and Crawlers Are Not Enemies

A mature WAF can distinguish:

Verified Googlebot.

Verified OAI-SearchBot.

Known good monitoring.

Unknown automation.

Malicious bot.

Human.

Policy.

Allow the ones business needs.

Challenge/block unknown abuse.

No need choose:
security or visibility.

The conflict appears when rule design is blunt.

Improve classification.

Not disable defense.

AI Search Claim Scope

If OAI-SearchBot receives 200 after WAF fix:
what can we claim?

Crawler access restored.

Maybe crawl activity increases.

Can content appear in Search if other conditions satisfied.

Cannot claim:
citation will increase.

Same Google:
Googlebot 200 doesn't guarantee indexing/ranking.

Transport is prerequisite.

Not outcome.

Measure downstream.

This boundary protects reporting.

Incident Example

Before:
robots allow.
WAF 403.

After:
verified bot exception.
OAI-SearchBot 200.

Report:
“Transport accessibility fixed.”

Not:
“GEO optimization increased visibility.”

Then run query observation later.

Different metric.

This is professional.

Why Official Crawler Can Still Be Blocked?

Because “official” identity doesn't override your infrastructure.

Your site controls access.

A WAF sees traffic and applies rules.

Unless rule recognizes/excepts it, crawler is just another automated request.

That is healthy security behavior.

The fix isn't to complain that bot should bypass.

The fix is to express policy correctly.

Who should get in?

On which paths?

For what purpose?

How verified?

At what rate?

Once those answers are in WAF/CDN configuration, robots and network stop fighting.

And crawler access becomes predictable instead of mystical.