Rate Limiting dan AI Bot: Cara Menghindari False Block

Ada dua cara membuat crawler problem.

Pertama:
tidak punya rate limit sama sekali.

Bot agresif datang.

Server panas.

Database overload.

User ikut kena.

Kedua:
rate limit terlalu ketat.

Crawler resmi datang.

Dua puluh request.

429.

Search crawl melambat atau gagal.

Dua-duanya buruk.

Rate limiting untuk AI bot bukan pilihan antara “allow” dan “block.”

Ia soal membatasi load tanpa membuat official crawler yang memang diinginkan dianggap abusive.

Di 2026, challenge makin menarik karena website menerima:

Googlebot.

OAI-SearchBot.

GPTBot.

ChatGPT-User.

OAI-AdsBot.

Other search crawlers.

Unknown scrapers.

Security scanners.

Humans.

API clients.

Semua punya traffic patterns berbeda.

Satu threshold global sering jadi false-block factory.

Rate Limiting Itu Control terhadap Frequency

HTTP 429 berarti Too Many Requests.

MDN menjelaskan status ini digunakan ketika client mengirim terlalu banyak request dalam periode tertentu, dan server dapat menyertakan `Retry-After` untuk menunjukkan kapan client sebaiknya mencoba lagi.

Google Search documentation juga menganggap 429 sebagai sinyal server overload dan akan mengurangi crawl rate ketika menerima cukup banyak 429/5xx.

Ini platform fact untuk Google.

Untuk AI crawlers lain:
jangan assume exact backoff.

Some may retry.

Some slow.

Some stop.

Read vendor docs/observe.

But universal principle:
429 is correct status to say “slow down,” not 403.

403 means forbidden.

Different semantics.

Jangan Pakai 403 untuk Rate Limit

Google explicitly says don't use 401/403 to limit crawl rate; 429 is the appropriate rate-limit signal.

Why?

403 says:
access forbidden.

429 says:
access allowed in principle, but too frequent now.

If business wants crawler:
use rate semantics.

If business doesn't want crawler:
robots/WAF policy.

Don't blur.

A crawler seeing 403 may interpret permanent denial.

A crawler seeing 429 can treat overload/backoff.

For non-Google bots exact handling varies, but HTTP semantics remain.

Security rules should use proper status.

One Global Threshold Is Usually Wrong

Rule:
50 requests/minute/IP.

Works for human.

Crawler:
legitimately fetches 200 URLs.

Blocked.

API:
needs 100 rpm.

Blocked.

Attacker:
rotates IP.
bypasses.

Global per-IP threshold can hurt good bots more than distributed bad bots.

Better:
segment.

Human web.

Verified Search bots.

Verified AI bots.

Unknown automation.

Login/API.

Expensive paths.

Static assets.

Rate policy per class.

This requires bot verification.

Without verified identity:
don't special-case by UA only.

Fake bot can exploit.

Crawler Access Matrix + Rate Matrix

Extend access matrix:

Agent.

Purpose.

Allowed?

Rate class.

Burst.

Path restrictions.

Verification.

Example conceptual:

Googlebot:
verified crawler class.

OAI-SearchBot:
verified search crawler.

GPTBot:
if disallowed, no rate rule needed except default security.

ChatGPT-User:
user-triggered, moderate normal public access, no bypass private.

OAI-AdsBot:
campaign landing, low expected volume.

Unknown bots:
strict.

This isn't a universal numeric recipe.

Threshold depends:
server capacity.
cache.
page cost.
site size.
crawler demand.

Avoid fake number like “100/min is optimal.”

Measure.

Rate Limits Should Be Capacity-Based

Question:

How many uncached page renders can server sustain without harming users?

Load test.

Suppose origin safely handles X concurrent expensive requests.

CDN absorbs static/cache.

Rate limit protects origin.

Define headroom.

Crawler verified:
allowed burst within capacity.

Unknown:
lower.

Admin/login:
strict.

API:
contract.

Numbers come from performance data.

Not SEO blog.

This is engineering.

Crawler-friendly doesn't mean unlimited.

Server stability has priority.

Use Cache before Rate Limit Where Possible

Public articles:
cache at CDN.

Crawler request served edge.

Origin load minimal.

Now official crawler can crawl more without risk.

This is better than high rate limit on uncached WordPress.

Cache:

HTML.
images.
assets.

Respect personalization.

Don't cache private.

Use stale-while-revalidate where architecture supports.

Performance engineering reduces conflict between security and discoverability.

If each bot request executes 50 database queries:
fix app.

Rate limit is guardrail, not performance strategy.

Path Cost Matters

`/article/`:
cheap cached.

`/search?q=`:
expensive.

`/api/report/generate`:
very expensive.

`/wp-login.php`:
security-sensitive.

Rate limit by route.

Allow crawler:
articles.

Restrict:
internal search.

Block:
admin if not needed.

Bots don't need every endpoint.

Robots can reduce crawl on patterns.

WAF can rate/block abusive expensive paths.

Security.

This prevents official crawler accidentally triggering compute-heavy utility.

Crawler should receive public content.

Not internal functionality.

False Block from IP Concentration

Official crawler may use a limited set of IPs.

Per-IP rate limit sees high request volume.

But that's normal centralized crawler architecture.

A human-based threshold doesn't fit.

Verified class should have separate policy.

Again:
verify official source.

OpenAI published IP ranges.

Google verification.

Do not allow by UA.

If unknown bot claims Googlebot:
default strict.

This protects.

False Block from IPv6 Handling

Some WAF rules aggregate IPv6 incorrectly.

Rate limit per full IPv6 address may be bypassed.

Or per /64 could overblock legitimate users.

Crawler IP ranges can include IPv6 depending vendor.

Use provider bot classification rather than hand-written IP heuristics where available.

For official crawler allow:
published ranges.

Keep current.

Rate limiting architecture should understand IPv4/IPv6.

Don't assume.

Infrastructure responsibility.

Burst vs Sustained Rate

Crawler may burst:

50 requests in 5 seconds.

Then quiet.

Average low.

A simple window:
20/min
blocks.

Token bucket/leaky bucket style rate limiting can allow reasonable bursts while capping sustained load.

Cloudflare rate limiting allows thresholds over periods and actions; other providers similar.

Choose based traffic.

Again:
no universal number.

Observe logs.

Plot requests/second.

Peak.

Duration.

Origin CPU.

Design.

False block is visible in 429 spikes.

Use data.

`Retry-After`

HTTP 429 may include `Retry-After`.

Good protocol hygiene.

But don't assume every crawler obeys exactly.

Google rate behavior documented generally.

Other bots:
unknown.

Still send sensible header if rate limiter supports.

It communicates.

For API clients:
important.

For crawlers:
potentially helpful.

Never set absurd retry:
one year
unless truly deny.

If permanent deny:
403/robots.

Semantics.

Rate Limiting Googlebot

Google's current crawling docs say crawling infrastructure reduces crawl rate when significant numbers of URLs return 500, 503, or 429.

Google deprecated its old Search Console crawl-rate limiter tool because crawling logic automatically responds to site conditions.

If urgent short-term reduction:
Google docs recommend 500/503/429 for short periods.

But repeated errors can affect crawling/indexing.

Don't deliberately 429 Googlebot as long-term crawl-budget strategy.

Better:
server health.
URL cleanup.
crawl efficiency.

Rate limit only when capacity protection needs.

Long-term intentional blocking:
robots.

Different.

Rate Limiting OAI-SearchBot

OpenAI publishes crawler identity and IPs but does not, in simple crawler documentation, give a universal webmaster rate-limit number.

So don't invent.

If SearchBot load high:

Verify identity.

Check paths.

Check robots.

Check cache.

Measure.

Set reasonable threshold.

Observe 429/retries.

If persistent problem:
contact/support channel if available.

No folklore:
“OpenAI requires X requests/sec.”

Unless official doc says.

GEO article must resist fake precision.

Rate Limiting GPTBot

If policy disallows GPTBot:
robots is primary documented training crawler control.

Why rate limit?
Maybe safety fallback for noncompliant/spoofed traffic.

But don't use rate limit as training opt-out.

A throttled crawler still crawls.

Business intent:
no training.
Use disallow.

Infrastructure intent:
prevent overload.
Use rate limit.

Two controls.

If allowed:
same capacity-based policy.

Don't confuse purpose.

ChatGPT-User Is Different

User-triggered.

A user may ask ChatGPT to read several pages.

Rate limit too strict:
user experience fails.

But do not whitelist unlimited.

Especially API/GPT Actions:
security requires contract.

Public GET:
normal bot-aware rate.

Authenticated action:
API rate.

No security bypass by UA.

A malicious bot can spoof ChatGPT-User.

Use official IP plus authentication where action matters.

User-triggered doesn't mean trusted to do anything.

Rate limit and authorization both.

OAI-AdsBot Expected Volume Low but Campaign-Critical

It visits submitted ad pages.

If landing page validation receives 429:
campaign may have issue.

No need huge rate allowance.

But ensure rule doesn't treat verified AdsBot as unknown scraper.

Paid media launch:
test.

If OAI-AdsBot 429:
adjust narrow path/class.

Do not disable rate limiting entire site.

Least privilege.

Unknown AI Scrapers

They may crawl aggressively.

No official identity.

Rotate.

Ignore robots.

Here rate limiting/bot protection crucial.

Use:

WAF bot scores.

behavior.

IP reputation.

ASN.

path pattern.

rate.

challenges.

Block.

Don't give “AI crawler exception” based on UA.

Verified vendors get specific path.

Unknown automation default restricted.

Security posture.

Good.

429 vs Challenge

Browser human:
challenge can work.

Search crawler:
challenge often fails.

If verified crawler exceeds rate:
429 preferable to interactive CAPTCHA.

A CAPTCHA page might return 200 but crawler cannot solve.

Worse diagnostic.

For known crawler:
skip interactive challenge.
apply rate limit.

For unknown:
challenge/block.

Separate actions.

Cloudflare verified-bot patterns can help.

Exact config depends plan/product.

Test.

False Block Detection

Dashboard:

Verified crawler.
Requests.
429%.
403%.
5xx%.
2xx%.
p95 latency.

Alert:
429 ratio spikes.

After WAF change:
compare.

If OAI-SearchBot 429 from 0.5% → 35%:
likely false block.

If server CPU stable:
threshold too low.

If CPU pegged:
rate limit protecting.
Need performance/crawl policy.

Data tells.

Don't blindly increase.

False block and real overload can look same from crawler side.

Origin metrics decide.

Server Health Correlation

Plot:
bot RPS.
CPU.
memory.
DB connections.
queue.
cache hit.
5xx.

If crawler burst correlates with saturation:
optimize/rate.

If no saturation but 429 due generic rule:
false limit.

This is observability.

SEO alone can't decide.

Infrastructure data.

Cross-team.

A crawler shouldn't get unlimited access because visibility team wants citations.

User performance is primary.

Graceful Rate Strategy

For desired verified crawler:

1. Serve cache.

2. Allow public cheap paths.

3. Disallow useless expensive URL spaces.

4. Use reasonable burst.

5. Use 429 when capacity limit reached.

6. Send Retry-After if supported.

7. Monitor.

8. Adjust.

For unknown:
strict bot mitigation.

This is more resilient than:
allow all official bots unlimited.

Search crawlers can generate load.

Business needs balance.

Do Not Block by Country as a Rate Shortcut

“Bot traffic mostly US, block US.”

Now Google/OpenAI crawlers gone.

Wrong.

Rate control should target behavior/identity, not audience geo unless actual policy.

Article 287 covers geo-block.

But in rate context:
country is poor proxy for abusive bot.

Use verified bot classification.

Path.

rate.

security signals.

Indonesia site still needs global crawler access if Search desired.

Test.

Rate Limiting and Crawl Budget Are Different Concepts

Crawl budget:
search engine's crawl demand/capacity concept.

Rate limiting:
server-enforced request control.

Don't say:
“Set rate limit 100 to optimize crawl budget.”

No.

You might reduce crawl, but not intelligently improve demand.

For Google, better:
avoid duplicate URLs.
fast server.
sitemaps.
proper status.
canonical.
internal links.

Rate limit is protection.

Not SEO tuning knob.

For AI crawlers:
same principle.

Use policy/discovery, not rate limit as ranking strategy.

No Citation Guarantee

If you remove false 429 and SearchBot crawls more:
direct outcome:
crawl access improved.

Maybe Search coverage improves.

Citation?
not guaranteed.

Content still must be relevant.

AI system may choose another source.

So report:

Before:
verified SearchBot 429 rate 40%.

After:
1%.

Crawl success:
improved.

Then separately:
citation observations.

This is strong evidence.

No need claim more.

Incident Playbook

Crawler overloading:

1. Verify identity.

2. Check whether crawler desired.

3. Check path.

4. Check cache.

5. Check origin health.

6. Apply temporary 429 if needed.

7. Add robots restriction for useless paths.

8. Optimize.

9. Monitor recovery.

False block:

1. Verify crawler.

2. Inspect 429 rule.

3. Compare server capacity.

4. Create verified-class exception/threshold.

5. Keep sensitive routes protected.

6. Deploy.

7. Monitor.

This is operational.

No panic.

No global WAF disable.

Rate Limit Is a Conversation between Server and Client

Server says:
“Too fast.”

Client decides:
slow, retry, stop.

Good protocol uses 429.

But the website must know which clients it wants.

A legitimate Search crawler and a credential-stuffing bot can both make 100 requests.

Intent and identity differ.

So rate limiting cannot be only arithmetic.

It needs classification.

Policy.

Path cost.

Capacity.

Verification.

And monitoring.

The goal isn't zero 429.

Sometimes 429 is exactly right.

The goal is zero unnecessary 429 to crawlers you intentionally want, while keeping enough protection that automated traffic never makes the website worse for actual users.

That's the balance.

Not “AI-friendly.”

Infrastructure-friendly, user-first, and precise.