Cara Audit Log Server untuk Melihat Aktivitas AI Crawler

Kalau robots.txt adalah policy, log server adalah bukti operasional.

Di file config kita bisa menulis:

OAI-SearchBot allow.

GPTBot disallow.

Googlebot allow.

Unknown bot challenge.

Tapi yang benar-benar terjadi baru terlihat di logs.

Apakah crawler datang?

Dari IP mana?

Path apa?

Status code?

Seberapa sering?

Diblok WAF atau sampai origin?

Apakah response 200?

429?

403?

5xx?

Apakah user-agent resmi atau spoof?

Tanpa log, crawler governance terlalu mudah berubah menjadi asumsi.

“Kayaknya SearchBot sudah masuk.”

“Sepertinya GPTBot berhenti.”

“Googlebot aman.”

“AI bot banyak.”

Semua terdengar plausible.

Tidak cukup.

Audit log server harus mengubah kalimat itu menjadi data yang bisa diuji.

Tapi audit crawler bukan sekadar grep user-agent.

Di stack modern, request bisa berhenti di CDN sebelum origin.

Bisa served dari cache.

Bisa datang dengan UA palsu.

Bisa lewat IPv6.

Bisa melewati reverse proxy.

Bisa menghasilkan response yang berbeda tergantung country/rule.

Jadi pertama-tama kita perlu tahu:

Log yang mana?

Map Request Path Dulu

Contoh:

Internet
→ Cloudflare/CDN
→ WAF
→ load balancer
→ Nginx
→ PHP/WordPress.

Possible logs:

CDN edge log.

WAF security events.

Load balancer access log.

Nginx access log.

Application log.

If request blocked at WAF:
Nginx mungkin tidak pernah melihat.

If served from CDN cache:
origin log bisa tidak ada.

If app returns 500:
edge log dan origin log mungkin sama-sama record.

Audit harus memilih level sesuai pertanyaan.

Question:
“Apakah OAI-SearchBot mencoba akses?”

Edge log.

Question:
“Apakah request sampai WordPress?”

Origin log.

Question:
“Rule mana block?”

WAF event.

Question:
“Content apa yang dikirim?”

Response/body tracing or sampled fetch.

Jangan mengandalkan satu log untuk semua.

Field Minimum yang Harus Ada

Ideal access log:

Timestamp.

Client IP.

Forwarded IP.

User-Agent.

Host.

Method.

Path.

Query string.

Status.

Response bytes.

Referer.

Latency.

Request ID.

Cache status.

WAF action if edge.

Country/ASN if available.

TLS/protocol optional.

Why these matter?

IP:
verification.

UA:
claimed identity.

Path:
crawl focus.

Status:
access outcome.

Bytes:
did crawler receive real page or tiny error?

Latency:
load.

Request ID:
trace across layers.

Query:
crawler traps.

Without fields:
analysis limited.

If current log format only has path/status:
upgrade.

But privacy/minimization matters.

Don't log secrets/query tokens unnecessarily.

Verify Crawler before Counting It

A log line:

`Mozilla… OAI-SearchBot …`

is not proof official OpenAI.

User-agent can be spoofed.

OpenAI publishes IP ranges for OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot.

Google publishes crawler verification guidance and IP ranges/reverse DNS methods.

So classify:

Verified.

Unverified.

Do not simply count all UA matches.

Example:

2,000 requests claim GPTBot.

1,850 outside official IP ranges.

Maybe spoof/scraper.

If dashboard says:
“GPTBot 2,000,”
misleading.

Better:
Verified GPTBot: 150.
Unverified GPTBot-like: 1,850.

Security consequences differ.

Reporting trust improves.

User-Agent Token vs Full String

OpenAI docs note versions in full user-agent strings may change.

So parse token.

OAI-SearchBot.

GPTBot.

ChatGPT-User.

OAI-AdsBot.

Don't require exact version substring.

Likewise Googlebot families.

Maintain parser.

But verification still IP/DNS.

UA token identifies claimed class.

Network verifies official source.

Two-factor crawler classification, conceptually.

Start with One Day, Then Expand

For first audit:
24 hours can reveal.

But crawler cadence may be intermittent.

Better:
7–30 days depending site.

Do not conclude:
“Bot never comes”
from two hours.

For large site:
one day enough to detect many crawlers.

For small site:
weeks.

Record window.

Timezone.

Indonesia:
WIB.

Logs often UTC.

Normalize.

If compare to robots change:
exact timestamps critical.

A robots update on August 9 10:00 WIB corresponds 03:00 UTC.

Without normalization, causal timeline wrong.

Basic Query: Requests per Bot

Counts by verified crawler.

Example dashboard:

Googlebot: X.

OAI-SearchBot: Y.

GPTBot: Z.

ChatGPT-User: A.

OAI-AdsBot: B.

Unknown automation: C.

Don't need publish raw numbers externally.

Internal trend.

Then split:

2xx.

3xx.

4xx.

429.

5xx.

Now health.

A bot with 10,000 requests and 95% 403:
not healthy.

A bot with 100 requests 99% 200:
better.

Volume alone isn't visibility.

Status Distribution adalah First Diagnostic

200:
success, but verify body.

301/302:
redirect.

304:
cache conditional response.

401:
auth required.

403:
forbidden/WAF.

404:
not found.

410:
gone.

429:
rate limited.

5xx:
server problem.

For Google, official docs treat 429 as overload signal and reduce crawl rate.

For other crawlers, behavior may vary.

So don't generalize backoff.

But 429 clearly means your infrastructure denied due rate.

Audit.

A 200 can still be wrong:
challenge page with HTTP 200.

So status is first layer.

Content/body sampling second.

Top Paths Tells Crawler Behavior

Group by path.

Are bots crawling:

Articles.

Sitemap.

robots.txt.

Feeds.

Search results.

Parameters.

API.

Images.

PDFs.

Admin.

404.

If GPTBot spends 70% on `?filter=` infinite URLs:
crawler trap.

If SearchBot only robots/sitemap and no content:
maybe blocked/discovery limited.

If Googlebot heavily crawls old faceted pages:
crawl waste.

Top-path audit informs robots/canonical/internal link cleanup.

Don't interpret all high crawl as good.

Useful crawl:
important content.

Waste:
infinite variants/errors.

Distinct URLs and Query Explosion

Crawler can hit:

`/article?id=1&utm=x`

`?replytocom=`

filters.

calendar.

sort.

session.

nocache.

Count distinct normalized URLs.

Then count query parameters.

If one content creates 5,000 variants:
fix.

Robots/canonical.

Internal links.

Parameter handling.

Rate limit is symptom treatment.

URL space is root.

Crawler logs reveal architecture bugs.

For AI bots, same principle.

No need platform-specific internal knowledge.

HTTP is HTTP.

Robots Fetch Itself Is Important

Does bot fetch `/robots.txt`?

Status?

200?

Content correct?

If robots returns:

403.

5xx.

HTML challenge.

redirect loop.

Different crawlers may behave differently.

Google has explicit rules for robots fetch status handling.

For OpenAI, don't assume all details.

But serve standard valid robots reliably.

Log fetches.

After policy change:
see crawler retrieve new robots?

OpenAI says Search robots changes can take approximately 24 hours.

Log helps.

If CDN caches old robots:
policy delayed.

Set sane cache.

Invalidate after update.

Sitemap Requests

Crawler hits sitemap.

Good discovery signal.

But OAI crawler docs may not promise sitemap behavior in same way Google does.

Don't interpret absence as failure.

Googlebot sitemap use is documented.

AI crawler:
observe.

If SearchBot requests sitemap:
record.

If not:
normal unknown.

Do not optimize around one anecdote.

Log observation should stay observation.

Referrer Is Usually Weak for Crawlers

Crawler requests may not send useful Referer.

Don't rely.

User referral from ChatGPT is different.

OpenAI says Search referrals use `utm_source=chatgpt.com`.

That is actual human/browser referral.

Bot request:
server crawl.

Don't combine.

Analytics:

Crawler log.
Referral traffic.
AI citation observation.

Three datasets.

A SearchBot fetch doesn't mean user clicked.

A user click doesn't necessarily correlate one-to-one with crawler fetch.

Keep separate.

ChatGPT-User Should Be Separate Segment

Because user-triggered.

Do not combine with OAI-SearchBot.

Log class:

search_crawler.

training_crawler.

user_triggered_agent.

ads_validator.

classic_search.

unknown.

This matches function.

Then metrics by class.

ChatGPT-User 403 may mean user cannot ask ChatGPT to read public page.

OAI-SearchBot 403 affects Search crawl.

Different remediation.

Same status.

Purpose matters.

Edge vs Origin Diff

This is powerful.

Edge logs:
1,000 OAI-SearchBot requests.

Origin:
300.

Why?

700 cache hits?
Fine.

700 WAF blocks?
Bad.

Need action field/cache status.

If edge 200 cache:
crawler got content, origin absent.

If edge block:
crawler didn't.

Compare.

Do not require every bot request reach WordPress.

CDN is designed to serve cache.

The success criterion:
valid response.

Not origin hit.

This avoids false diagnosis.

WAF Rule IDs

If provider logs rule ID:
aggregate.

Example:

403 by:
geo rule 70%.
bot challenge 20%.
IP deny 10%.

Now fix exact.

No need disable WAF.

Create exception.

Security review.

After deploy:
403 falls.

This is measurable.

A crawler access project can show direct outcome:

verified OAI-SearchBot 2xx rate:
before/after.

That's more defensible than claiming citation.

Transport metric.

Good.

429 Audit

Count by bot/path/hour.

Burst pattern.

Example:
SearchBot gets 429 after 20 requests/min.

Googlebot maybe adapts.

Unknown AI bot hammering:
rate limit desired.

Official crawler:
maybe threshold too low.

Do not exempt all verified bots unlimited.

Server capacity.

Cache.

Path cost.

Rate limit policy.

Article 285.

Logs give empirical basis.

Latency

Crawler may get 200 but 20 seconds.

Timeout/retry.

Server load.

Measure p50/p95 response time for bot vs normal.

If bots hit uncached expensive pages:
optimize.

Search crawl infrastructure may reduce load automatically under errors/slow response depending platform.

Don't rely.

Faster public pages help humans too.

Caching.

Database.

Avoid heavy per-bot dynamic rendering.

No cloaking.

Same content.

Performance.

Log Sampling vs Full Retention

Huge site:
billions lines.

No need forever.

Store raw for short operational window.

Aggregate longer.

Retention policy depends:

Security.

Privacy.

Compliance.

Cost.

Crawler analysis.

Article 297 covers retention.

For current audit:
ensure enough window to compare.

30 days often useful, but not universal rule.

Don't invent legal retention number.

Consult policy.

Mask/anonymize where appropriate.

Crawler IP itself may be business telemetry, but logs can include human IP too.

Privacy controls.

Build a Simple Parser

You can use:

BigQuery.

Splunk.

Elastic.

Cloudflare Logpush.

AWS Athena.

Shell/Python for smaller logs.

No need expensive “AI crawler platform.”

Pipeline:

ingest.
normalize.
classify UA.
verify IP.
aggregate.

Important:
verification data updated.

OpenAI IP JSON.

Google IP ranges/DNS.

Cache results.

Do not external DNS verify every request in real time.

For Google reverse DNS:
batch or use published ranges.

Performance.

But implementation details depend stack.

Don't overcomplicate.

Sample SQL Logic Conceptually

Select:
crawler_class,
status,
count,
distinct_path.

Group.

Filter verified.

Then:
top paths.
error rates.
hourly trend.

No need article include literal code.

Purpose is audit method.

If handing to infra:
spec.

Make reproducible query.

Save.

Next month compare.

Crawler Monitoring Baseline

First week:
baseline.

Requests/day.

2xx rate.

403.

429.

5xx.

Top paths.

Response bytes.

Latency.

Then policy change.

Compare.

If after WAF exception:
2xx increases.

Direct evidence.

If citation later changes:
separate observation.

Don't conflate.

The baseline turns technical SEO into operations.

Screenshot is less useful than log table.

Bots that Don't Identify Honestly

Unknown scrapers may rotate UA/IP.

Crawler audit won't perfectly classify.

Use security bot management.

Behavioral signals.

ASN.

Request pattern.

But don't label as GPTBot without verification.

If UA spoof:
unverified.

If invisible/headless:
unknown automation.

The goal isn't identify every bot in internet.

The goal:
verify important official crawlers and detect abusive automation.

Precision over completeness.

Anomaly Alerts

Useful alerts:

Verified Googlebot 403 spike.

OAI-SearchBot 403 > threshold.

GPTBot appears despite policy? Check robots behavior, maybe robots fetch/allowed path/spoof.

Crawler 5xx spike.

429 spike.

Robots returns non-200.

Sitemap 5xx.

Huge parameter crawl.

AdsBot blocked during active campaign.

No need alert every request.

Threshold based baseline.

Avoid pager fatigue.

Security vs SEO Shared Dashboard

Security wants:
blocks.
abuse.
unknown bots.

SEO wants:
crawl success.
Search bots.
status.

Marketing:
ads bot.

Legal:
training policy.

One dashboard can have views.

Same logs.

Different purpose.

This reduces cross-team argument.

“Why did you block?”

Rule ID shows.

“Is it official?”

Verification shows.

“Should we allow?”

Matrix shows policy.

Audit becomes evidence-driven.

Don't Try to Infer Training from Log Alone

GPTBot fetch indicates training crawler access.

OpenAI says content crawled by GPTBot may be used to improve/train models.

But you cannot determine from one fetch whether specific content entered a particular model training set.

Don't claim.

Logs show crawler behavior.

Not downstream model usage.

Likewise SearchBot fetch doesn't prove citation.

Keep scope.

This is crucial for public report.

Operational observation vs causal outcome.

A Good Crawler Audit Produces Five Answers

1. Who is visiting?

Verified identity.

2. Why?

Documented purpose.

3. What do they access?

Paths.

4. What response do they receive?

Status/body.

5. Does that match policy?

Matrix.

If yes:
system controlled.

If no:
fix.

The rest is optimization.

No AI mysticism.

Server logs are boring, messy, and extremely valuable because they tell us what actually crossed the network.

In an era where everyone debates what AI “might” be crawling, the log gives a better starting point:

A timestamp.

An IP.

A user-agent.

A path.

A response.

From there, we can stop guessing.