Robots.txt Besar dan Rumit: Cara Menjaganya Tetap Bisa Diaudit

Ada robots.txt yang bisa dibaca dalam dua menit.

Ada yang terlihat seperti configuration language hasil tiga generasi SEO, developer, security, agency, dan plugin.

Ratusan baris.

User-agent lama.

Bot yang sudah tidak ada.

Rule duplicate.

Path typo.

Comments tahun 2022.

`Crawl-delay` untuk Google.

Wildcard tidak jelas.

Staging rule yang ikut production.

GPTBot ditulis dua kali dengan keputusan berbeda.

OAI-SearchBot belum ada.

Dan di bawah sekali:

`User-agent: *`
`Disallow: /`

Satu rule bisa meniadakan seluruh niat di atas.

Robots.txt besar bukan otomatis buruk.

Website besar memang punya banyak path dan crawler policy.

Yang buruk adalah file yang tidak bisa diaudit manusia.

Kalau engineer membutuhkan 40 menit untuk menjawab:

“Apakah OAI-SearchBot boleh crawl `/research/`?”

governance sudah gagal.

Robots.txt harus diperlakukan seperti configuration code:

Readable.

Minimal.

Versioned.

Tested.

Owned.

Comments meaningful.

No dead rules.

Dan memahami constraint consumer.

Google misalnya menetapkan limit robots.txt 500 KiB; content setelah batas itu diabaikan. Google juga hanya mendukung field tertentu seperti user-agent, allow, disallow, dan sitemap, sementara `crawl-delay` tidak didukung.

Jadi kompleksitas bukan hanya maintenance issue.

Bisa menjadi functional risk.

Start dari Policy Matrix, Bukan Edit File Langsung

Before:

Googlebot:
public yes.

OAI Search:
public yes.

GPTBot:
training no.

AdsBot:
campaign landing yes.

Unknown:
default.

Admin:
no crawler.

Internal search:
no.

Assets:
needed rendering yes.

Now generate/hand-write concise rules.

If policy changes:
matrix first.

robots second.

This prevents config becoming source of business truth.

The file is implementation.

Policy elsewhere.

Otherwise comment:
“block GPTBot”
may remain after legal changes.

No owner.

Drift.

Group by Purpose, Not Historical Addition

Bad order:

# added Jan
GPTBot.

# added Feb
Google.

# agency rule
wildcard.

# new AI
OAI.

# old Bing.

Better:

# Search crawlers.

# Training crawlers.

# Ads validators.

# Common public-path exclusions.

# Sitemap.

Robots parser groups by user-agent semantics, so exact grouping must follow standard behavior.

But humans can still structure comments.

Keep categories.

Don't duplicate same agent in multiple ambiguous groups unless you understand merge semantics of target crawler.

Google combines groups for same user-agent token according to its documented matching behavior.

Complex file needs deliberate parser expectations.

One Agent, One Intent Block

Example:

# OpenAI Search
User-agent: OAI-SearchBot
Allow: /
Disallow: /internal-search/
Disallow: /preview/

# OpenAI training
User-agent: GPTBot
Disallow: /

Clean.

If same GPTBot appears again 300 lines later:
risk.

Lint duplicates.

Maybe duplicate groups intentionally combine for Google-like behavior, but other crawlers can vary.

Simpler:
one block per bot where practical.

No contradictions.

Humans can read.

Do Not List Every Bot in the Internet

A common robots file:

AhrefsBot.
SemrushBot.
MJ12.
DotBot.
ClaudeBot.
GPTBot.
CCBot.
PetalBot.
Yandex.
100 more.

Maybe intentional.

But every line creates maintenance.

If default policy is:
unknown crawl allowed,
why list 80 allow?

Unnecessary.

If default:
unknown blocked at WAF,
robots doesn't need all.

Only include bots requiring specific policy.

Robots.txt isn't encyclopedia.

Use security platform for hostile/unknown.

Use explicit groups for business-important compliant crawlers.

This reduces size.

Auditability.

Wildcard Policy Should Be Clear

`User-agent: *`

is powerful.

Decide what it means.

Often:
default crawl public except known low-value paths.

Then specific agents override according their group semantics.

But don't assume specific group inherits wildcard rules in every crawler the way you imagine.

For Google, crawler uses most specific user-agent group and matching rules; wildcard group may not combine with specific group as naive inheritance.

So if you need exclusions for all, duplicate/generate appropriately or verify target crawler behavior.

This is why robots config should be tested per important crawler.

Don't write CSS mental model:
wildcard base + overrides.

Read specification.

Complexity hides semantic assumptions.

Google Does Not Support `crawl-delay`

Robots files often contain:

`Crawl-delay: 10`

Team believes Google crawl controlled.

Google official robots spec says Google doesn't support crawl-delay.

So line adds false confidence.

For Google crawl-rate issues:
use server health/status guidance, URL architecture, Search crawling behavior.

For other bots:
some may support crawl-delay, check docs.

Don't put unsupported directive expecting universal.

If line exists for Bing/another:
comment.

Who consumes?

If nobody:
remove.

Each directive should have consumer.

Unsupported rules are configuration debt.

500 KiB Is a Hard Google Constraint

Google docs current:
robots.txt content after 500 KiB ignored.

Most sites nowhere near.

But generated enterprise robots can.

Examples:
per-product disallow.
per-query rules.
thousands bots.
automation dumps.

If file 600 KiB:
bottom 100 KiB effectively irrelevant to Google.

What if critical:
`Disallow: /private-render/`
at bottom?

Google ignores due size.

This is why huge per-URL rules are bad.

Consolidate by directories.

Fix URL architecture.

Noindex/canonical where appropriate.

WAF for abuse.

robots should express patterns.

Not every URL.

Monitor file size in CI.

Alert at lower internal threshold.

No need wait 500 KiB.

Robots.txt Must Be Valid UTF-8 Plain Text

Google says file should be UTF-8 plain text.

Invalid characters/lines can be ignored.

Generated file should not return HTML template.

Common incident:

WordPress security intercepts `/robots.txt`.

Returns 200 HTML challenge.

Google tries to parse rules from response but intended policy gone.

Or PHP fatal returns HTML.

Need monitor:

Content-Type.

body.

status.

size.

hash.

Robots availability is production health.

A correct repo file means nothing if CDN serves another response.

Fetch external.

Comments Are for Humans, Use Them

Example:

# GPTBot blocked per training policy LEGAL-2026-08
# OAI-SearchBot allowed for ChatGPT Search
# /preview/ excluded because draft render

Good.

Not:

# don't touch this
# important SEO
# new rule

Comments explain intent.

Ticket ID optional.

Date maybe.

But don't clutter every line.

Policy repository has detail.

Robots comment gives quick context.

No secrets.

robots is public.

Don't put:
internal usernames.
incident details.
private paths unnecessarily.

Comments should be safe.

Robots.txt Is Public Documentation of Some Architecture

Disallow:

`/secret-client-x/`

Now you've advertised path.

Security through obscurity fails.

Sensitive path should auth.

Maybe still disallow crawler, but naming exposes clue.

Design paths without leaking sensitive identities.

No comments:
“private M&A deal.”

Never.

Treat robots as public.

Audit comments for disclosure.

This is overlooked.

Generate Rules from Route Classes, Not Individual URLs

CMS has:

/search/
/preview/
/admin/
/cart/
/checkout/
/account/
/filter/

Write patterns.

Avoid:

Disallow: /search?q=1
Disallow: /search?q=2
…

Robots path matching can use supported wildcard patterns depending crawler; Google supports `*` and `$` in URL path matching.

Use cautiously.

Test.

A simple directory is better.

If URL architecture impossible:
fix app.

robots complexity often signals routing complexity.

Don't solve every crawl issue in config.

Rule Minimization Exercise

Take file.

For each rule:

What crawler?

What business purpose?

What path problem?

Still exists?

Supported?

Can route be consolidated?

Is WAF better?

Is noindex better?

Is authentication required?

If no clear answer:
candidate remove.

Do not remove production rules blindly.

Stage/test.

Changelog.

But dead-rule cleanup makes file understandable.

Robots archaeological layers accumulate.

Annual cleanup.

After site migration.

Plugin removal.

Crawler policy update.

Specific vs Wildcard Precedence Needs Tests

Don't rely memory.

Google documentation describes matching by most specific user-agent and longest path rule, with allow winning if allow/disallow patterns have equal length under its behavior.

Other crawler implementations may follow REP but edge details differ.

For business-important bot:
test intended paths.

Build unit tests:

OAI-SearchBot `/article/` → allowed.

OAI-SearchBot `/preview/` → blocked.

GPTBot `/` → blocked.

Googlebot `/article/` → allowed.

Googlebot `/search/` → blocked.

Assertions.

Use parser compatible with target behavior where possible.

And verify live logs.

Config test.

No manual eyeballing only.

Robots Testing Can Be CI

On deploy:

Fetch generated robots.

Check status 200.

UTF-8.

Size < internal threshold.

No duplicate conflicting blocks.

No `Disallow: /` under wildcard unless explicitly approved.

Required bot blocks exist.

Staging/prod differ.

Sitemaps correct host.

No old domain.

No forbidden directives.

Then deploy.

This prevents catastrophic rule.

A site can lose crawl due one line.

CI cheap insurance.

Robots is code.

Review pull request.

No WordPress UI random edit.

Version Control Is Non-Negotiable for Complex File

Git history:

who.

when.

diff.

reason.

Rollback.

If marketing says:
“ChatGPT Search disappeared last week.”

Check robots diff.

OAI Search blocked?
Maybe.

If not:
look elsewhere.

Without history:
can't.

Complex files absolutely need.

Even small robots benefit.

No need expose repo.

Internal.

If CMS dynamically generates:
version source/template.

Store output snapshot.

Monitoring Hash

Fetch `/robots.txt` hourly/daily depending criticality.

Hash.

Expected.

If changes unexpectedly:
alert.

Plugin or security tool may modify.

WordPress virtual robots can change from plugin settings.

CDN injection.

Compare.

Search/GEO team should know.

No need page all night for harmless sitemap update.

Severity classification.

Wildcard `Disallow: /`:
critical.

GPTBot policy change:
legal review.

SearchBot block:
visibility.

Hash + semantic diff.

Not just bytes.

Robots and Sitemap

`Sitemap:` lines can point full URL.

Keep current.

HTTPS.

Canonical host.

No old staging.

If multiple:
fine.

Don't list 200 if sitemap index exists.

Sitemap discovery can also happen elsewhere; directive supported by Google.

Robots file can include.

But sitemap line doesn't override disallow.

An URL in sitemap but blocked creates mixed intent.

Audit.

Sitemap:
we want discover.

Robots:
don't crawl.

Maybe intentional for URL known but no crawl?
Usually question.

Resolve.

Robots and Noindex Audit Together

Even though different files/directives.

For each template:

Robots crawl allowed?

Meta noindex?

Canonical?

If:
blocked + noindex:
noindex unreadable to Google.

If:
allowed + index:
normal.

If:
allowed + noindex:
public no-search.

Don't analyze robots alone.

Big file often accumulated to “deindex” paths that should use noindex/canonical/status.

Move correct controls.

Robots shrinks.

Better semantics.

Robots and Training Policy

Explicit blocks:

GPTBot.

Other documented training crawlers.

Keep comments.

Don't use wildcard unless business intends.

Search crawler separately.

OpenAI docs current make this clear.

If new training bot:
matrix.

Then file.

Don't add based rumor.

Each new user-agent requires:
official docs.
purpose.
verification.
owner.

Otherwise robots becomes AI bot zoo.

Not sustainable.

Robots and Ads

Campaign landing may live under `/promo/`.

If wildcard disallows `/promo/`, OAI-AdsBot blocked.

Ads review issue.

Complex file makes this hard to see.

Unit test campaign paths against AdsBot.

Paid media can register active landing paths.

No manual carve-out after rejection.

But don't keep endless old campaign allow rules.

After campaign:
clean.

Lifecycle.

Complexity grows when temporary exceptions never expire.

Add expiry ticket.

Robots for Multi-Environment

Production:
crawl policy.

Staging:
Disallow all as secondary plus authentication.

Development:
private.

Don't share same generated file accidentally.

Environment variable.

CI assert.

One classic disaster:
staging config deployed prod.

`Disallow: /`.

Audit environment.

Exact hostname.

Google robots rules are host/protocol/port specific.

A robots file for one host doesn't automatically govern another.

So each domain/subdomain needs correct.

Multi-domain brand article 299 will go deeper.

Readable Robots Standard

Internal standard:

Under 200 lines if possible, not universal.

Grouped comments.

No duplicate agent blocks.

No dead bots.

No unsupported directives without comment.

No individual URLs unless exceptional.

No secrets.

Versioned.

Tested.

Policy reference.

Sitemap at bottom/top consistent.

Size monitored.

Why no universal 200 line?
Large sites differ.

The goal isn't arbitrary count.

A human can answer policy quickly.

Audit time matters.

Measure:

Can engineer answer 10 key path-agent combinations in 5 minutes?

If no:
simplify.

That is a better KPI than line count.

A Good Robots.txt Is an Executive Summary of Crawl Policy

It shouldn't encode every security concern.

WAF handles abuse.

Auth handles private.

Noindex handles indexing.

Canonical handles duplicates.

Sitemap helps discovery.

robots handles compliant crawl access.

When robots tries to do every job, it becomes huge.

The way to keep it auditable isn't better comments alone.

It's putting problems in the correct layer.

Once private content moves to auth, duplicate URLs to canonical, no-search pages to noindex, and malicious bots to WAF, robots shrinks naturally.

Then each remaining line has a clear purpose.

That's the real cleanup.

Not compression for its own sake.

Clarity.