Crawler Policy untuk Multi-Domain Brand

Brand besar jarang hidup di satu domain.

Corporate:
brand.com.

Indonesia:
brand.co.id.

Shop:
shop.brand.com.

Support:
support.brand.com.

Docs:
docs.brand.com.

Investor:
investor.brand.com.

Legacy:
oldbrand.com.

Campaign:
brand2026.id.

Partner microsite.

Subsidiary domain.

Setiap host bisa punya robots.txt sendiri.

Setiap host bisa punya CDN/WAF sendiri.

Setiap host bisa punya crawler policy berbeda tanpa ada yang sadar.

Inilah kenapa multi-domain brand membutuhkan crawler governance di level brand, bukan domain-by-domain improvisation.

Google robots.txt rules bersifat host/protocol/port specific. Satu robots.txt tidak otomatis mengatur host lain.

Jadi:

`https://brand.com/robots.txt`

tidak mengontrol:

`https://shop.brand.com/`

atau:

`https://brand.co.id/`

Masing-masing perlu policy.

Untuk AI crawlers juga, request diarahkan ke host tertentu dan host tersebut menentukan response/config.

Satu brand.

Banyak pintu.

Crawler Policy Harus Dimulai dari Domain Inventory

List:

Host/domain.

Purpose.

Owner.

Platform.

Environment.

Public/private.

Search goal.

Training policy.

WAF/CDN.

Canonical relationship.

Sitemap.

Status.

Example:

brand.com:
corporate.
public.
Search yes.
training no.

brand.co.id:
local market.
Search yes.
training no.

shop.brand.com:
commerce.
Search yes.
training maybe no.
Ads yes.

support.brand.com:
public docs.
Search yes.
training policy.

investor.brand.com:
public.
Search yes.

staging.brand.com:
private auth.

oldbrand.com:
redirect only.

Now governance.

Without inventory:
old campaign domain may accidentally allow GPTBot.
support domain blocks SearchBot.
corporate WAF geo-block.

Central policy catches.

One Legal Training Policy, Banyak Implementasi

Leadership says:
“No GPTBot training across brand.”

But each domain has separate infrastructure.

WordPress corporate.

Shopify commerce.

Zendesk support.

Custom docs.

Cloudflare.

Akamai.

If policy only deployed brand.com:
incomplete.

Need rollout across all public hosts under scope.

Robots.

Vendor controls.

WAF.

Verify.

Logs.

Exception if content license differs.

Maybe open-source docs allow training.

Then policy not uniform.

Document.

Don't assume central brand decision applies blindly.

Content rights can differ.

But if policy uniform:
audit coverage.

Domain inventory is compliance surface.

Search Policy Bisa Berbeda

Corporate:
Google/Search yes.

Support:
yes.

Internal status:
maybe yes.

Campaign:
maybe noindex.

Partner microsite:
depends.

Training:
maybe all no.

AdsBot:
only landing hosts.

ChatGPT Search:
maybe public.

One crawler access matrix can have rows per domain class.

No need copy identical robots if purpose differs.

But core naming/comments consistent.

Central template plus local overrides.

This balances governance and autonomy.

Subdomain vs Separate Domain Still Needs Separate Robots

Google's robots docs say robots.txt applies to protocol, host, and port where served.

So:

`www.example.com` vs `example.com`
can have separate robots behavior if both hosts active.

Best:
canonical host redirect.

Don't maintain two public versions.

But subdomains:
distinct.

`docs.` needs own.

`shop.` own.

Don't assume root file inherited.

This mistake common.

Audit each host.

HTTP/HTTPS too.

In modern sites HTTP should redirect, but robots behavior during redirects/crawl can matter.

Keep consistent.

WAF Can Differ per Domain Even if Robots Same

brand.com:
Cloudflare.

support:
SaaS vendor.

shop:
Commerce CDN.

OAI-SearchBot gets:

200 corporate.
403 support.
429 shop.

Brand says:
“ChatGPT crawler allowed.”

Only partly.

Crawler success must be measured per host.

Edge logs available differently.

Maybe support SaaS doesn't expose logs.

Then test.

Vendor docs.

Allowlist request.

No one dashboard automatically.

Create health table.

Multi-domain GEO visibility can fail at weakest source domain.

If AI wants product support facts but support host blocks, corporate page may not substitute.

Content role matters.

Canonicalization Across Domains Harus Benar

Corporate vs country site may be distinct content.

Don't canonical all country domains to global.

Translations/local variants use international SEO strategies.

Separate legal entities.

Offers.

Locations.

Need own canonical.

Legacy duplicate content:
cross-domain canonical/redirect may be appropriate.

Campaign copy duplicate:
maybe canonical.

Syndication:
agreement.

Crawler policy cannot fix canonical fragmentation.

But domain migration must coordinate.

If old domain still 200 and crawlable:
search/AI can keep discovering.

If moved:
redirect.

Update external profiles.

Robots shouldn't block old domain before crawlers can see redirect; blocking could prevent processing.

Migration sequence matters.

Don't use robots to “kill” legacy domain.

Redirect.

Training policy separate.

Legacy Domain Is a Risk

Old domain may have forgotten:

robots allow all.

outdated content.

stale company name.

expired TLS.

malware risk.

redirect broken.

search index.

AI discovery through backlinks.

Include in governance until fully retired.

If still controlled:
redirect.

if not:
brand risk.

Domain ownership retention/security.

Crawler policy of legacy host is part of information hygiene.

A multi-domain brand can't only audit current homepage.

The internet may cite old.

Subdomain Takeover Risk Is Separate but Relevant

Unused:
campaign.brand.com.

DNS points old SaaS.

Potential takeover.

Crawler finds.

Search surfaces.

Security issue.

Not crawler policy.

But domain inventory reveals.

Remove DNS.

Control host.

Robots can't protect abandoned subdomain.

Multi-domain governance should involve security.

Crawler audit often uncovers forgotten infrastructure.

Good.

Don't make robots solve.

Entity Identity Should Connect Domains

Organization canonical identity.

Official domains.

Country subsidiaries.

Brands.

Products.

Docs.

A public entity hub can clarify.

sameAs curated.

But sameAs all domains only if they represent same entity identity appropriately.

A support subdomain is part of same web presence, not necessarily sameAs target needed.

Internal graph can map.

Organization `url` official primary.

Other URLs relationships.

Don't dump 20 domains in sameAs because multi-domain.

Semantic precision.

Crawler policy and entity mapping complement.

Not same.

SearchBot Policy Across Domains

If brand wants ChatGPT Search:

OAI-SearchBot should access relevant public source domains.

Corporate alone may not be enough.

Product:
shop.

Support:
docs.

Research:
lab.

Local:
country.

OpenAI docs say non-blocking SearchBot helps content inclusion in summaries/snippets.

So make a source map.

Which host contains authoritative answer categories?

Allow there.

If training opt-out:
GPTBot disallow per host.

No assumption one robots propagates.

Test each.

Logs.

Citation remains not guaranteed.

But access aligned.

GPTBot Training Policy Across SaaS Platforms

Some hosted platforms let custom robots.

Some don't.

Maybe custom headers.

Maybe vendor-level policy.

Need platform support.

If company policy requires block and SaaS cannot:
escalate vendor.

Maybe move content.

Maybe contractual route.

Don't assume technical control available everywhere.

Document gap.

Risk register.

This is governance maturity.

A checkbox on corporate WordPress doesn't satisfy brand-wide policy if important support content lives elsewhere.

Ads Crawler Only Needs Relevant Domains

OAI-AdsBot:
landing pages submitted.

If ads destination shop:
allow shop.

Corporate no campaign:
no special AdsBot exception necessary.

Google Ads crawlers similarly per campaign.

Don't create brand-wide WAF bypass.

Narrow.

Multi-domain can actually make least privilege easier.

Campaign host:
specific.

But ensure redirects through tracking domain also accessible.

Third-party tracking domain may be outside your direct crawler policy.

Coordinate vendor.

Crawl path includes all hops.

Geo-Blocking May Differ Per Country Domain

brand.co.id:
non-ID block.

brand.sg:
SG only.

Crawler outside both blocked.

Search visibility suffers.

If public country sites need global crawl:
verified crawler exception per WAF.

Do not centralize into:
“allow all verified bots everywhere”
if legal restrictions differ.

Country sites may have licensing.

Policy matrix.

Business owner local.

Central standard:
how to verify.

Local decision:
what allow.

Federated governance.

Sitemaps Per Host

Google sitemap files generally list URLs under appropriate host constraints, with cross-site submission mechanisms under verified ownership in certain setups.

Keep simple:

Each public host has sitemap/index.

Robots references.

Search Console properties.

No staging URLs.

No cross-environment leak.

Multi-domain brand should monitor sitemap health centrally.

Which domains:
indexable count.

errors.

canonical.

This supports Search/AI Search.

No special AI sitemap needed for Google's AI features.

Don't invent.

For OpenAI, no official requirement for special sitemap in crawler overview.

Normal web discovery.

Cross-Domain Analytics

ChatGPT referral can land on any host.

OpenAI says referrals can include `utm_source=chatgpt.com`.

Analytics must consolidate across domains.

Otherwise:

Corporate 100.
Shop 50.
Support 20.

Teams each think partial.

AI visibility at brand level requires:

Citation domain.

Landing domain.

Query class.

Country.

But don't merge all sites' Search performance blindly.

Each domain role.

A support citation can be success for technical query.

Corporate for company query.

Shop for product.

Source role.

Domain-level dashboards plus brand rollup.

Robots Change Management

Central repository:

domain.
policy version.
robots hash.
WAF state.
last checked.
owner.

When legal changes GPTBot:
generate tickets.

When OpenAI adds bot:
review all.

When new domain launched:
crawler policy mandatory.

No domain goes live without matrix.

When domain retired:
redirect/security.

This is scalable.

Manual spreadsheet may suffice.

Enterprise:
configuration management.

Important:
no forgotten host.

A Multi-Domain Audit

For each host:

DNS resolves?

TLS?

HTTP final status?

robots 200?

OAI-SearchBot policy?

GPTBot?

Googlebot?

noindex defaults?

canonical host?

sitemap?

WAF?

geo?

rate?

verified crawler logs?

Search Console?

owner?

content role?

legacy?

Then classify:

PASS.

CONFLICT.

UNKNOWN.

PRIVATE.

RETIRED.

Don't force pass for private.

Staging should be blocked/auth.

Crawler governance is not maximize access.

It is intentional access.

Policy consistency means matching purpose.

Not same rule everywhere.

The Biggest Multi-Domain Mistake Is Assuming “Brand Policy” Automatically Exists Technically

A slide says:

“We allow AI Search but block training.”

Great.

Then:

Corporate follows.

Shop unknown.

Support vendor default.

Country domain blocks all foreign.

Legacy allows everything.

Reality does not match slide.

Governance closes gap.

The website estate is the policy surface.

Every public domain is another place a crawler can learn something.

Every forgotten domain is another source of contradiction.

So multi-domain crawler strategy begins with inventory, not robots syntax.

Know every door.

Know what is behind it.

Know which machine you want to let through.

Then make sure the same business decision is actually implemented on every door where it applies.