Bagaimana AI Crawler Menemukan URL dari Sumber Pihak Ketiga

Tim mengubah robots.txt.

OAI-SearchBot diblok pada satu page.

Beberapa hari kemudian, URL itu masih muncul dalam sebuah konteks ChatGPT.

Reaksi pertama mudah ditebak.

“Berarti robots.txt diabaikan?”

Belum tentu.

Ada perbedaan besar antara:

Crawler mengambil content page.

System mengetahui bahwa sebuah URL ada.

System menampilkan link atau judul URL yang diketahui dari sumber lain.

OpenAI sendiri menjelaskan nuance ini secara resmi. Dalam FAQ publisher/developer current, OpenAI menyatakan bahwa jika mereka memperoleh URL sebuah page yang disallow dari third-party search provider atau dari crawling halaman lain, dan ada signal bahwa URL tersebut relevan untuk query user, mereka dapat menampilkan hanya link dan page title dalam ChatGPT Atlas.

OpenAI lalu menyarankan penggunaan `noindex` jika publisher tidak ingin URL muncul dengan cara tersebut, dengan catatan crawler harus diizinkan crawl page agar dapat membaca meta tag itu.

Ini contoh sangat bagus tentang bagaimana crawler dapat “menemukan URL” tanpa berarti ia berhasil mengambil seluruh content dari URL tersebut.

Discovery dan Crawl Bukan Hal yang Sama

Discovery:
System mengetahui URL.

Crawl:
System fetch URL.

Index/retrieval representation:
System memproses content atau metadata untuk penggunaan tertentu.

Citation:
System memilih source dalam answer.

Empat stage.

Sumber discovery bisa banyak.

Internal link.

Sitemap.

External link.

Third-party search provider.

Redirect.

Canonical reference.

Feed.

Social profile.

Directory.

User-submitted URL.

Browser-agent action.

Crawler page lain yang menyebut URL.

Tidak semua route berarti target page langsung dicrawl.

Google juga menjelaskan prinsip serupa pada web crawling secara umum. Googlebot menemukan URLs melalui pages yang sudah dicrawl dan melalui sitemap. Bahkan page yang diblok robots dapat tetap diketahui dan URL-nya bisa muncul di Search jika page itu ditautkan dari tempat lain, walaupun Google tidak dapat membaca content page yang diblok.

Jadi:
“Bot tahu URL”
tidak sama dengan
“Bot membaca content.”

External Link Adalah Discovery Channel yang Paling Mudah Dipahami

Media menulis:

“Lihat laporan lengkap di https://brand.id/report-q3/”

Search crawler membaca media.

Sekarang system tahu URL report.

Even if target report is temporarily blocked, URL string exists.

The web is a graph.

URLs travel through links.

A website does not own every occurrence of its URL on the internet.

Partner.

Directory.

Social media.

Press release.

Archive.

Forum.

Search result.

All can expose.

This is why robots.txt cannot be described as:
“hide URL from internet.”

It controls compliant crawl access on your host.

Not mentions elsewhere.

Sitemap Juga Bentuk Explicit Discovery

Sitemap says:
“These URLs exist.”

If you put URL in sitemap while disallowing its crawl:
mixed operational intent.

Maybe deliberate.

Often mistake.

Google documentation says sitemap can help discover URLs but does not guarantee crawl/index.

For AI crawlers, sitemap support varies by vendor.

Do not assume every bot uses sitemap identically.

But if crawler fetches sitemap:
URL discovery possible.

So when auditing:
check sitemap.

A page you “hid” might still be advertised there.

If page should not be Search destination:
remove from sitemap according architecture.

If page public/noindex:
maybe sitemap exclusion makes sense because index not desired.

Exact policy depends platform.

Don't use sitemap as privacy boundary.

Canonical Tags Can Expose URLs Too

Duplicate page A:

canonical → B.

Crawler fetches A.

Now it knows B.

That's intended.

Canonical helps identify representative URL.

But it means URL B can be discovered from A even if B has few internal links.

Again:
discovery edge.

If B is supposed to be secret:
canonical is wrong architecture.

Private content shouldn't be referenced publicly.

Canonical is public signal.

Structured data `url`, `mainEntityOfPage`, `sameAs`, and other URL properties can likewise expose URLs.

Don't hide a resource in JSON-LD thinking “user can't see.”

Source is public.

Crawlers can parse.

Machine-readable links are still links.

Third-Party Search Provider adalah Jalur yang OpenAI Akui Terang-Terangan

This point is unusually explicit.

OpenAI publisher FAQ says URL of a disallowed page can be obtained from a third-party search provider.

That means ChatGPT ecosystem may know web URLs not exclusively through OAI-SearchBot fetching target page itself.

Do not extrapolate beyond documentation.

We should not claim:
“OpenAI uses provider X.”

No official source here.

We should not claim:
“all content comes from third parties.”

No.

The supported statement:
a URL can be obtained from third-party search provider.

Then only link/title may surface in the documented Atlas scenario if target is disallowed and relevant.

This is why evidence must distinguish:

URL presence.

Snippet/content use.

Citation.

One screenshot can blur.

Server Logs Help Determine the Route

Suppose URL appears.

Check logs:

Did verified OAI-SearchBot fetch target?

When?

Status?

Did it fetch robots?

Was page previously allowed?

Any ChatGPT-User request?

Any user-triggered action?

No target fetch:
URL may have been discovered elsewhere.

Target fetch before block:
could be stale prior crawl.

Target fetch after block:
verify IP, syntax, timing, cache.

This is technical investigation.

No need speculate.

Robots changes may take time to propagate.

OpenAI indicates Search robots changes can take around 24 hours to adjust.

Record timeline.

If page was crawled yesterday and blocked today, system may retain previously known metadata for some period; exact retention/refresh behavior is not fully public, so don't invent.

Say unknown.

Internal Links Can Reintroduce “Hidden” URLs

Page removed from navigation.

But article from 2024 still links.

Crawler finds.

Footer.

Author archive.

XML sitemap.

Related content plugin.

Search endpoint.

RSS.

Old PDF.

A site has many discovery surfaces.

If objective:
reduce crawl/discovery of retired URL.

Audit all.

Return 301/410/404 according page lifecycle.

Do not depend only robots.

If moved:
301 to successor.

If gone permanently:
410/404.

If duplicate:
canonical/redirect.

If public but no Search:
noindex.

Different.

Crawler discovery audits often reveal content lifecycle debt.

Backlinks Mean Retired URL Can Survive Years

External site links old URL.

You cannot edit all.

So old URL remains discoverable.

Best response:
server-side status.

301 if equivalent replacement.

410/404 if removed.

Don't return 200 “not found” soft page.

Google uses HTTP status in processing.

AI crawlers may also interpret standard HTTP semantics, though exact behavior varies.

The web remembers URLs through links.

Govern URL lifecycle, not discovery fantasy.

A retired URL can stay known while no longer serving old content.

That's normal.

URL Knowledge Doesn't Equal Current Content Knowledge

This distinction protects reporting.

If ChatGPT shows:

Title:
“Old Pricing 2024”

Link:
old URL.

Maybe it learned title previously.

Maybe third-party listing.

Maybe search index.

Maybe current fetch.

We need evidence.

Can we fetch current target ourselves?

What does bot get?

Logs?

Citation content?

Don't claim model has current page content because URL visible.

A link is not proof of fetch.

Similarly, brand mention isn't proof owned domain was crawled.

Could come from media.

Entity memory.

Search.

User context.

AI visibility audit must classify source.

This is why article 206 separated mention/citation/recommendation.

Crawler discovery belongs technical evidence.

Social Platforms and Directories Are URL Amplifiers

Brand profile:

Website field.

Company directory.

App store.

Marketplace.

Industry association.

Conference speaker bio.

Every profile can expose official domain.

This can help entity/discovery.

But don't spam directories for “AI crawling.”

Quality and relevance matter.

The primary benefit:
public identity consistency and links.

Whether specific AI crawler uses particular directory depends system.

Don't promise.

If official URL changes:
update high-value external profiles.

Otherwise old domain keeps circulating.

This is part of rebrand cleanup.

Third-party URL discovery can preserve old identity.

Multi-Domain Brands Need Extra Care

Brand has:

brand.com.

brand.co.id.

shop.brand.com.

docs.brand.com.

oldbrand.id.

Partners link all.

Crawler may discover every host.

If canonical/redirect/entity relationships unclear:
system sees fragmented sources.

Article 299 covers policy.

For discovery:
map external mentions.

Which domain is authoritative for what?

Corporate.

Commerce.

Docs.

Country.

Legacy.

Don't assume crawler chooses same.

Use redirects and canonical identity where appropriate.

Do not canonical distinct regional sites incorrectly.

Discovery is easy.

Disambiguation is harder.

What If You Truly Don't Want URL Public?

Then don't publish publicly.

Authentication.

Remove public links.

Remove sitemap.

No public structured references.

If accidentally exposed:
security incident process.

robots/noindex cannot make a public resource secret.

Noindex asks search systems not to index.

Robots asks crawler not to fetch.

A person with URL can still access if 200.

Third-party can re-link.

Security must use authorization.

This is recurring because crawler controls are not privacy controls.

URL discovery on the open web is inherently hard to prevent.

Control access, not obscurity.

User-Triggered Agents Add Another Discovery Path

OpenAI ChatGPT-User is used for certain user actions.

A user can provide URL directly.

Even if automatic Search crawler doesn't discover through normal crawl, a user can ask an agent to visit public URL.

OpenAI says ChatGPT-User isn't automatic Search crawl and robots may not apply in same way because requests are user-initiated.

So:
“SearchBot blocked” doesn't mean “no OpenAI product can ever make an HTTP request to public URL.”

Different access mode.

If page must not be accessible:
auth.

If page public:
user can share.

This is common-sense once access modes are separated.

Search Engines Are Discovery Graphs

Google's sitemap documentation says crawlers generally discover URLs by following links from previously crawled pages; sitemap helps larger/new sites.

This is basic web architecture.

AI Search products may combine their own crawlers, search providers, search indexes, and user-triggered requests depending vendor.

Don't overgeneralize implementation.

But discovery is naturally multi-source.

A URL exists in a network of references.

The crawl of one host is only one edge.

This is why robots does not erase graph knowledge.

A Better Investigation Checklist

When URL appears despite target crawler block:

1. What exactly appeared?
URL only?
Title?
Snippet?
Quote?
Citation?

2. Which product/surface?
ChatGPT Search?
Atlas?
Another?

3. When robots changed?

4. Did verified target crawler fetch before/after?

5. Was robots itself fetched?

6. Is page noindex?

7. Can crawler access noindex?

8. Does external search show URL?

9. Are there backlinks?

10. Sitemap?

11. Redirect/canonical?

12. User-triggered requests?

Then interpret.

This prevents sensational claim:
“robots ignored.”

Maybe true bug.
Maybe not.

Evidence.

What Should GEO Report?

Strong wording:

“The URL remained discoverable through external references after target page crawl was restricted.”

If supported.

“OpenAI documentation states disallowed URLs may still be surfaced as link/title in Atlas when obtained via third-party search provider or other crawled pages.”

Supported.

Weak/unsupported:

“OpenAI secretly crawled through another bot.”

No evidence.

“AI scraped our page despite block.”

Need logs/content proof.

“Robots cannot control AI.”

Too broad.

Robots controls compliant automatic crawlers for documented purposes.

Scope.

Technical credibility comes from careful language.

Discovery Is Not a Failure

If website wants Search visibility:
external discovery is good.

Backlinks.

Profiles.

Sitemaps.

Internal links.

They help URLs get found.

If website wants exclusion:
same mechanisms need cleanup.

No contradiction.

The web is built to connect.

Crawler governance is about controlling what a crawler may do when it reaches your host, not making the wider web forget that a URL once existed.

Once that is understood, “URL ditemukan dari pihak ketiga” stops sounding mysterious.

It is simply how a linked web behaves.