What is Perplexity?

Perplexity says its current search runs on its own crawling and indexing, with an index that tracks over 200 billion unique URLs; its first answer engine relied on external search APIs^[3]. Its help center adds that it "partners with third-party crawlers to help build our search index"^[2]. No syndication of results from another engine is described for the current system.

What the figures measure: StatCounter's AI chatbot market share, October 2026. StatCounter does not describe how it measures AI chatbot share on the pages read, so no method is claimed here. StatCounter's label is “Perplexity”; this page is about Perplexity. Source: StatCounter Global Stats, licensed CC BY-SA 3.0.

Stance recordedIts pages say nothing about it

None of the Perplexity pages read^[1-5] mentions schema.org, JSON-LD or structured data. The search architecture article describes rule-based parsing of page content instead.^[3]

“Not documented” means the pages read do not say; it does not mean the format is ignored.

The research lists 2 crawler tokens for Perplexity: the names a site uses in robots.txt rules, with what each one does where the source says.

• PerplexityBot

  Search crawler. Shows and links websites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models. The crawler page recommends allowing it; the compliance statement is on the help article^[2] and the search architecture article^[3]. follows robots.txt.^[1]

• Perplexity-User

  User-triggered fetcher. Visits a page when a Perplexity user asks about it. Not used for web crawling or for AI training. may ignore robots.txt (fetches a person asked for).^[1]

The research records no markup testing tool of its own for Perplexity.

Guidelines

robots.txt

Perplexity says it respects robots.txt. PerplexityBot will not index the full or partial text of a site that disallows it, but Perplexity may still index the domain, the headline and a brief factual summary.^[2]

user fetch

Perplexity-User fetches a page because a user asked for it, so the page says this fetcher "generally ignores robots.txt rules".^[1]

timing

Each token is controlled separately in robots.txt. A change may take up to 24 hours to be reflected.^[1]

training

Perplexity says it does not build foundation models, so crawled content is not used for model pre-training.^[2]

third-party crawlers

Perplexity says it updated its agreements so that the third-party crawlers that help build its index also respect robots.txt, in particular on news publisher sites. A feature that summarized a blocked URL at a user's request was disabled.^[2]

content parsing

Pages are parsed with dynamic rule sets that an LLM feedback loop refines, and documents are split into self-contained spans that are retrieved and ranked one by one. Sites with well-structured lists or tables may get more formulaic extraction rules. Schema.org and JSON-LD are not mentioned.^[3]

publisher program

The Publishers' Program shares revenue with a publisher when an answer that earns revenue references its content. Launch partners were TIME, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune and WordPress.com.^[4]

feeds

The Merchant Program lets large retailers share product specifications with Perplexity, and the 2024 post says it is free for merchants. The post does not describe the data format. Buy with Pro was limited to Perplexity Pro users in the US.^[5]

access

To allow the bots through a firewall, match the User-Agent together with the published IP ranges, and automate updates from the JSON files.^[1]

These claims about Perplexity came up in the research, but no page that could be read confirms them. They are kept apart, and nothing above rests on them.

• Perplexity reads schema.org structured data when it indexes or answers.

  Why it is open: No primary statement either way. Recorded as not documented, not as unsupported.

  How it was checked

  Read the crawler page^[1], the robots.txt help article^[2], the search architecture article^[3], the publishers program post^[4] and the shopping post^[5]. None mentions schema.org, JSON-LD or structured data.

• The Perplexity Merchant Program accepts a product feed in a documented format.

  Why it is open: The feed format is not published on a page that could be read without submitting a form.

  How it was checked

  The shopping post^[5] describes the program but not the data format. Sign-up is through a form, which the researcher did not open.

The pages the research used for Perplexity, numbered as cited above. Accessed November 10, 2026 unless shown.

1. 1.Perplexity Crawlers · Perplexity
2. 2.How does Perplexity follow robots.txt? · Perplexity
3. 3.Architecting and evaluating an AI-first Search API (reached by 308 redirect from research.perplexity.ai) · Perplexity · updated September 25, 2025
4. 4.Introducing the Perplexity Publishers' Program · Perplexity · updated July 30, 2024
5. 5.Shop like a pro: Perplexity's new AI-powered shopping assistant · Perplexity · updated November 18, 2024

Market share figures: StatCounter Global Stats, AI chatbot market share, October 2026, licensed CC BY-SA 3.0.