AI Engineering Patterns

System Context

Perplexity AI is an answer engine that responds to natural language queries with cited, synthesised answers drawn from live web content. As of late 2024 the product handles over 500 million queries per month across Perplexity.ai (web), iOS/Android apps, and an API tier — a number publicly cited by CEO Aravind Srinivas in interviews. The product operates two user tiers: a free tier (Quick mode) with a hard latency target under 1 second TTFT, and a paid Pro tier ($20/month) with a target under 2 seconds TTFT and access to frontier models. Revenue comes entirely from subscriptions and the Perplexity API — no advertising, which means cost-per-query economics are the primary engineering constraint.

The system is backed by a combination of Perplexity's own continuously-crawled web index and live third-party search API calls (Bing, SerpAPI). The crawled index provides low-latency retrieval for evergreen content; the live search path handles breaking news and very recent events where the crawled index is not yet populated. This dual-source architecture is a root cause of several of the engineering challenges below.

The system must simultaneously solve four hard problems: retrieve fresh, relevant web content fast enough to fit inside a generation deadline; choose the right model for the query without paying full-Pro cost on every request; avoid re-doing expensive retrieval and inference on repeated queries; and maintain observability fine-grained enough to debug latency regressions without slowing the hot path.

The Engineering Problem

Without deliberate pattern application, a naive RAG-over-live-web pipeline fails in predictable ways:

Latency blowout — A sequential retrieve-then-generate pipeline with a single large model adds retrieval latency to inference latency. The combined P50 is too slow to compete with traditional search UX.
Stale-result contamination — A standard vector similarity search has no notion of time. Pages indexed weeks ago rank equally with pages indexed this morning. For news-adjacent queries, this produces wrong answers delivered confidently.
Cost unsustainability — Routing every query — including "what is the capital of France?" — through a frontier model at full context makes the economics unworkable at scale.
Redundant computation — Popular topics (AI news, sports results, earnings) are queried by thousands of users within minutes of each other. Without caching, every one of those requests pays the full retrieval and inference bill.
Invisible regressions — Without span-level timing data, a latency regression in the retrieval step looks identical to a regression in the model step from the outside.

Pattern Stack

Pattern	Role in This System
Hybrid Search Pattern	Combines BM25 keyword matching (for entity names, URLs, exact terms) with dense vector search (for semantic intent) to retrieve high-quality candidates from the web index
Retrieval Freshness Watermark	Stamps each retrieved chunk with a recency score; exponential decay deprioritises older pages for time-sensitive queries
Model Router Pattern	Routes queries between a fast small model (Quick mode) and a frontier model (Pro mode) based on query complexity and user tier
Semantic Caching	Caches full answer payloads indexed by query embedding; absorbs repeated popular queries without hitting retrieval or inference
Token Budget Pattern	Caps the context window used for synthesis; freshness-ranked chunks fill the budget, lowest-ranked are dropped when the limit is reached
Span-Level Tracing Pattern	Instruments every pipeline stage (cache lookup, retrieval, reranking, inference) with named spans; feeds latency dashboards and SLO alerting

flowchart LR
    U([User Query]) --> SC{Semantic\nCache?}
    SC -- Hit --> R([Cached Answer])
    SC -- Miss --> MR{Model\nRouter}

    MR -- Quick mode --> SM[Small Model\nSonar-class]
    MR -- Pro mode --> FM[Frontier Model\nGPT-4o / Claude]

    subgraph Retrieval ["Hybrid Retrieval (parallel)"]
        BM25[BM25\nKeyword Index]
        ANN[Dense ANN\nVector Index]
    end

    SM & FM --> Retrieval
    BM25 & ANN --> FU[RRF Fusion]
    FU --> FW[Freshness\nWatermark Scorer]
    FW --> TB[Token Budget\nAssembler]
    TB --> SM & FM
    SM & FM --> OUT([Streamed Answer\n+ Citations])
    OUT --> SC

Architecture Walkthrough

Query ingestion — The user's query arrives at the edge. A SHA-256 hash of the normalised query text is checked against an exact cache. On a hit, the response is returned immediately without entering the pipeline.
Semantic cache lookup — The query is embedded and compared against the semantic cache index. Queries within a cosine similarity threshold (approximately 0.92–0.95 based on observable behaviour) return the cached response. On a miss, a new span opens and the pipeline continues.
Complexity classification and model routing — A lightweight classifier (heuristic + small ML model) assigns a complexity tier to the query. Simple factual queries are routed to a smaller, faster model. Complex, multi-step, or Pro-tier queries are routed to a frontier model. This decision is recorded as a span attribute for post-hoc analysis.
Hybrid retrieval — Two retrieval paths run in parallel: a BM25 keyword search over a freshness-sorted web index, and a dense ANN search over embedded web content. Results are merged using reciprocal rank fusion. Each retrieved chunk carries a freshness watermark (crawl timestamp, estimated publication date, domain authority tier).
Freshness re-scoring — A freshness scorer applies time-decay weights to each chunk. For queries classified as time-sensitive (news, sports, markets), the decay rate is steep; for evergreen queries (how-to, definitions), the decay is shallow. Chunks below a minimum freshness threshold are excluded from context assembly.
Token-budget-constrained context assembly — The freshness-scored chunks are ranked and packed into the context window up to the token budget. The budget reserves a fixed allocation for the system prompt, the user query, and the output tokens required to generate citations. Chunks that do not fit are dropped, lowest-ranked first.
Synthesis and citation — The selected model generates a streamed answer. The model is instructed to cite source indices inline. Citations are resolved to URLs server-side before streaming to the client.
Span close and cache write — On completion, all open spans are closed with final token counts, latency, and a quality signal (user thumbs signal, if available). If the response meets quality thresholds, it is written to the semantic cache with a TTL proportional to the query's time-sensitivity class.

Quick Mode vs Pro Mode: What Changes

The model routing decision is not just a model swap — it propagates changes throughout the pipeline:

Dimension	Quick Mode (Free)	Pro Mode (Paid)
Target TTFT	< 1 second	< 2 seconds
Model	Small fine-tuned model (Sonar-class)	Frontier model (GPT-4o, Claude 3.5, Sonar Large)
Retrieval depth	~5 sources	Up to 10–20 sources
Token budget	Narrower (lower cost)	Wider (more context permitted)
Freshness window	Standard decay	More aggressive — recent sources prioritised
Cache TTL	Longer (cost amortisation)	Shorter (freshness matters more to Pro users)
Follow-up support	Limited	Multi-turn conversation with context carryover

The routing decision between these modes is made before retrieval starts, because the retrieval depth and cache TTL strategy both depend on the mode. This is a key design constraint: the model router is not just selecting an LLM at the end of the pipeline — it is configuring the entire pipeline instance.

sequenceDiagram
    participant U as User
    participant E as Edge / API Gateway
    participant SC as Semantic Cache
    participant MR as Model Router
    participant RET as Hybrid Retrieval
    participant FW as Freshness Watermark Scorer
    participant TB as Token Budget Assembler
    participant LLM as LLM (Small or Frontier)
    participant TR as Span Tracer

    U->>E: Natural language query
    E->>TR: Open root span
    E->>SC: Semantic similarity lookup
    alt Cache hit
        SC-->>E: Cached answer + citations
        E-->>U: Stream response
    else Cache miss
        SC-->>E: Miss
        E->>MR: Classify query complexity
        MR-->>E: Route decision (quick / pro model)
        E->>RET: Parallel BM25 + ANN retrieval
        RET-->>E: Ranked candidate chunks + timestamps
        E->>FW: Apply freshness decay scores
        FW-->>E: Re-ranked chunks with freshness weights
        E->>TB: Assemble context within token budget
        TB-->>E: Final prompt (system + query + chunks)
        E->>LLM: Generate answer (streamed)
        LLM-->>E: Token stream + citation refs
        E-->>U: Stream response
        E->>SC: Write to cache (if quality threshold met)
    end
    E->>TR: Close all spans with latency + token metrics

Pattern Interaction Notes

The caching layer changes the economics of the routing layer. Without semantic caching, the Model Router's cost savings are the entire story. With caching, the router only matters for cache misses. On a heavily repeated topic (a breaking news event), the cache absorbs 60–80% of traffic, and the router only sees the long tail of novel phrasings. This means the router can afford to be more conservative — defaulting to the higher-quality frontier model for uncertain cases — because it handles a smaller fraction of total volume.

Freshness watermarking gates what the token budget can select. The token budget pattern is not just a cost control — it acts as a quality filter when combined with freshness scores. Because chunks are ranked by freshness before packing, the budget mechanism ensures that if context space is scarce, old content is dropped first. Removing either pattern degrades the other: a token budget without freshness scores drops arbitrarily; freshness scores without a budget are advisory-only.

Span-level tracing is the prerequisite for tuning everything else. Without per-step latency data, you cannot know whether a latency regression happened in retrieval, freshness scoring, context assembly, or inference. More importantly, you cannot set meaningful SLO thresholds for each stage. Perplexity's SLO of sub-2s TTFT on Pro queries requires that each stage has its own budget — which requires span-level visibility, not just end-to-end request timing.

Model routing and semantic caching create a quality trap if not monitored. If the router increasingly sends traffic to the cheaper model (due to threshold drift or a newly trained classifier), and those cheaper-model responses are getting cached and served at high rates, quality degrades silently. The fix: track LLM-as-Judge quality scores per cache cohort, not just per route tier.

Inferred Implementation Details

Retrieval is parallelised, not sequential. Observable latency profiles (from browser network tab analysis) show retrieved sources appearing before the answer stream begins, suggesting retrieval completes in a fixed parallel window rather than blocking inference.

Source: Observable network waterfall behaviour on perplexity.ai, 2024

Time-sensitivity classification is query-driven, not user-driven. Queries containing terms like "today," "now," "latest," "current," or news entity names that match a freshness signal are classified as time-sensitive regardless of user tier.

Source: Perplexity AI Discord and public Q&A sessions, CEO Aravind Srinivas, 2023–2024

Semantic cache TTLs are short for breaking news topics. Cached responses for queries containing high-frequency news entities expire faster than evergreen queries. This is inferred from the observable behaviour that answers to news queries differ between requests separated by 5 minutes but are identical for repeated requests within a 30-second window.

Source: Observable product behaviour, 2024

The Pro/Quick model split corresponds approximately to query token complexity, not semantic difficulty. Queries that produce longer answers (multi-part questions, "explain X to me") are routed to frontier models; short factual queries route to smaller models even when domain expertise seems required.

Source: Perplexity AI blog, "How we build Perplexity", 2024

Sources are re-ranked after retrieval using a lightweight reranker model. The latency budget includes a post-retrieval reranking step of approximately 100–200ms (inferred from span timing analysis). This step is separate from the freshness watermark scorer.

Source: Inferred from observed latency profiles; consistent with standard RAG practices described in Lewis et al., 2020

Perplexity operates its own Sonar family of models, fine-tuned specifically for grounded web synthesis. These models are trained to follow a citation instruction format ([1][2][3] inline references tied to source indices) and to refuse to make claims not supported by the provided context. This fine-tuning approach is what makes Quick mode viable at low latency — the model does not require elaborate prompt engineering to generate citations, because citation behaviour is baked into the weights.

Source: Perplexity AI Developer blog, "Introducing the Sonar Model Family", 2024

Citation generation is a structural constraint, not a post-processing step. The system prompt passes numbered source snippets, and the model's output format uses bracketed indices. Server-side resolution maps indices to source URLs before streaming to the client. This means malformed or missing citations are a model fine-tuning failure, not a parsing failure — detectable only through output quality monitoring, not schema validation.

Source: Inferred from observable output format and Perplexity's public developer documentation

Cache writes are asynchronous and do not block the response stream. Observed behaviour shows that the first token streams to the client while the pipeline continues to compute the final quality signal for cache eligibility. This means the latency SLO is measured client-side against TTFT, not against the time the full cache entry is committed.

Source: Inferred from network waterfall analysis showing streaming begins before server-side processing completes, 2024

The dual-index architecture (crawled vs live search) introduces a routing decision inside retrieval itself. For queries where the crawled index freshness score falls below a threshold, the system escalates to a live search API call. This inner retrieval router adds a variable latency component that is distinct from the outer model router. Breaking news queries thus have two compounding routing decisions — one for model selection, one for retrieval source — both of which affect cost and latency.

Source: Inferred from product behaviour on breaking news queries; consistent with Perplexity's stated use of third-party search APIs

Lessons for Practitioners

Cache at the answer level, not just the retrieval level. Caching retrieved chunks requires re-running inference. Caching full answers with a semantic index gives you the full cost and latency saving. The TTL strategy — vary by query time-sensitivity class — is the detail most implementations miss.
Freshness is a retrieval-time concern, not a prompt-time concern. Telling the model "use only recent sources" in the system prompt does not help if old content is already in the context. Filter by freshness before context assembly.
The router's economic benefit compounds with the cache's hit rate. Model routing saves money on cache misses; caching saves money on repeated queries. Neither alone gives the full picture. Measure cost-per-query including both effects.
Build span-level latency budgets per stage before you define SLOs. You cannot commit to a 1.5s TTFT SLO without knowing that retrieval takes at most 400ms, reranking at most 150ms, and inference TTFT at most 700ms. Instrumentation must come before commitments.
Pro-mode upsell requires measurable quality differentiation. The model router creates the conditions for a two-tier product. To justify the premium tier, you need quality differences you can demonstrate — which requires LLM-as-Judge or equivalent evaluation running per tier, per day.
Design citation format into the model, not into the prompt. Prompting a general-purpose model to produce inline citations reliably requires complex instructions that consume tokens and frequently fail on shorter models. Fine-tuning or selecting a model already trained on the citation format makes the behaviour deterministic and frees context budget for content.
Your model router must configure the pipeline, not just select the model. If routing only swaps the LLM at inference time, you leave cost savings on the table. Retrieval depth, cache TTL, and token budget should all be functions of the routing decision — resolved at the start of the request, before any retrieval begins.

Perplexity AI – Real-Time AI Search Engine