A team ships its first production RAG pipeline. It works. Then a user complains that the answer takes eight seconds, and ChatGPT gives comparable answers in two. The team assumes they need a bigger model, or a faster inference provider, or a different LLM entirely. They're almost always wrong about the diagnosis. The model isn't your problem. Your pipeline is. Profile before you tune The first thing to do is get numbers. Put a simple timer wrapper around each pipeline stage and log to stdout. After a hundred queries, you have a distribution. A typical RAG pipeline looks like this, with numbers I've seen repeatedly on OpenAI, Anthropic, and Google-hosted embedding + completion APIs: - Embed the user query: **800ms - 1,500ms** (OpenAI `text-embedding-3-small` or `-large`, single call) - Vector store search: **200ms - 2,000ms** (depends heavily on whether it's warm, sized right, and hosted in the same region) - Reranker call: **400ms - 800ms** (if using Cohere rerank or a cross-encoder model) - LLM generation (non-streamed): **2,000ms - 6,000ms** (Claude Sonnet or GPT-4o, ~300 token response) - Overhead: **200ms - 500ms** (serialization, logging, the surprising amount of time your own framework takes) Add those, and you're at 4-10 seconds. That's the envelope. ChatGPT's web UI feels faster because OpenAI doesn't do most of these steps — their retrieval is lighter, their embeddings are pre-warmed at a massive scale, and they stream the response from the first token. You can close most of the gap without changing your model. Here are five fixes that consistently moved the needle on real pipelines. Fix 1: Cache embeddings The single largest win for most pipelines is embedding caching. Query embeddings are deterministic for the same input text. If two users ask the same question, you should not re-embed. An in-process LRU cache of size 10,000 keyed on the normalized query string hits 30-50% cache rate on typical support-chat and docs-search workloads. Hit cost: **~0.5ms**. Miss cost: the same as before. If you have more than one pipeline instance, move the cache to Redis. The network round-trip adds ~2ms, but the hit rate stays. For product analytics workloads where queries are more varied, the hit rate drops. But the floor is never zero — even varied queries benefit from caching the last 15 minutes of repeated queries. Fix 2: Warm your vector store, or move to in-memory A cold vector store is the second-most-common latency trap. Pinecone, Weaviate, and Qdrant all have a "first query is slow" behavior — the index is lazily loaded from disk on the first read. If your traffic is bursty (which most production RAGs are), you hit the cold path constantly. Two fixes: **Keep it warm.** A cron every 30 seconds running a trivial query against each index. This is cheap and the simplest fix. **Move small indexes in-memory.** If your corpus is under ~1 million documents and you're running a dedicated Python service, an in-memory FAISS index is genuinely faster than any managed vector store. Query latency drops to 20-50ms. The tradeoff is operational — you lose the managed-service ergonomics, and loading the index on service start is slow, so you need a warm-up script. For corpora above 1 million documents, stick with managed and optimize cold-start. Fix 3: Skip the reranker when you can Cross-encoder rerankers (Cohere rerank, BAAI bge-reranker) genuinely improve retrieval quality on hard queries. They also cost 400-800ms every call, and on easy queries, they don't change the top result. Two rules of thumb: - Short factoid queries ("what is our refund policy"): skip the reranker. The top embedding hit is almost always right. - Long, ambiguous queries ("how should I handle the case where a user churns but still has an active subscription"): keep the reranker. Ambiguity is where rerankers earn their cost. The heuristic that worked in one setup: if the query is under 8 words AND the top embedding hit has a cosine similarity above 0.85, skip rerank. Otherwise, rerank. This saved ~500ms on 60% of queries with no measurable quality drop. Fix 4: Stream the LLM response Non-streaming is the single most common perf bug in RAG demos. You wait for the full response, then display it. The user sees a spinner for 3-6 seconds. Streaming the response via Server-Sent Events or Anthropic's streaming API changes the perceived latency dramatically. Time-to-first-token on Claude Sonnet via the streaming API is ~500-800ms. The user sees text appearing after half a second. The total generation still takes the same wall-clock time, but the perception is "fast" instead of "broken". Implementation is straightforward — the Anthropic Python SDK exposes `client.messages.stream()`; OpenAI has `stream=True`. The frontend needs SSE or WebSocket handling, which is where teams stall. Worth the investment. Fix 5: Right-size the model for the task Not every step in a RAG pipeline needs the most capable model. A typical pipeline has three LLM-shaped tasks: 1. **…