5/19/2025
Published on Convert on 19/05/2025.
Big context windows (1M–2M tokens) are often tested with literal matching. For real user text (different words, same idea), latent matching is what matters—and effective length can be much smaller (e.g. 2k). This article gives concrete ways to work within those limits.
Simple:
- Start new conversations to reset context.
- Re-edit earlier prompts to “branch” the conversation.
- Summarise the chat (one sentence, main goal, key points) and reuse as context.
Advanced: structure before you seek.
- Tag and structure: Add sentiment, themes, or tags to feedback so you can filter and send only relevant bits to the LLM. Accuracy goes up and context use goes down.
- Progressive summarisation: Build shorter versions (e.g. <20 words, then 6–10 words) with clear examples so labels stay consistent.
- Recursive summarisation: Summarise summaries for a high-level view; keep links back to detail.
- RAG: Vector DB + semantic search for very large scale; good structure often reduces how much you need it.
The article also covers how different models (GPT-4, Claude, Gemini) behave as context grows.
