Case Study: Optimizing a Knowledge Base for AI Search Visibility
Client Profile: A mid-sized SaaS Fintech (Anonymized).
The Problem: They deployed an AI chatbot to handle customer support. Within 48 hours, they had to shut it down.
The Reason: The bot promised a customer a 50% discount that didn’t exist.
Most companies blame the model (GPT-4) when this happens. They are wrong. The model is fine; your data is messy.
This case study breaks down exactly how we took a “drunk” AI agent and turned it into a precision instrument by fixing the underlying knowledge base.
The “PDF Trap” (Why Your Bot is Lying)
The Gist: AI models are not readers; they are retrievers. If you feed them a 50-page PDF, they lose context. To fix hallucinations, you must break content into atomic, semantic chunks.
Our client made the classic mistake: “Let’s just upload all our Help Center PDFs into the vector database.”
This is fatal. When a user asks, “What is the refund policy for premium users?”, the AI retrieves a chunk of text. If that chunk comes from Page 42 (which discusses refunds) but misses the header on Page 40 (which says “For Standard Users Only”), the AI will confidently lie.
The Diagnosis:
- Too much noise: The AI was retrieving marketing fluff along with technical facts.
- Zero hierarchy: The vector database couldn’t distinguish between a “User Guide” (Fact) and a “Blog Post” (Opinion).
The Fix: “The Chunking Protocol”
We didn’t change the AI model. We changed the ingestion pipeline.
We stripped their 200+ PDF documents into raw text and applied a strict Semantic Chunking Strategy.
Step 1: Atomic Segmentation
We broke long articles into discrete “Answer Blocks.” Each block had to stand alone as a complete thought.
Step 2: Metadata Injection
This was the turning point. We injected invisible metadata tags into every single chunk.
- Before: “Refunds are processed in 5 days.” (Ambiguous).
- After:
[Product: Ent-Tier] [Region: EU] [Topic: Billing]Refunds are processed in 5 days.
Now, when a user asks about US Enterprise refunds, the AI ignores the EU chunk entirely. This structured approach aligns with proven optimization strategies outlined in our comprehensive the 2026 GEO optimisation framework.
This context-aware approach demonstrates why traditional keyword strategies no longer apply in modern AI-powered search systems.
This structured approach aligns with what happened when we tested AI search against Google regarding context-aware retrieval systems.
This precision targeting reflects the fundamental shift from SEO to GEO where context-aware retrieval becomes essential for accurate AI responses.
This precision targeting is especially valuable for location-based queries, where our GEO implementation services can help ensure users receive region-specific and contextually accurate information.
The Results: From Hallucination to Precision
We ran a 30-day A/B test comparing the “Raw PDF” agent against the “Optimized Knowledge Base” agent.
| Metric | Raw PDF Ingestion (The “Lazy” Way) | Structured Chunking (The Innovate Way) |
| Hallucination Rate | 18% (High Risk) | 0.2% (Enterprise Safe) |
| Retrieval Speed | 4.2 Seconds | 1.1 Seconds |
| Citation Accuracy | 40% | 98% |
| Token Cost | $0.12 per query | $0.03 per query |
Here is the math: By retrieving smaller, more relevant chunks, we fed less data to the LLM per query. This reduced token consumption by 75% while increasing accuracy.
How to Audit Your Own Knowledge Base
If your chatbot is giving vague or wrong answers, stop tweaking the system prompt. Look at the source data.
The “Clean Data” Checklist:
- Remove the formatting: Headers, footers, and page numbers confuse the retriever. Strip them.
- Q&A Formatting: Rewrite passive documentation into active Q&A pairs. AI models love the “Question -> Answer” structure.
- Fact Isolation: Do not bury a critical policy inside a paragraph about company culture. Isolate the rule.
We previously discussed how to establish Entity Identity to get cited by Perplexity. This is the internal version of that same discipline. You are teaching your own AI how to read your own manual.
What Should You Do Next?
A hallucinating AI is a liability. A structured AI is an asset. We can run a “Retrieval Audit” on your current documentation and show you exactly where your data is confusing your bot.
Fix Your Knowledge Base