RAG that actually ships: lessons from production support agents
Retrieval-augmented generation is easy to demo and hard to ship. Here is what changed once ours had to survive real customers.
A retrieval demo that dazzles on stage tends to wilt in production. The gap is rarely the model — it is everything around it: chunking, freshness, and knowing when to stay silent.
Chunk for meaning, not for tokens
Naive fixed-size chunks split answers down the middle. We chunk on semantic boundaries and keep a little overlap, so a retrieved passage is a complete thought the model can actually reason over.
Teach it to say "I do not know"
The single biggest quality win was a confidence gate. If retrieval returns nothing relevant, the agent acknowledges and escalates instead of inventing a policy that does not exist. Silence beats a confident wrong answer every time.
Trust is lost in one bad answer and earned back over a hundred good ones. Design for the downside.
Ship the gate before you ship the cleverness. Your customers will notice the difference long before your benchmark does.
