A support bot that never says “I don’t know” is a liability

The cost of a confident guess

A visitor asks about your returns window. Your site never states one. A bot with no refusal path answers anyway — thirty days, generously worded, entirely invented. The visitor screenshots it. Now you either honour a policy you never wrote, or you explain to a paying customer why your own support channel lied to them.

Neither outcome is a support outcome. Both started as a design decision that nobody made on purpose: the assumption that an answer is always better than silence.

The failure mode of a grounded agent is a dead end. The failure mode of an ungrounded one is a commitment.

Confidence is not accuracy

Language models are fluent independently of whether they are right. Fluency is what makes the ungrounded answer dangerous: it arrives in the same tone, at the same length, with the same punctuation as the correct one. There is no surface signal for the reader to catch.

So the first check has to happen before generation, not after. Rinhelp retrieves candidate passages, reranks them, and looks at the top hit. If that rerank score is missing or below the bar, generation never runs. The model does not get a chance to talk its way into a policy.

If the bar is cleared, generation still has to cite a source. An uncited reply is thrown away and replaced with the same fallback.

Search is not enough

Hybrid search will surface a pricing page for almost every pricing question. That page can still say nothing about annual discounts. Rerank is the score that asks whether the passage actually answers. Search-only scores cannot pass the gate, even when they look high.

Where the threshold sits

The threshold is a product decision wearing a number. Raise it and the agent goes quiet on questions it could have handled. Lower it and it starts stretching a loosely related paragraph into a policy statement.

if (!ranked.length || top.scoreSource !== 'rerank' || top.score < MIN_RERANK) {
  return fallback()
}

const answer = generate(ranked)
if (!answer.citations.length) return fallback()
return answer

We landed where a wrong answer costs more than a missing one, which is where most small teams sit: you have no support queue to absorb the cleanup, and one invented policy can outweigh a hundred deflected questions.

What a refusal should look like

A refusal is still a support interaction, and a bad one still loses the visitor. Ours is a fixed line:

I couldn’t find enough information to answer that. Try asking about features, pricing, setup, or comparisons.

That copy has to do a few things:

• Say the agent does not have enough information, without explaining how retrieval works.
• Stay the same every time, so it cannot restate the question as if it were an answer, and cannot hedge into “it is typically around 30 days”.
• Point the visitor at topics a site is likely to cover. It does not pick the nearest retrieved page and offer that as a substitute answer.
• Leave the conversation open so the visitor can rephrase instead of starting over.

What it must never do is invent a policy with a disclaimer stapled to the front.

Refusals are a roadmap

Every live widget conversation lands in the inbox. When the agent used the fallback, that reply is marked “Couldn’t answer from sources”. Read a week of those markers and you are looking at the questions your site cannot carry.

The inbox does not rank or cluster those questions for you. It is still the list. Teams that write the missing pages see fewer of those markers. A bot that always answers reports a perfect coverage rate and teaches you nothing.

What we still get wrong

Multi-hop questions are the weak spot. Retrieval and the confidence gate look at passages one at a time. Generation is handed up to five of them, but it is told not to answer unless those passages contain enough. A question that needs one fact from the pricing page and one from the docs often gets declined even though both pages were in the index.

We would rather fail in that direction while we work on it. A quiet agent is a smaller problem than a confident one, and it is the only failure mode you can measure honestly.