# QuitSure AI Support — Full Design

The complete picture of what we build on our side: a support chatbot that gives the best
possible answers, **never makes things up**, knows the edge of its own knowledge, and gets
smarter over time. Health-adjacent, so safety is a first-class concern, not an afterthought.

> One sentence to hold onto: **if it isn't in our approved knowledge, the bot does not answer it — it hands off to a coach.** Everything below is in service of that rule.

---

## 1. What we own (and what we don't)

**We build:** one stateless HTTP API (`/chat`) — answer + escalation decision — plus the
knowledge pipeline, the safety logic, logging, and the self-improvement loop. A tiny test
UI ships with it so the whole team can try it and rate answers.

**We do NOT build:** the in-app chat screen, the Freshchat connection, or the coach handoff.
Those are the app/dev team's job. When our API says `escalate: true`, they route to a coach.

**Hard constraints (Ram):** knowledge is curated/approved only. **No Freshchat transcripts** —
raw coach chatter is inconsistent and would dilute the model's reasoning. Garbage in, garbage out.

---

## 2. The five principles

1. **Grounded** — every answer comes from approved knowledge, not the model's memory.
2. **Abstaining** — when the knowledge doesn't cover it, the bot says so and escalates. Confidently
   not-answering is a feature, not a failure.
3. **Escalating** — medical, crisis, "talk to a human," frustration, and account actions always go to a coach.
4. **Curated** — knowledge quality is the whole game; we invest in the KB, not in clever prompting tricks.
5. **Measured** — nothing ships without an eval. We track faithfulness and escalation accuracy on a golden set.

---

## 3. Request flow

```
user message + recent history
        │
        ▼
1. Condense to a standalone query  ──► (resolve follow-ups: "how about iOS?" → full question)
        │
        ▼
2. Embed query → cosine search over 483 approved Q&A   ── top-K + best similarity score
        │
        ├─ best similarity < FLOOR ────────────────► ABSTAIN: escalate "no_confident_answer"
        │                                             (deterministic — the model never even runs)
        ▼
3. Gemini Flash answers, grounded ONLY in retrieved approved answers,
   following the safety rules (medical / crisis / human / frustration / account-action)
        │
        ▼
4. Backstop check: if the model answered but retrieval was weak → override to escalate
        │
        ▼
5. Return { answer, escalate, escalate_reason, coach_summary, sources }
        │
6. Log everything (query, retrieved, answer, decision) for the self-improvement loop
```

The two abstention gates (step 2 deterministic floor + step 4 backstop) are what make
"it doesn't know → it doesn't answer" actually true, instead of a hope pinned on the prompt.

---

## 4. No hallucinations — the layered defense

No single trick prevents hallucination. We stack cheap, independent guards so a failure of
one is caught by the next:

| Layer | Guard | What it stops |
|---|---|---|
| Knowledge | Curated, approved Q&A only (no transcripts, no web) | The model learning wrong/inconsistent facts |
| Retrieval | Hard similarity **floor** → abstain before generating | Answering when nothing relevant was found |
| Prompt | "Answer ONLY from the approved answers below; never invent prices/steps/policies" | The model filling gaps from its own memory |
| Decoding | Low temperature (0.3) | Creative drift on factual support content |
| Output | Structured JSON + **sources** (which KB entries were used) | Untraceable answers; lets us audit faithfulness |
| Backstop | If answered on weak retrieval → override to escalate | The model overriding the abstain instinct |
| Scope | Medical/health/off-topic → escalate, never answered by AI | Unsafe advice on a health-adjacent product |
| (optional) Verify | Second-pass "is this answer supported by the context?" judge | Subtle unsupported claims — adds latency/cost |

The optional groundedness verifier (a fast second LLM check) is held in reserve: we turn it on
only if eval shows faithfulness slipping, because it roughly doubles per-answer latency/cost.

---

## 5. "What it doesn't know, it doesn't answer" — abstention & escalation

This is the heart of the request. The bot has **exactly one job when unsure: hand off cleanly.**

Escalation reasons the API returns:

| reason | trigger | who decides |
|---|---|---|
| `no_confident_answer` | retrieval below floor, or model can't ground an answer | retrieval gate + model |
| `medical_topic` | medication, NRT dosage, withdrawal symptoms, diagnosis, drug interactions | model rule (hard) |
| `crisis` | self-harm / suicidal ideation / acute distress | model rule (hard) + show help resources |
| `user_requested` | "talk to a human / coach / agent" | model rule |
| `user_frustrated` | clear anger/upset | model rule |
| `account_action` | needs a refund / data change the AI can't perform | model rule |

For every escalation the API also returns a one-line `coach_summary` so the human picks up
with full context and the user never repeats themselves. (The dev side drops it into Freshchat.)

**Design stance:** we tune toward *escalating too often, not too rarely.* A needless handoff
costs a coach a minute. A confident wrong answer about a refund or a medication costs trust —
and on a health product, potentially more. Deflection rate is a metric we grow carefully, never chase.

---

## 6. Retrieval — getting the right knowledge in front of the model

- **Embeddings:** `gemini-embedding-001` at 1536 dims (Matryoshka-truncated — negligible quality
  loss at this corpus size, cheaper to store/compare).
- **Atomic chunks:** our KB is already one Q&A per entry — no messy chunking decisions. Each entry
  is embedded on `question + tags + topic` (what a user query should match), with the full approved
  answer carried as the payload.
- **Search:** brute-force cosine over 483 normalized vectors — instant, no vector DB needed.
- **Multi-turn:** condense recent user turns into a standalone query before retrieval so follow-ups
  ("what about on iPhone?") still find the right entry.
- **Confidence floor:** the single most important number in the system — below it, we abstain.
  It must be **calibrated on real queries** once we can run (see Eval). We ship a sane default and tune.
- **If precision lags:** add hybrid keyword+semantic search (support queries contain exact terms like
  shortcut codes, "refund," "Stay Free") and/or a reranker. Not needed at 483 entries unless eval says so.

---

## 7. Context limit — the honest answer

**The model's window is not the constraint.** Gemini Flash takes ~1,000,000 input tokens. What we
actually feed it per turn is tiny and *deliberately bounded*:

| Piece | Approx tokens |
|---|---|
| System rules + safety instructions | ~600 |
| Top-K retrieved approved answers (5 entries) | ~1,500–2,500 |
| Recent conversation history (last ~6 turns) | ~500–1,500 |
| **Total per call** | **~3k–5k of a ~1,000k budget** |

So we use under 0.5% of the window. We could stuff all 483 entries in and skip retrieval —
**we deliberately don't.** Two reasons:

1. **Precision beats recall for support.** Five on-topic answers produce a sharper, more faithful
   reply than 483 mostly-irrelevant ones drowning the signal (the "lost in the middle" effect).
2. **Cost & latency scale with tokens.** Small context = fast, cheap, snappy chat.

We bound history to the last ~6 turns for the same reason — enough for continuity, not so much
that old tangents pull the model off course. **The limit we manage is precision, not capacity.**

---

## 8. The self-improving loop — how it gets to "best answers"

Be precise about what "self-improving" means here: **not** a model that retrains its own weights
(that's how bots get worse and unsafe). It's a **human-in-the-loop flywheel** where the *knowledge*
and the *thresholds* improve continuously from real usage:

```
        ┌─────────────────────────────────────────────────────┐
        │                                                       │
   real chats ──► log every (query, retrieved, answer, decision)│
        │                                                       │
   team & users rate answers (👍/👎 in the test UI / app)       │
        │                                                       │
   weekly review:                                               │
     • cluster 👎 + "no_confident_answer" escalations           │
        → these are KNOWLEDGE GAPS                               │
     • a coach writes the approved answer for each gap          │
     • add to the Excel / KB                                    │
        │                                                       │
   rebuild index ──► regression-test vs golden set ──► deploy ──┘
```

What we capture from day one (already in the test build):
- **Full chat logs** (`data/chatlog.jsonl`): query, condensed query, top similarity, retrieved IDs,
  answer, escalate decision + reason. This is the raw material.
- **Feedback** (`data/feedback.jsonl`): 👍/👎 + optional note per answer.

The flywheel turns those into KB growth:
1. **Gap mining** — every `no_confident_answer` escalation is a question we *should* be able to answer.
   Cluster them; the big clusters become new approved entries. This is the main engine.
2. **Correction** — every 👎 is reviewed; if the approved answer was wrong/stale, fix it at the source (the KB).
3. **Coach answers** — when a coach resolves an escalation well, that approved answer can be added to the KB
   (curated, not auto-ingested — keeps Ram's "no raw transcripts" rule intact).
4. **Threshold tuning** — usage tells us if the confidence floor is too eager (missing answerable
   questions) or too loose (answering things it shouldn't). We adjust and re-eval.

Net effect: coverage and accuracy climb every week, the escalation rate falls *for the right reasons*,
and we never risk the model drifting because the model itself never changes — only the curated knowledge does.

---

## 9. Evaluation — nothing ships unmeasured

Build a **golden set**: ~100–150 real questions (seed from the KB + the gap log) with the expected
behavior (correct answer's source entry, or "should escalate" + reason). On every KB/threshold change:

| Metric | Question it answers | Target |
|---|---|---|
| Retrieval recall@5 | Is the right approved answer in the top 5? | high — this caps everything downstream |
| Answer faithfulness | Is the answer supported by retrieved knowledge (no invention)? | ~100% — the non-negotiable |
| Escalation precision | When it escalated, should it have? | high |
| Escalation recall | When it *should* escalate (esp. medical/crisis), did it? | ~100% on safety reasons |
| Deflection rate | % handled without a human | grows over time, never forced |
| p50 latency | snappy enough for chat? | sub-second to first token (Flash) |

Faithfulness and safety-recall are the ones we never trade away for a higher deflection number.

---

## 10. Safety — because this is health-adjacent

Smoking cessation borders on medical advice (medication, withdrawal, mental health). 2025–26 saw
real lawsuits and regulatory moves over AI giving unlicensed health/therapy advice. Our guards:

- **Scope the bot to product/app support**, not clinical advice — enforced in the system prompt.
- **Hard-escalate** medication, withdrawal-symptom, diagnosis, and mental-health questions to a human.
- **Crisis path:** self-harm/suicidal signals → immediate human handoff + surface help resources;
  the model never freelances here.
- **Standing disclaimer** available in the UI: "I'm QuitSure's support assistant, not a medical professional."
- **Continuous human review** of a sample of conversations (the logs make this easy).

---

## 11. Tech choices (and why)

| Choice | Why |
|---|---|
| Gemini **Flash** for answering | fast (~0.2–0.4s first token), cheap, uses our existing Gemini credits |
| `gemini-embedding-001` (1536-dim) | strong retrieval, small footprint for ~500 entries |
| **numpy cosine**, no vector DB | 483 vectors = instant brute force; one less moving part |
| **Stateless** API (`history` passed in) | simplest thing for the app team to consume; we hold no session |
| **FastAPI**, committed static test UI | matches our other QuitSure web projects; pull-and-go |
| JSONL logs for chats + feedback | zero-infra start to the self-improvement loop; graduate to a DB later |

---

## 12. Roadmap

| Phase | What | Status |
|---|---|---|
| **0** | KB pipeline from the Excel (483 entries) | ✅ done |
| **1** | `/chat` API: RAG + grounding + abstain floor + escalation; test chat UI with 👍/👎 | ⏳ code written, needs `GEMINI_API_KEY` to run & tune |
| **2** | Calibrate confidence floor + build golden eval set; share test URL with the team | next |
| **3** | Self-improvement loop in practice: weekly gap mining → KB growth; optional groundedness verifier | after real usage |
| **4** | (dev side) wire the API into the app + Freshchat handoff; we support the integration | dev team |

The blocker for everything past "code written" is a **Gemini API key**. With it, we build the index,
run real tests, calibrate the floor, and put a test URL in front of the team this week.