# QuitSure AI Support Assistant — System Overview **Audience:** technical review (Vaibhav) · **Date:** 2026-07-03 · **Status:** working MVP, tested locally, not yet deployed. This document describes the full system: the problem, the architecture, every design decision, the safety and quality measures, the test evidence, and an honest list of what is done versus pending. The system is intentionally modular, so any component can be swapped based on direction from this review. --- ## 1. Goal Today, QuitSure app users ask support questions through Freshchat, and 7 to 8 human coaches answer them by hand. The goal is an **AI-first** support layer: an assistant answers common questions instantly, and escalates to a human coach only when needed. **What we build:** a single stateless HTTP API. **What the app team builds:** the chat UI, and the handoff into the existing Freshchat coach inbox on escalation. Clean separation of concerns. --- ## 2. Constraints (set by Ram) 1. **Restricted knowledge.** The AI answers only from content we provide (Ram's approved "Chat Shortcuts" doc, and program/rules files he will supply). It does **not** use Gemini's general internet knowledge, and it does **not** learn from Freshchat coach transcripts. This is enforced architecturally (retrieval-augmented generation grounded strictly in the approved content). 2. **Health-adjacent safety.** Smoking cessation borders on medical topics, so escalation and safety are first-class, not an afterthought. 3. **Per-program.** Knowledge and answers can differ by program (Original vs Relaxed), so the system is program-aware. --- ## 3. Architecture Stateless request/response. No session state on our side; the caller passes recent history each call. ``` user message (+ history, + program/platform/day/subscription from the app) │ [1] CRISIS GATE (deterministic): self-harm/suicide -> immediate human handoff. Always. │ [2] CONDENSE: fold recent turns into a standalone retrieval query │ [3] RETRIEVE: embed query -> cosine search over the approved knowledge base -> filter by the user's program and platform -> top-K approved answers │ [4] GENERATE (Gemini Flash, grounded ONLY in those approved answers): ├─ off-topic / time-pass ............ DECLINE politely (no human) ├─ covered by approved answers ...... ANSWER (in the user's language) └─ not covered / needs a human ...... ESCALATE (with a one-line coach summary) │ [5] BACKSTOP: if it answered on weak retrieval -> override to human handoff (anti-hallucination) │ [6] RESPONSE: { answer, escalate, escalate_reason, coach_summary, suggested_channel, message_id, ... } │ [7] LOG every turn + feedback -> the improvement loop ``` Everything runs server-side. Wrapped in error handling: any model error or timeout returns a graceful human handoff instead of a crash. --- ## 4. Knowledge base - **Source:** Ram's approved "Chat Shortcuts" Excel (5 sheets: PQP, LTP/Relaxed, Tech, Final shortcut sheet, New queries). Parsed to **483 Q&A entries**. - **Curation at build time:** each entry is tagged with the programs it applies to (PQP applies to P3 and P9, LTP to P9 only, the rest to all), and platform (iOS/Android for tech answers). **14 entries** that are coach-only scripts or contain fill-in placeholders (e.g. "Hi %name%, I am ___ your coach") are excluded from what the AI can send, leaving **469 answerable entries**. - **Em-dashes and merge fields** are normalized out so the bot never emits raw template text. - **Build pipeline:** `build_kb.py` (Excel -> normalized JSON) then `build_index.py` (embeddings). When the sheet changes, we re-run these; the app needs no change. --- ## 5. Retrieval (RAG) - **Embeddings:** `gemini-embedding-001` at 1536 dimensions (Matryoshka-truncated). - **Index:** 469 normalized vectors; cosine similarity by brute-force numpy. At this corpus size a vector database is unnecessary, retrieval is sub-millisecond. - **Context filter:** candidate answers are filtered to the user's program and platform, so a Relaxed-only answer never reaches an Original user, and iOS steps never reach Android. - **Confidence floor:** if nothing is similar enough, we do not guess. This is the primary anti-hallucination gate. --- ## 6. Answering and grounding (no hallucination) Answers are produced by **Gemini 2.5 Flash**, grounded strictly in the retrieved approved answers. Hallucination is prevented by layered, independent guards: | Layer | Guard | |---|---| | Knowledge | curated approved content only; no web, no transcripts | | Retrieval | confidence floor; abstain if nothing relevant | | Prompt | "answer only from the approved answers; never invent prices, steps, or policy" | | Decoding | low temperature (0.2) for consistency | | Output | structured JSON with the source entries cited | | Backstop | if it answers on weak retrieval, override to a human handoff | The answering model is a config value (`GEMINI_MODEL`); it can be swapped to any Gemini tier, or the provider abstracted, without touching the rest of the system. --- ## 7. Escalation and safety The response always carries a decision. Escalation reasons: | reason | escalate | meaning | |---|---|---| | `null` / `greeting` | no | answered / social reply | | `out_of_scope` | no | off-topic, politely declined (not sent to a human) | | `no_confident_answer` | yes | on-topic but not in the approved content | | `medical_topic` | yes | a medical question the approved content does not cover | | `crisis` | yes | self-harm / suicide (deterministic gate, always) | | `user_requested` | yes | asked for a human | | `user_frustrated` | yes | upset | | `account_action` | yes | account-specific billing (needs a lookup) | | `service_error` | yes | our service errored; fail into a human | **Medical policy (decided by Ram/Suraj):** the AI answers medical questions **that the approved doc covers** (e.g. using Champix or nicotine substitutes with the program, common withdrawal symptoms), because those are the exact answers coaches send today. It escalates medical questions **beyond** the doc (e.g. personal dosing). The one hard, non-negotiable line: **self-harm/suicide always escalates** via a deterministic filter, regardless of any content. Each escalation includes a `coach_summary` and a `suggested_channel` (coaching vs technical), so the app can route and the coach has context immediately. --- ## 8. Multilingual Answers are given in the user's language automatically (Gemini and the embedding model are multilingual). The knowledge base stays in English; only the reply is translated, so approved facts are preserved. Tested across Spanish, French, German, Arabic, Tamil, Marathi, Bengali, Hindi, and Hinglish: near 100% answered with correct source retrieval. Hinglish reply-consistency is the one soft spot (~83%), which is a polish item, not a correctness issue. --- ## 9. Conversation handling - **Continuity:** short follow-ups ("yes", "sure", "which one?") continue the topic instead of being treated as new, unanswerable questions. - **Multi-part questions:** answers the parts it can and escalates only the specific part that needs a human. (Extreme prompts bundling many unrelated questions are a known limit, see §12.) - **Context:** the app can pass `program`, `platform`, `program_day`, and `subscription_status` for program- and stage-specific answers. --- ## 10. Integration with the app and Freshchat - The app calls `POST /chat` per user message and renders `answer`. - On `escalate: true`, the app routes the conversation into the **existing Freshchat coach inbox**, passing `coach_summary` and the transcript so the handoff is seamless. - **Freshchat is reused as the human layer, not replaced.** We never touch Freshchat ourselves, so Ram's rule (no coach transcripts into the AI) holds by construction. - The frontend supplies user context; **our service does not need database access**, which keeps it decoupled from production. (Full contract in `docs/API.md`.) --- ## 11. Testing and evaluation Quality is measured, not asserted. The harness runs the real pipeline and scores it. - **Golden set + LLM-judge:** a labelled set graded on decision correctness (code), retrieval (code), and faithfulness/relevance (a stronger model judges the answer against the approved source, i.e. grading with an answer key, not from memory). - **Large simulations (~1000+ prompts each):** paraphrased real questions, scenario categories (emotional, technical, billing, account, off-topic, medical, crisis), multilingual, and complex bundles of 2 to 10 questions. Representative results from the latest runs: - **Doc coverage: 98%** (461/469 answerable questions answered when asked directly). - **Off-topic: 95% correctly declined. Self-harm: 100% caught as crisis.** - **Multilingual: ~100% answered** with correct retrieval across 7+ languages. - **Warmth: ~4/5** on a coach-like-tone judge. - **Prompt injection / jailbreak:** tested and held (does not leak the system prompt or break scope). - **Latency:** p50 ~2.5s, p95 ~4s (Gemini Flash), with a 20s server timeout. Every user thumbs-down and every escalation feeds a weekly improvement loop: gaps become new approved answers, and a regression eval runs before each change. --- ## 12. Known limitations (honest) - **Not deployed.** Runs locally; needs a host with auto-restart, plus auth on the endpoint (planned: validate the app's existing QuitSure access token) and a log-retention/PII policy. - **Coverage gaps** in transactional topics (change email/phone, restore purchase, delete account), to be closed with real Freshchat questions (approved by Ram, questions only). - **Extreme multi-part prompts** (many unrelated questions in one message) degrade, because top-K retrieval cannot ground 10 disparate topics. Realistic 2 to 4 part questions work. The proper fix is per-sub-question retrieval, added only if needed. - **Golden eval set is still small**; growing it with real questions makes the scores more trustworthy. - **Hinglish reply-consistency** ~83% (polish, not correctness). --- ## 13. Tech stack | Component | Choice | Why / swappability | |---|---|---| | API | FastAPI (Python), stateless | simple, standard; matches other QuitSure web services | | Answering | Gemini 2.5 Flash | fast, cheap, uses existing Gemini credits; one env var to change tier or provider | | Embeddings | gemini-embedding-001 (1536-d) | strong multilingual retrieval | | Vector search | numpy cosine | trivial at ~500 entries; drop-in vector DB if the corpus grows | | Logging | JSONL (chats + feedback) | zero-infra start; moves to a DB at scale | Every heavy dependency is behind a small boundary, so the knowledge system, the model, or the vector store can each be replaced independently. --- ## 14. What we would most value direction on 1. The **knowledge system**: how content is authored, versioned, and kept current at scale. 2. The **evaluation framework**: what quality bar and metrics to hold before production. 3. **Multilingual** strategy for the languages that matter most. 4. Anything in the existing system (yours) that we should adopt instead of, or alongside, this. The build follows current best practices and is modular by design, so aligning it to a stronger approach should be a small change, not a rewrite. --- **Reference docs in the repo:** `DESIGN.md` (deep design), `API.md` (integration contract), `INTEGRATION.md` (app-side overview), `eval/` (test harness and reports).