Why retrieval quality and latency matter
A language model can write a convincing answer without knowing whether it is correct. Retrieval-augmented generation (RAG) reduces that risk by supplying relevant material from a separate knowledge base before the model responds.
For a Python assistant, that means looking up information in a reference document instead of relying only on the model's training data. The approach introduces two engineering problems. The retriever must choose useful passages, and the pipeline must return an answer quickly enough to feel responsive.
The project described in the supplied paper uses LangChain, OpenAI embeddings, FAISS, and Streamlit to build a Python-focused chatbot. Its reported optimization work focuses on chunking, filtering retrieved passages, index reuse, and response latency.
What the project aims to contribute
- A RAG pipeline for querying a Python knowledge base.
- A practical comparison of retrieval settings and their impact on irrelevant context.
- A staged approach to latency reduction, from index reuse to streaming.
- A Streamlit deployment with reported usage, reliability, and answer-quality measurements.
System architecture
The system is organized around three stages: ingestion, retrieval, and generation. Ingestion prepares the knowledge base, retrieval selects passages for a question, and generation uses those passages to compose an answer.
python_guide.txt and create overlapping chunks.Architecture reconstructed from the supplied project paper, not independently checked against source code.
Technology stack
| Component | Technology | Purpose |
|---|---|---|
| Language model | OpenAI GPT-3.5 Turbo | Answer generation |
| Embeddings | text-embedding-3-small | 1,536-dimensional text vectors |
| Vector search | FAISS | Similarity retrieval |
| Orchestration | LangChain | Connect loading, retrieval, and generation |
| Interface | Streamlit | Chat and response display |
| Language | Python | Application code |
The stack above is documented in the screenshots supplied for this paper. Version numbers and settings may have changed since that write-up.
Preparing the knowledge base
The knowledge base starts as a Python reference text file. The reported pipeline splits it into chunks, embeds each chunk, and indexes the vectors for retrieval.
Read python_guide.txt as the source document.
Split the document into overlapping sections. The write-up identifies an 800-token target with a 200-token overlap as the selected setting.
Convert each chunk into a vector using text-embedding-3-small.
Store vectors in FAISS so the retriever can compare incoming questions with the document chunks.
Overlap helps keep an example and its explanation together when they fall near a chunk boundary. It also means some text appears in more than one stored chunk, so more overlap is not automatically better.
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
loader = TextLoader("data/python_guide.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=200,
)
chunks = splitter.split_documents(documents)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
index = FAISS.from_documents(chunks, embeddings)Illustrative implementation based on the described method. The screenshots do not establish the exact source code or whether these size settings represent tokens or characters.
Implementation note. LangChain's RecursiveCharacterTextSplitter counts characters by default. To enforce token-based chunk sizes, a token-aware length function or token splitter is needed. The supplied paper describes token counts but does not show enough code to confirm how they were measured.
Improving retrieval quality
Early testing reportedly found that roughly a quarter of retrieved passages were only loosely connected to the question. That creates two risks: relevant material gets crowded out, and the model may use unrelated context when answering.
Filtering and reranking
The supplied paper describes an additional model-based filtering step. After vector search returns candidate chunks, the model selects passages that directly help answer the question. This uses the question itself as part of the relevance test rather than relying only on vector similarity.
Given the user question and the retrieved passages:
- Select only passages that directly help answer the question.
- If none are relevant, return NO_RELEVANT_CONTEXT.
- Return the selected passage IDs before generating the answer.Simplified from the prompt in the supplied paper; not presented as an exact production prompt.
Query expansion
For a vague question, the described approach generates a few alternate phrasings and searches with each. A query about “decorators,” for example, may be expanded to “Python function decorators” and “@decorator examples.” The resulting passages are deduplicated before generation.
Chunk-size trade-offs
Short chunks can make individual results more focused, but they may lose context. Long chunks preserve explanations but can introduce unrelated details. The write-up reports the following irrelevant-retrieval shares for different sizes.
Smaller chunks
Medium chunks
Selected setting
Larger chunks
Reported selected overlap
Reported later setting
Values are transcribed from the supplied screenshots. No test dataset or per-query labels were provided for independent verification.
Reported retrieval result
The share described as irrelevant falls by 20 percentage points, equivalent to an 80% relative reduction. The supplied document calls this a “20% reduction,” which is imprecise.
The paper attributes the change to prompt-based filtering, query expansion, and chunk tuning. Without the original labels and query-level results, it is not possible to isolate how much each change contributed.
Reducing response latency
Response time includes more than model generation. A query must be embedded, matched against the index, packaged into context, sent to the model, and displayed to the user. The supplied paper reports a baseline of 4.0 seconds and a later response time of 1.5 seconds.
The screenshots describe this as 67%. The arithmetic from 4.0s to 1.5s yields a 62.5% decrease, so this paper uses that calculated figure.
What changed
Build or load the vector index once rather than repeating setup for every request. The document links this to a reduction from 4.0s to 3.2s.
Reduce per-chunk Python work during retrieval. The paper describes a move from 3.2s to 2.1s using vectorized operations.
Lower the retrieved passage count from five to three after checking that the extra passages rarely helped. The reported time falls from 2.1s to 1.7s.
Show tokens as they arrive rather than waiting for the full answer. The paper associates this with a change from 1.7s to 1.5s, although streaming is best measured using time to first token rather than total completion time.
Baseline timing breakdown
| Stage | Reported baseline |
|---|---|
| Query embedding | 400 ms |
| Retrieval and reranking | 800 ms |
| Model generation | 2,500 ms |
| Interface rendering | 300 ms |
| Total | 4,000 ms |
The screenshots also report a load test with more than 100 concurrent Streamlit users and P95 latency around 1.4 to 1.8 seconds. Those figures are retained as project-reported results, not independently established measurements.
Measurement distinction. Index initialization, query search, model completion, and perceived streaming delay are different measurements. Future benchmarks should record each separately under the same workload.
Deployment and reported results
The supplied paper describes a Streamlit Cloud deployment. The frontend runs in Streamlit, the backend calls LangChain and OpenAI, and a Python guide file supplies the knowledge base. It also describes API-key configuration through Streamlit secrets and logging query-response pairs for review.
Figures are taken from the screenshots. The raw monitoring data, load-test traces, and response annotations were not supplied.
Grounding evaluation
The paper describes a manual review of 50 responses by a Python developer. Its reported distribution is 93% accurate and grounded, 5% partially correct, and 2% hallucinated. These percentages may represent aggregate judgments or rounded values, since a 50-response sample supports increments of two percentage points.
The reported failure cases focused on advanced language topics, including metaclasses and asynchronous context managers, where the knowledge base had less coverage. This points to a common RAG limitation: retrieval cannot supply information that the indexed documents do not contain.
Reliability
The source describes no major outages during its three-week test period, with occasional API rate limits and Streamlit reconnection delays. It also mentions backoff retries. These are useful operational observations, but the screenshots do not include logs or a formal incident record.
Engineering lessons
Chunking is a quality setting. Chunk size affects the evidence the model sees. It should be tested on representative questions, not chosen once and forgotten.
Similarity is only a first pass. A passage can look similar to a query without answering it. Query-aware filtering may help, though it adds model cost and latency.
Index lifecycle matters. Building embeddings on every request wastes time. Index reuse can remove initialization work from the critical path.
Streaming changes perception. A faster first token can make an application feel more responsive, even if total answer completion takes the same time.
The source also reports that FAISS search remained under 200 ms for collections approaching 50,000 chunks. That is a reported observation, not a general scaling law. Actual search performance depends on index type, hardware, vector count, dimensionality, and accuracy settings.
Implementation outline
The supplied document names a Streamlit interface, a console chatbot, a Python knowledge-base file, dependency definitions, and spreadsheet-based response logging. The following structure and function illustrate the design without claiming to reproduce the original repository exactly.
rag-chatbot/
├── data/
│ └── python_guide.txt
├── chatbot.py
├── streamlit_app.py
├── requirements.txt
└── .streamlit/
└── secrets.toml # local secret file, do not commitdef answer_question(question, vector_store, model):
passages = vector_store.similarity_search(question, k=3)
context = "\n\n".join(doc.page_content for doc in passages)
prompt = (
"Answer using the context when possible. "
"If the context is insufficient, say so.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
return model.invoke(prompt)This is explanatory pseudocode, not verified source code. The screenshot describes additional reranking and streaming behavior that are not implemented in this short example.
For a deployed version, secrets should remain outside source control, logs should avoid sensitive user inputs, and citation support should connect each answer to the passages it used.
Limits and next steps
The project makes a useful case for measuring retrieval and latency together. The supplied screenshots do not, however, include the data or code needed to reproduce their benchmark claims.
Evaluation
Publish representative questions, relevance labels, response annotations, and the exact scoring method. Separate retrieval precision from answer grounding and hallucination rates.
Performance
Capture request-level timings for embedding, retrieval, filtering, model output, and rendering. Report time to first token and full completion independently.
Implementation
Make chunk units, retrieval k, index persistence, reranking, and query expansion settings explicit. Record dependency and model versions.
Operations
Keep API keys in a secret manager, add rate-limit handling and structured monitoring, and store reproducible load-test artifacts.
The reported gains are promising, but without test artifacts they should be presented as observations from the original paper rather than independent evidence of production-scale behavior.
Conclusion
This chatbot combines document chunking, embeddings, vector retrieval, and model generation to answer Python questions using an external reference. The supplied project paper reports better retrieval precision after filtering and chunk tuning, and faster responses after changes to indexing, retrieval, and output delivery.
The key takeaway is straightforward: a RAG system needs to be evaluated at each stage. Better answers depend on useful context, and lower latency depends on knowing where time is spent. The next step is to make the reported improvements repeatable with public test sets, timing traces, and clearly defined metrics.
Source and reporting notes
Primary material. Eight screenshots of Context-Aware AI Chatbot: Optimizing Retrieval Quality and Latency in Production RAG Systems, supplied by the project author for this adaptation on November 9, 2026.
Project. production-rag.netlify.app. Linked for readers; not consulted or independently verified for this adaptation.
Scope. This paper is a rewrite of the supplied document. No repository review, independent experiment, load test, or source-code inspection was performed. Performance figures are identified as reported results, and arithmetic corrections are called out where needed.