Production RAG / Technical paper
RG.
Engineering research / Technical report
2026 · Project study
Retrieval-augmented generation · LLM systems

Context-aware AI chatbot:
Retrieval quality and latency in a production RAG system

How a Python knowledge-base chatbot uses retrieval, filtering, and pipeline tuning to improve the answers it returns.

AuthorDev Sharma
Technical stackPython · LangChain · OpenAI · FAISS
ProjectProject website ↗

Abstract

This paper describes a retrieval-augmented generation chatbot built around a Python reference document. The system uses OpenAI embeddings, FAISS, LangChain, and Streamlit to retrieve passages and generate answers. The supplied project write-up reports a drop in irrelevant retrieval from 25% to 5%, a reduction in response time from 4.0 seconds to 1.5 seconds, and 93% answer grounding accuracy. This adaptation explains the design, performance trade-offs, and reported results while separating those claims from measurements that can be independently verified. It also outlines how to make the evaluation reproducible.

RAGVector retrievalLatencyChunk tuningStreamlit

Why retrieval quality and latency matter

A language model can write a convincing answer without knowing whether it is correct. Retrieval-augmented generation (RAG) reduces that risk by supplying relevant material from a separate knowledge base before the model responds.

For a Python assistant, that means looking up information in a reference document instead of relying only on the model's training data. The approach introduces two engineering problems. The retriever must choose useful passages, and the pipeline must return an answer quickly enough to feel responsive.

The project described in the supplied paper uses LangChain, OpenAI embeddings, FAISS, and Streamlit to build a Python-focused chatbot. Its reported optimization work focuses on chunking, filtering retrieved passages, index reuse, and response latency.

What the project aims to contribute

  • A RAG pipeline for querying a Python knowledge base.
  • A practical comparison of retrieval settings and their impact on irrelevant context.
  • A staged approach to latency reduction, from index reuse to streaming.
  • A Streamlit deployment with reported usage, reliability, and answer-quality measurements.

System architecture

The system is organized around three stages: ingestion, retrieval, and generation. Ingestion prepares the knowledge base, retrieval selects passages for a question, and generation uses those passages to compose an answer.

System flow
Ingestion
Load and splitRead python_guide.txt and create overlapping chunks.
Indexing
Embed and storeGenerate OpenAI embeddings and create a FAISS index.
Retrieval
Search and filterEmbed a question, retrieve candidates, and select useful context.
↓
Context
Assemble evidenceCombine the user question with selected passages.
Generation
Compose answerUse the language model to respond from the supplied context.
Interface
Return responseDisplay the answer through Streamlit.

Architecture reconstructed from the supplied project paper, not independently checked against source code.

Technology stack

ComponentTechnologyPurpose
Language modelOpenAI GPT-3.5 TurboAnswer generation
Embeddingstext-embedding-3-small1,536-dimensional text vectors
Vector searchFAISSSimilarity retrieval
OrchestrationLangChainConnect loading, retrieval, and generation
InterfaceStreamlitChat and response display
LanguagePythonApplication code

The stack above is documented in the screenshots supplied for this paper. Version numbers and settings may have changed since that write-up.

Preparing the knowledge base

The knowledge base starts as a Python reference text file. The reported pipeline splits it into chunks, embeds each chunk, and indexes the vectors for retrieval.

Document loading

Read python_guide.txt as the source document.

Chunking

Split the document into overlapping sections. The write-up identifies an 800-token target with a 200-token overlap as the selected setting.

Embedding

Convert each chunk into a vector using text-embedding-3-small.

Indexing

Store vectors in FAISS so the retriever can compare incoming questions with the document chunks.

Overlap helps keep an example and its explanation together when they fall near a chunk boundary. It also means some text appears in more than one stored chunk, so more overlap is not automatically better.

Illustrative ingestion code
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

loader = TextLoader("data/python_guide.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=200,
)
chunks = splitter.split_documents(documents)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
index = FAISS.from_documents(chunks, embeddings)

Illustrative implementation based on the described method. The screenshots do not establish the exact source code or whether these size settings represent tokens or characters.

Implementation note. LangChain's RecursiveCharacterTextSplitter counts characters by default. To enforce token-based chunk sizes, a token-aware length function or token splitter is needed. The supplied paper describes token counts but does not show enough code to confirm how they were measured.

Improving retrieval quality

Early testing reportedly found that roughly a quarter of retrieved passages were only loosely connected to the question. That creates two risks: relevant material gets crowded out, and the model may use unrelated context when answering.

Filtering and reranking

The supplied paper describes an additional model-based filtering step. After vector search returns candidate chunks, the model selects passages that directly help answer the question. This uses the question itself as part of the relevance test rather than relying only on vector similarity.

Example filtering instruction
Given the user question and the retrieved passages:
- Select only passages that directly help answer the question.
- If none are relevant, return NO_RELEVANT_CONTEXT.
- Return the selected passage IDs before generating the answer.

Simplified from the prompt in the supplied paper; not presented as an exact production prompt.

Query expansion

For a vague question, the described approach generates a few alternate phrasings and searches with each. A query about “decorators,” for example, may be expanded to “Python function decorators” and “@decorator examples.” The resulting passages are deduplicated before generation.

Chunk-size trade-offs

Short chunks can make individual results more focused, but they may lose context. Long chunks preserve explanations but can introduce unrelated details. The write-up reports the following irrelevant-retrieval shares for different sizes.

Reported chunk-size comparison
256 tokens
12% irrelevant

Smaller chunks

512 tokens
18% irrelevant

Medium chunks

800 tokens
5% irrelevant

Selected setting

1,024 tokens
22% irrelevant

Larger chunks

200 tokens
Overlap

Reported selected overlap

3 chunks
Retrieved per query

Reported later setting

Values are transcribed from the supplied screenshots. No test dataset or per-query labels were provided for independent verification.

Reported retrieval result

Before and after tuning
25%
5%

The share described as irrelevant falls by 20 percentage points, equivalent to an 80% relative reduction. The supplied document calls this a “20% reduction,” which is imprecise.

The paper attributes the change to prompt-based filtering, query expansion, and chunk tuning. Without the original labels and query-level results, it is not possible to isolate how much each change contributed.

Reducing response latency

Response time includes more than model generation. A query must be embedded, matched against the index, packaged into context, sent to the model, and displayed to the user. The supplied paper reports a baseline of 4.0 seconds and a later response time of 1.5 seconds.

Reported response-time comparison
4.0sInitial response time
1.5sReported optimized time
62.5%Elapsed-time reduction

The screenshots describe this as 67%. The arithmetic from 4.0s to 1.5s yields a 62.5% decrease, so this paper uses that calculated figure.

What changed

Reuse the index

Build or load the vector index once rather than repeating setup for every request. The document links this to a reduction from 4.0s to 3.2s.

Use vectorized similarity search

Reduce per-chunk Python work during retrieval. The paper describes a move from 3.2s to 2.1s using vectorized operations.

Retrieve fewer chunks

Lower the retrieved passage count from five to three after checking that the extra passages rarely helped. The reported time falls from 2.1s to 1.7s.

Stream the model output

Show tokens as they arrive rather than waiting for the full answer. The paper associates this with a change from 1.7s to 1.5s, although streaming is best measured using time to first token rather than total completion time.

Baseline timing breakdown

StageReported baseline
Query embedding400 ms
Retrieval and reranking800 ms
Model generation2,500 ms
Interface rendering300 ms
Total4,000 ms

The screenshots also report a load test with more than 100 concurrent Streamlit users and P95 latency around 1.4 to 1.8 seconds. Those figures are retained as project-reported results, not independently established measurements.

Measurement distinction. Index initialization, query search, model completion, and perceived streaming delay are different measurements. Future benchmarks should record each separately under the same workload.

Deployment and reported results

The supplied paper describes a Streamlit Cloud deployment. The frontend runs in Streamlit, the backend calls LangChain and OpenAI, and a Python guide file supplies the knowledge base. It also describes API-key configuration through Streamlit secrets and logging query-response pairs for review.

Metrics reported in the source paper
100+Peak concurrent users
1.5sReported median latency
93%Reported grounding accuracy
>99.5%Reported uptime
0Reported major failures
3 weeksReported test window

Figures are taken from the screenshots. The raw monitoring data, load-test traces, and response annotations were not supplied.

Grounding evaluation

The paper describes a manual review of 50 responses by a Python developer. Its reported distribution is 93% accurate and grounded, 5% partially correct, and 2% hallucinated. These percentages may represent aggregate judgments or rounded values, since a 50-response sample supports increments of two percentage points.

The reported failure cases focused on advanced language topics, including metaclasses and asynchronous context managers, where the knowledge base had less coverage. This points to a common RAG limitation: retrieval cannot supply information that the indexed documents do not contain.

Reliability

The source describes no major outages during its three-week test period, with occasional API rate limits and Streamlit reconnection delays. It also mentions backoff retries. These are useful operational observations, but the screenshots do not include logs or a formal incident record.

Engineering lessons

Chunking is a quality setting. Chunk size affects the evidence the model sees. It should be tested on representative questions, not chosen once and forgotten.

Similarity is only a first pass. A passage can look similar to a query without answering it. Query-aware filtering may help, though it adds model cost and latency.

Index lifecycle matters. Building embeddings on every request wastes time. Index reuse can remove initialization work from the critical path.

Streaming changes perception. A faster first token can make an application feel more responsive, even if total answer completion takes the same time.

The source also reports that FAISS search remained under 200 ms for collections approaching 50,000 chunks. That is a reported observation, not a general scaling law. Actual search performance depends on index type, hardware, vector count, dimensionality, and accuracy settings.

Implementation outline

The supplied document names a Streamlit interface, a console chatbot, a Python knowledge-base file, dependency definitions, and spreadsheet-based response logging. The following structure and function illustrate the design without claiming to reproduce the original repository exactly.

Illustrative project layout
rag-chatbot/
├── data/
│   └── python_guide.txt
├── chatbot.py
├── streamlit_app.py
├── requirements.txt
└── .streamlit/
    └── secrets.toml  # local secret file, do not commit
Simplified retrieval path
def answer_question(question, vector_store, model):
    passages = vector_store.similarity_search(question, k=3)
    context = "\n\n".join(doc.page_content for doc in passages)
    prompt = (
        "Answer using the context when possible. "
        "If the context is insufficient, say so.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return model.invoke(prompt)

This is explanatory pseudocode, not verified source code. The screenshot describes additional reranking and streaming behavior that are not implemented in this short example.

For a deployed version, secrets should remain outside source control, logs should avoid sensitive user inputs, and citation support should connect each answer to the passages it used.

Limits and next steps

The project makes a useful case for measuring retrieval and latency together. The supplied screenshots do not, however, include the data or code needed to reproduce their benchmark claims.

Evaluation

Publish representative questions, relevance labels, response annotations, and the exact scoring method. Separate retrieval precision from answer grounding and hallucination rates.

Performance

Capture request-level timings for embedding, retrieval, filtering, model output, and rendering. Report time to first token and full completion independently.

Implementation

Make chunk units, retrieval k, index persistence, reranking, and query expansion settings explicit. Record dependency and model versions.

Operations

Keep API keys in a secret manager, add rate-limit handling and structured monitoring, and store reproducible load-test artifacts.

The reported gains are promising, but without test artifacts they should be presented as observations from the original paper rather than independent evidence of production-scale behavior.

Conclusion

This chatbot combines document chunking, embeddings, vector retrieval, and model generation to answer Python questions using an external reference. The supplied project paper reports better retrieval precision after filtering and chunk tuning, and faster responses after changes to indexing, retrieval, and output delivery.

The key takeaway is straightforward: a RAG system needs to be evaluated at each stage. Better answers depend on useful context, and lower latency depends on knowing where time is spent. The next step is to make the reported improvements repeatable with public test sets, timing traces, and clearly defined metrics.

Source and reporting notes

Primary material. Eight screenshots of Context-Aware AI Chatbot: Optimizing Retrieval Quality and Latency in Production RAG Systems, supplied by the project author for this adaptation on November 9, 2026.

Project. production-rag.netlify.app. Linked for readers; not consulted or independently verified for this adaptation.

Scope. This paper is a rewrite of the supplied document. No repository review, independent experiment, load test, or source-code inspection was performed. Performance figures are identified as reported results, and arithmetic corrections are called out where needed.