01 · A
Problem
Enterprise manuals are long, terminology-heavy and distributed across sections. A fluent answer is not useful unless the system retrieved the right evidence and the user can inspect it.
All projects08 / 09
Requirements, prototype and evaluation
Enterprise Manual RAG Thesis
A Design Science thesis on turning technical-manual questions into a source-grounded local RAG prototype and a separated retrieval/answer evaluation plan.
- Period
- Diplom thesis
- Focus
- Enterprise RAG · Design Science · User Research
- Core stack
- Python · Local Embeddings · Hybrid Retrieval · Reranking
TL;DR
- Problem
- Enterprise manuals are long, terminology-heavy and distributed across sections. A fluent answer is not useful unless the system retrieved the right evidence and the user can inspect it.
- What I did
- Translated user needs into system requirements and designed the architecture.
- Result / limit
- Because the thesis used confidential enterprise manuals and study material, I do not publish participant counts, internal documents or unreleased evaluation tables.
01 · B
Context
I used a Design Science process spanning user needs, architecture, a local retrieval-and-generation prototype, technical evaluation and user feedback.
Workflow illustration
The answer, its passages and its evaluation belong in one view
I use this workflow illustration to show a cited answer beside ranked passages, with retrieval quality separated from answer quality.
Why it matters
A compelling demo cannot establish that an enterprise knowledge system works. Retrieval relevance, answer support and user trust are different questions and need different evidence.
Explore technical detailsWhat made it hard · System path · Key trade-offs
What made it hard
- 01
User language and manual terminology do not always match.
- 02
High vector similarity does not guarantee a passage can answer the question.
- 03
Retrieval and answer quality need separate evaluation.
- 04
Expert-created questions and relevant sources are needed for a domain benchmark.
My approach
System path
Discover
Translate user needs into retrieval, citation and local-deployment requirements
Retrieve
Combine local embeddings, lexical signals and reranking
Answer
Generate from selected passages while retaining source traceability
Evaluate
Separate technical retrieval/answer checks from interviews and feedback
Grounded answer path
Retrieval and generation remain separately inspectable
The prototype first gathers and reranks support material, then generates from selected passages while keeping source identities attached.
01need
User question
Natural language differs from manual terminology
02recall layer
Hybrid retrieval
Lexical and local embedding candidates
03precision layer
Reranking
Reorder passages for the specific question
04synthesis
LLM answer
Generate only from selected context
05trust
Citations
Expose supporting sections for inspection
Key trade-offs
Start from the use case
Design Science keeps the prototype tied to user tasks rather than one preferred model.
Evaluate two failure layers
Finding the wrong passage and answering badly from the right passage are measured separately.
Experts define relevance
The system cannot create its own benchmark and then treat that benchmark as independent evidence.
04 · A
My contribution
- Translated user needs into system requirements and designed the architecture.
- Implemented local embeddings, hybrid retrieval, reranking and LLM answer generation.
- Worked with domain experts on technical questions and combined technical checks with user feedback.
04 · B
What was implemented
- Completed the research path from requirements through a fully local RAG prototype to technical and user evaluation.
- Because the thesis used confidential enterprise manuals and study material, I do not publish participant counts, internal documents or unreleased evaluation tables.
Evaluation design
A correct passage and a correct answer are different claims
Separating the layers makes failure diagnosis possible and prevents a single aggregate score from hiding where the system broke.
01evidence access
Retrieval evaluation
Did the system find the expert-relevant material?
02response quality
Answer evaluation
Is the response supported, complete and appropriate?
03experience
User feedback
Is the workflow understandable and useful?
To protect confidential thesis materials, I do not include participant counts or detailed metrics here.
Lessons learned
01RAG is a system of source, failure-mode and trust decisions—not only a vector database.
02User feedback explains experience; it does not replace technical retrieval evidence.
What I can share
The company manuals, repository and user-study materials are confidential, so I cannot publish them. I can share the research design, local RAG architecture and evaluation method; specific participant and benchmark numbers remain private.
Public sources