RAG Knowledge Base Architect

Design retrieval-augmented generation systems with corpus strategy, chunking rules, metadata schemas, evaluation plans, and citation-safe answer flows.

The Prompt

You are a senior AI architect specializing in retrieval-augmented generation (RAG). Design a production-ready knowledge base system that improves answer quality, traceability, and maintainability.

## 1. Business Goal and User Context
- Summarize the use case in one sentence
- Identify target users
- Define the top 5 questions the system must answer well
- Define unacceptable failure modes

## 2. Source Audit
For each knowledge source, evaluate:
- File or source type
- Update frequency
- Trust level
- Access restrictions
- Typical structure quality
- Cleanup needs before ingestion

## 3. Ingestion Architecture
Specify:
- Connectors or ingestion paths
- Parsing and normalization steps
- Deduplication rules
- Redaction or privacy controls
- Refresh cadence
- Versioning strategy

## 4. Chunking and Metadata Strategy
Recommend:
- Chunk size and overlap
- Document splitting rules
- Metadata schema
- Citation format
- Access control tags
- When to keep full sections intact vs split aggressively

## 5. Retrieval Pipeline
Design:
- Query preprocessing
- Embedding or indexing strategy
- Hybrid search considerations
- Re-ranking approach
- Filtering rules
- Citation selection logic

## 6. Answer Generation Rules
- Require evidence-backed responses
- Define when the assistant should say "I don't know"
- Require source attribution
- Separate retrieved facts from model-generated synthesis
- Prevent unsupported claims

## 7. Evaluation Plan
Create an evaluation framework with:
- Golden questions
- Relevance checks
- Citation accuracy checks
- Hallucination failure cases
- Latency targets
- Escalation thresholds

## 8. Output Format
Return:
1. System overview
2. Source readiness assessment
3. Recommended architecture
4. Chunking and metadata policy
5. Retrieval and ranking design
6. Answer policy
7. Evaluation plan
8. MVP rollout plan

Inputs:
Use case: [DESCRIBE]
Users: [WHO]
Knowledge sources: [FILES / DOCS / WIKIS / TICKETS / URLS]
Security constraints: [NONE / INTERNAL / REGULATED]
Expected question types: [LIST]
Volume and update frequency: [DESCRIBE]

How to Use This Prompt

1. Give concrete source types instead of saying “internal docs.”
2. Include at least three representative question types.
3. Ask for failure modes before you choose a vector database or embedding setup.
4. Use the chunking section to align engineering and content teams.
5. Run the evaluation plan before shipping any chat UI.

Example Input

Use case: Internal support copilot for a B2B SaaS company
Users: Support agents and customer success managers
Knowledge sources: Help center, product docs, incident postmortems, internal runbooks, Zendesk macros
Security constraints: Internal
Expected question types: Troubleshooting, policy clarification, feature limitations, escalation steps
Volume and update frequency: 8,000 articles and docs, updated daily

Example Output

1. System Overview

Build a support-focused RAG assistant optimized for troubleshooting accuracy, policy consistency, and fast retrieval of current operational guidance.

2. Source Readiness Assessment

• Help center: High trust, customer-safe language, frequent updates
• Runbooks: High value but inconsistent structure, needs cleanup
• Postmortems: Good for root cause context, weak for prescriptive answers unless tagged carefully
• Macros: Useful for response phrasing, should not be treated as policy sources

3. Recommended Architecture

• Scheduled ingestion from docs and runbooks
• Metadata-enriched indexing with document type, product area, environment, audience, and last updated date
• Hybrid retrieval with semantic search plus keyword filters for product names and error codes
• Response layer that cites 2-3 retrieved snippets before generating a final answer

4. Chunking Policy

• Product docs: Section-based chunks with moderate overlap
• Runbooks: Keep procedures intact by step group
• Postmortems: Split into issue summary, timeline, root cause, and prevention actions
• Metadata required: source type, owner, last reviewed date, audience, product, permission tier

5. Evaluation Plan

• 100 golden support questions across 10 product areas
• Score retrieval relevance, answer correctness, citation accuracy, and refusal quality
• Fail the response if citation does not support the core claim

6. MVP Rollout

• Phase 1: Internal pilot with support leads
• Phase 2: Broader support access with feedback capture
• Phase 3: Add macro suggestions and escalation routing

Practical Variations for RAG Knowledge Base Architect

• SOP mode: Ask for trigger, input, output, owner, and acceptance criteria for each step.
• Automation mode: List systems involved and ask where AI should draft, classify, summarize, or route work.
• Monitoring mode: Ask for quality checks, failure signals, and human review points.

Follow-Up Prompts

Use these after the first answer:

• “Rewrite this using only the context I provided. Label assumptions instead of hiding them.”
• “Give me a conservative version, a direct version, and a version optimized for speed.”
• “Create a final review checklist I can use before I publish, send, ship, or present this.”

What Makes This Page Different

This page is useful when you are working on rag knowledge base architect and need more than a blank chat box. It gives you a starting prompt, context checklist, review criteria, and practical variations so the answer can be tested instead of merely accepted. If your task is broader, start with a workflow guide first, then come back to this prompt once the input, audience, and success criteria are clear.