Agent Knowledge Base

[jezweb]

Overview

Add a "Knowledge Base" feature to agents that allows attaching domain-specific information (documents, URLs, and text snippets) to help agents answer questions more accurately. This is inspired by ElevenLabs' agent knowledge base implementation and leverages existing infrastructure in the codebase.

Problem Statement

Currently, users must copy-paste large amounts of documentation, guides, or reference material directly into an agent's system prompt. This makes:

• System prompts difficult to maintain and update
• Hard to reuse the same documentation across multiple agents
• Challenging to organize knowledge into logical chunks
• Difficult to see what context an agent has access to

Proposed Solution

Add an "Agent Knowledge Base" section to the agent configuration that supports three input methods:

1. 📄 Upload Files - Upload .md, .txt, .json, .pdf files (drag-and-drop support)
2. 🌐 Add URL - Scrape and store web page content
3. 📝 Create Text - Write or paste text directly in the UI

All knowledge base content is automatically loaded and appended to the agent's system prompt at runtime.

UI Reference (ElevenLabs Implementation)

The following screenshots show ElevenLabs' knowledge base implementation, which serves as our design inspiration:

Main Knowledge Base View

Shows the main knowledge base interface with document list, search, and action buttons

Document Library with Multiple Input Methods

Displays all available documents with search functionality and three input methods: Add URL, Add Files, Create Text

File Upload Modal

Drag-and-drop file upload interface showing supported formats and size limits

Create Text Modal

Simple form for creating text snippets directly in the UI

Add URL Modal

URL input form for scraping web page content

User Experience (Based on ElevenLabs Pattern)

Main Knowledge Base View

┌──────────────────────────────────────────────────────────┐
│ Agent Knowledge Base                         [Add ▾]     │
│ Provide the LLM with domain-specific information to help │
│ it answer questions more accurately.                     │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  🔍 Search knowledge base...                             │
│                                                          │
│  📝 WordPress Elementor SEO Guide               [🗑️]     │
│      wp-seo-guide.md · 12.3 KB                           │
│                                                          │
│  📝 Shopify SEO Guide                           [🗑️]     │
│      shopify-guide.md · 8.7 KB                           │
│                                                          │
│  🌐 React Documentation - Getting Started       [🗑️]     │
│      https://react.dev/learn · 45.2 KB                   │
│                                                          │
│  📝 Brand Voice Guidelines                      [🗑️]     │
│      Created 2 days ago · 3.1 KB                         │
│                                                          │
│  ─────────────────────────────────────────────────────   │
│                                                          │
│  💡 Your knowledge base (69 KB) is small enough to be    │
│     included directly in the prompt for faster responses.│
│                                                          │
│  [🌐 Add URL]  [📄 Upload Files]  [📝 Create Text]       │
└──────────────────────────────────────────────────────────┘

Upload Files Modal

┌─────────────────────────────────────────┐
│  📄  Upload Files                       │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │                                 │   │
│  │         📄                      │   │
│  │                                 │   │
│  │  Click or drag files to upload  │   │
│  │     Up to 500 KB total.         │   │
│  │                                 │   │
│  │   md   txt   json   pdf         │   │
│  │                                 │   │
│  └─────────────────────────────────┘   │
│                                         │
│                        [Upload Files]   │
└─────────────────────────────────────────┘

Add URL Modal

┌─────────────────────────────────────────┐
│  🌐  Add URL                            │
│                                         │
│  URL                                    │
│  ┌─────────────────────────────────┐   │
│  │ https://example.com             │   │
│  └─────────────────────────────────┘   │
│                                         │
│  💡 We'll fetch and store the page      │
│     content for the agent to reference. │
│                                         │
│                           [Add URL]     │
└─────────────────────────────────────────┘

Create Text Modal

┌─────────────────────────────────────────┐
│  📝  Create Text                        │
│                                         │
│  Text Name                              │
│  ┌─────────────────────────────────┐   │
│  │ Enter a name for your text      │   │
│  └─────────────────────────────────┘   │
│                                         │
│  Text Content                           │
│  ┌─────────────────────────────────┐   │
│  │ Enter your text content here    │   │
│  │                                 │   │
│  │                                 │   │
│  │                                 │   │
│  └─────────────────────────────────┘   │
│                                         │
│                      [Create Text]      │
└─────────────────────────────────────────┘

Technical Implementation

Architecture

The feature leverages existing infrastructure:

1. File Storage: Existing Vercel Blob storage system (/src/lib/file-storage/)
2. Agent Instructions: Extend instructions JSON field (no DB migration)
3. System Prompt Builder: Modify buildUserSystemPrompt() to load knowledge
4. Web Scraping: Use existing web content fetching if available, or add simple scraper

Data Structure

Extend AgentInstructionsSchema:

export const KnowledgeBaseItemSchema = z.object({
  id: z.string(),              // Unique ID for the item
  type: z.enum(['file', 'url', 'text']),
  name: z.string(),            // Display name

  // Storage (one of these depending on type)
  storageKey: z.string().optional(),     // For uploaded files
  url: z.string().optional(),            // Original URL (for type: 'url')
  content: z.string().optional(),        // Inline content (for type: 'text')

  // Metadata
  size: z.number(),            // Size in bytes
  contentType: z.string(),     // MIME type
  createdAt: z.date(),
  updatedAt: z.date(),
});

export const AgentInstructionsSchema = z.object({
  role: z.string().optional(),
  systemPrompt: z.string().optional(),
  mentions: z.array(ChatMentionSchema).optional(),
  knowledgeBase: z.array(KnowledgeBaseItemSchema).optional(),
});

System Prompt Integration

// src/lib/ai/prompts.ts
export const buildUserSystemPrompt = async (
  user?: User,
  userPreferences?: UserPreferences,
  agent?: Agent,
) => {
  let prompt = `You are ${assistantName}`;

  // ... existing prompt building ...

  // Load and append knowledge base
  if (agent?.instructions?.knowledgeBase?.length > 0) {
    const knowledgeContents = await Promise.all(
      agent.instructions.knowledgeBase.map(async (item) => {
        try {
          let content = '';

          switch (item.type) {
            case 'file':
              const buffer = await serverFileStorage.download(item.storageKey!);
              content = buffer.toString('utf-8');
              break;
            case 'url':
              // Load cached URL content from storage
              const urlBuffer = await serverFileStorage.download(item.storageKey!);
              content = urlBuffer.toString('utf-8');
              break;
            case 'text':
              content = item.content!;
              break;
          }

          return `### ${item.name}\n\n${content}`;
        } catch (error) {
          logger.error(`Failed to load knowledge base item: ${item.id}`, error);
          return `### ${item.name} (failed to load)`;
        }
      })
    );

    prompt += `

<knowledge_base>
The following knowledge base has been provided for your reference. Use this information to answer questions accurately:

${knowledgeContents.join('\n\n---\n\n')}
</knowledge_base>`;
  }

  return prompt.trim();
}

API Endpoints

New endpoints needed:

1. POST /api/agent/knowledge/upload - Upload file

   • Validates file type and size
   • Stores in Vercel Blob under agents/{agentId}/knowledge/
   • Returns metadata

2. POST /api/agent/knowledge/url - Fetch URL content

   • Scrapes webpage using existing tools or simple fetch
   • Stores content in Vercel Blob
   • Returns metadata

3. POST /api/agent/knowledge/text - Create text item

   • Validates text size
   • Stores content in Vercel Blob or inline (for small text)
   • Returns metadata

4. DELETE /api/agent/knowledge/:id - Delete knowledge item

   • Removes from agent instructions
   • Deletes file from storage
   • Returns success

5. PATCH /api/agent/:agentId/knowledge/reorder - Reorder items

   • Updates order in agent instructions array

Web Scraping Strategy

Option 1: Use existing web content tool (if available in codebase)
Option 2: Add simple scraper

// src/lib/scraper/web-content.ts
export async function fetchAndCleanWebContent(url: string): Promise<string> {
  const response = await fetch(url);
  const html = await response.text();

  // Use existing HTML-to-markdown converter or simple cleanup
  // Could use libraries like: turndown, jsdom, mozilla/readability
  const cleanedText = htmlToMarkdown(html);

  return cleanedText;
}

Configuration

• Supported file types: .md, .txt, .json, .pdf (requires PDF parser)
• Maximum size: 500KB total per agent (~125,000 tokens)
• Individual file limit: 100KB per item
• Storage: Existing file storage system (Vercel Blob/S3)

Benefits

1. Uses Existing Infrastructure: File storage, upload APIs already exist
2. No Breaking Changes: Purely additive, doesn't affect existing agents
3. Flexible Storage: JSON field in DB, actual content in cloud storage
4. Reusable Pattern: Could extend to shared knowledge bases later
5. Better UX: Clean separation of system prompt vs. reference materials
6. Search & Organization: Easy to find and manage knowledge items

Implementation Phases

Phase 1: MVP - File Upload (Week 1)

Goal: Basic file upload and system prompt integration

• Extend AgentInstructionsSchema with knowledgeBase field
• Modify buildUserSystemPrompt() to load knowledge items
• Create file upload API endpoint
• Add basic UI section to agent edit form
• File upload button (no drag-and-drop yet)
• Display uploaded files in list
• Delete functionality
• Support .md and .txt files only
• Enforce 500KB total limit
• Basic error handling

Deliverable: Can upload markdown/text files to agents and they appear in context

Phase 2: Enhanced Upload UX (Week 2)

Goal: Polished file upload experience

• Drag-and-drop upload zone
• File type icons (📝, 📄, 📋, 🌐)
• Show file size in human-readable format
• Upload progress indicator
• Size warnings (approaching limit, over limit)
• Search/filter knowledge items
• Add .json file support
• Better error messages and validation
• Loading states during upload

Deliverable: Professional file upload UX matching ElevenLabs quality

Phase 3: Create Text Feature (Week 3)

Goal: Allow inline text creation

• "Create Text" modal UI
• Text name input field
• Large textarea for content
• Character/size counter
• Save text as knowledge item
• Store small text inline, large text in storage
• Edit existing text items
• Preview text content

Deliverable: Can create text snippets without uploading files

Phase 4: URL Ingestion (Week 4)

Goal: Scrape and store web page content

• "Add URL" modal UI
• URL validation
• Web scraping implementation
  • Fetch HTML
  • Clean and convert to markdown
  • Handle errors (404, timeout, etc.)
• Store scraped content
• Show original URL in metadata
• Re-scrape functionality (update content)
• Handle large pages (chunking or size limit)

Deliverable: Can add documentation from URLs

Phase 5: Advanced Features (Week 5)

Goal: Polish and optimization

• PDF file support (add PDF parser library)
• Knowledge base size optimization message
  • "Small enough for direct inclusion" (< 100KB)
  • "Consider RAG for large bases" (> 300KB)
• Reorder knowledge items (drag-and-drop)
• Duplicate detection (same file uploaded twice)
• Preview knowledge item content (modal)
• Edit knowledge item name
• Batch delete
• Export knowledge base (download all as zip)

Deliverable: Feature-complete knowledge base system

Phase 6: Shared Knowledge Library (Future)

Goal: Reuse knowledge across agents

• Global knowledge library (user-level)
• Checkbox to select items per agent
• Track "used by X agents"
• Shared vs. agent-specific toggle
• Knowledge base templates

Deliverable: ElevenLabs-style document library

Files to be Modified

Backend

• src/types/agent.ts - Type definitions
• src/lib/ai/prompts.ts - System prompt builder (make async, load knowledge)
• src/app/api/chat/route.ts - Handle async prompts
• src/app/api/chat/openai-realtime/route.ts - Handle async prompts
• src/app/api/agent/knowledge/upload/route.ts - New file
• src/app/api/agent/knowledge/url/route.ts - New file
• src/app/api/agent/knowledge/text/route.ts - New file
• src/app/api/agent/knowledge/[id]/route.ts - New file (delete)
• src/lib/scraper/web-content.ts - New file (URL scraping)

Frontend

• src/components/agent/edit-agent.tsx - Main UI additions
• src/components/agent/knowledge-base-section.tsx - New file
• src/components/agent/knowledge-base-item.tsx - New file
• src/components/agent/add-knowledge-dropdown.tsx - New file
• src/components/agent/modals/upload-files-modal.tsx - New file
• src/components/agent/modals/add-url-modal.tsx - New file
• src/components/agent/modals/create-text-modal.tsx - New file

Documentation

• docs/features/KNOWLEDGE_BASE.md - New file (user guide)
• docs/architecture/KNOWLEDGE_BASE_ARCHITECTURE.md - New file (dev docs)

Dependencies

Required Libraries

{
  "dependencies": {
    // For web scraping (choose one)
    "@mozilla/readability": "^0.5.0",     // Clean article extraction
    "turndown": "^7.1.2",                 // HTML to Markdown
    "jsdom": "^23.0.0",                   // DOM parsing

    // For PDF support (Phase 5)
    "pdf-parse": "^1.1.1",                // PDF text extraction

    // Already available (likely)
    "zod": "^3.x",                        // Validation
    "@vercel/blob": "^0.x"                // Storage
  }
}

Estimated Effort

Phase	Tasks	Time Estimate
Phase 1: MVP	Backend + basic UI	8-10 hours
Phase 2: Upload UX	Drag-drop, icons, search	6-8 hours
Phase 3: Create Text	Modal + inline text	4-5 hours
Phase 4: URL Scraping	Scraper + URL modal	6-8 hours
Phase 5: Advanced	PDF, optimization, polish	8-10 hours
Phase 6: Shared Library	Multi-agent reuse	10-12 hours
Documentation	User + dev docs	2-3 hours

Total MVP (Phases 1-2): ~14-18 hours (~15-20 min with Claude Code CLI)
Total Feature-Complete (Phases 1-5): ~32-41 hours (~30-40 min with Claude Code CLI)

Example Use Cases

1. Customer Support Agent

Knowledge Base:

• Product documentation (uploaded .md)
• FAQ from company website (scraped URL)
• Troubleshooting guide (created text)
• API reference (uploaded .json)

2. Code Review Agent

Knowledge Base:

• Style guide (uploaded .md)
• Security best practices (scraped from OWASP)
• Component library docs (URL)
• Team coding standards (created text)

3. Content Writing Agent

Knowledge Base:

• Brand voice guidelines (uploaded .pdf)
• Terminology glossary (created text)
• Competitor analysis (uploaded .md)
• SEO best practices (scraped URL)

4. SEO Optimization Agent

Knowledge Base:

• WordPress SEO Guide (uploaded .md)
• Shopify SEO Guide (uploaded .md)
• Google Search Central docs (scraped URL)
• Client-specific keywords (created text)

Edge Cases & Considerations

1. File Size Management

Problem: Knowledge base too large for context window
Solution:

• Hard limit at 500KB total
• Show warning at 400KB
• Smart message: "Consider splitting into multiple specialized agents"

2. Stale URL Content

Problem: Web page content changes after scraping
Solution:

• Show "Last updated" timestamp
• Add "Re-scrape" button to refresh
• Cache with TTL (optional)

3. Failed File Downloads

Problem: File deleted from storage or download fails
Solution:

• Log error server-side
• Continue building prompt without failed item
• Show warning in UI: "1 knowledge item failed to load"

4. Large File Processing

Problem: 100KB text file takes time to load
Solution:

• Load files in parallel (Promise.all)
• Cache downloaded content per request
• Consider lazy loading (load only when agent used)

5. Encoding Issues

Problem: Non-UTF-8 files fail to parse
Solution:

• Validate encoding on upload
• Convert to UTF-8 if possible
• Reject invalid files with clear error

6. PDF Parsing Quality

Problem: Complex PDFs don't parse well
Solution:

• Use quality parser (pdf-parse)
• Show preview before adding
• Allow manual text extraction (copy-paste)

Security Considerations

1. File Type Validation: Server-side MIME type checking
2. Content Scanning: Check for malicious content in uploads
3. URL Validation: Prevent SSRF attacks, whitelist protocols (http/https)
4. Size Limits: Prevent DoS via large uploads
5. Access Control: Only agent owner can modify knowledge base
6. Storage Isolation: Files namespaced by agent ID

Success Metrics

• ✅ Users can upload files and see them in agent context
• ✅ Agents provide accurate answers using knowledge base
• ✅ Knowledge base reduces need for long system prompts
• ✅ Upload success rate > 95%
• ✅ URL scraping success rate > 80%
• ✅ Average knowledge base size: 50-150KB (sweet spot)

Future Enhancements (Post-Launch)

1. Vector Search / RAG: For large knowledge bases (> 500KB)
2. Knowledge Base Analytics: Track which items are most useful
3. Version Control: Track changes to knowledge items
4. Knowledge Base Templates: Pre-built knowledge sets (e.g., "React Developer")
5. Collaborative Knowledge: Share knowledge bases across team
6. Auto-Chunking: Intelligently split large documents
7. Citation: Agent cites which knowledge item answered question

Questions for Discussion

1. Should we support RAG for large knowledge bases? (Phase 6+)

   • Pro: Unlimited knowledge size
   • Con: Added complexity, requires vector DB

2. Should knowledge items be editable after creation?

   • Pro: Easy to fix typos/update content
   • Con: More complex UI

3. Should we support folders/categories for organization?

   • Pro: Better organization for many items
   • Con: Added UI complexity

4. Should URL scraping be recursive (follow links)?

   • Pro: Can scrape entire doc sites
   • Con: Hard to control, slow, large size

5. Should we support collaborative knowledge bases (team feature)?

   • Pro: Great for teams
   • Con: Requires permissions system

Comparison with ElevenLabs

Feature	ElevenLabs	Our Implementation
File Upload	✅ Up to 21MB each, epub/pdf/docx/txt/html	✅ Up to 100KB each, md/txt/json/pdf
URL Ingestion	✅ Simple URL input	✅ Same
Create Text	✅ Name + content modal	✅ Same
Document Library	✅ Searchable, reusable	🔮 Phase 6
Drag & Drop	✅ Yes	✅ Phase 2
File Type Icons	✅ Yes	✅ Phase 2
Size Optimization Messages	✅ "Small enough for direct inclusion"	✅ Phase 5
RAG for Large Knowledge	✅ Mentioned	🔮 Future

Conclusion

This feature is highly feasible and aligns perfectly with the existing architecture. The ElevenLabs pattern is well-designed and user-friendly, and we can implement a similar experience while leveraging our existing file storage, agent system, and UI components.

Recommended Approach:

1. Start with Phase 1 MVP (file upload) to validate the approach
2. Quickly add Phase 2 (UX polish) to match quality expectations
3. Roll out Phases 3-4 (text + URL) to achieve feature parity with ElevenLabs
4. Iterate on Phase 5+ based on user feedback

The phased approach allows us to ship value quickly while building toward a comprehensive knowledge base system.

[cgoinglove]

good idea. i want it