What You Can Add to the Knowledge Base

  • Website pages
  • Product documentation
  • Service descriptions
  • Policies (pricing, refunds, support)
  • Custom written content

Ways to Build Your Knowledge Base

1. Website Scraping

Automatically pull content from your website to create a knowledge base.

Best for:

  • Marketing websites
  • Product pages
  • Help content

2. Manual Content

Add custom text directly when:

  • You don’t have a website
  • You want full control over answers
  • You want to simplify complex content

Web Scraping Module

The Web Scraping Module automatically gathers and integrates relevant data from external websites into your chatbot’s knowledge base. It streamlines onboarding by pre-populating initial content from your website so the bot can respond effectively from day one.

Core System Description

The module operates in two phases:

  • Onboarding Homepage Scraping: During bot setup, the system automatically scrapes the provided website’s homepage to create an initial dataset for responses.
  • Fallback Phase (Homepage Scraping): If a sitemap.xml file is unavailable, the system reverts to scraping the homepage to extract valuable information.

Enhanced Onboarding Features

  • First Page Scraping During Setup

    • Automatically collects data from the provided website URL’s homepage during bot registration
    • Extracts visible, high-value content (e.g., service descriptions, FAQs, business offerings) to seed a preliminary knowledge base
  • Quick Setup Workflow

    • Reduces manual effort by preloading chatbot data based on homepage content
    • Ensures the chatbot is ready for basic interactions immediately after onboarding
  • User Notifications

    • Shows progress indicators during onboarding (e.g., “scraping in progress”, “scraping completed”)
    • Sends a post-onboarding confirmation that the knowledge base has been populated

This onboarding flow guarantees a smoother setup while ensuring the chatbot has enough data to deliver accurate and contextually relevant responses from the start.

Best Practices

FAQ vs Knowledge Base (Quick Difference)

FeatureFAQKnowledge Base
StructureFixed Q&AContextual understanding
Best forCommon questionsOpen-ended queries
ResponseDirectAI-generated
ControlHighFlexible
Learning depthLowHigh
  1. Start with FAQs for top 10–20 common questions
  2. Add Knowledge Base content for deeper understanding
  3. Review chatbot responses
  4. Optimize based on real conversations

Next Steps