> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ravenna.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web scraping

> Import content from public websites by scraping single pages or crawling up to 100 pages per domain to build agent knowledge from help centers.

Import content from public websites to power agent responses. Scrape individual pages or crawl entire documentation sites to build knowledge from help centers, docs, and web resources.

## What you can do

* Import single pages or crawl up to 100 pages from a domain
* Scrape JavaScript-rendered and formatted content
* Auto-sync to keep website content up-to-date
* Organize crawled sites with automatic folder structures

## Setup

<Steps>
  <Step title="Select Website as your source">
    When adding documents to a knowledge folder, select **Website** from the available sources.
  </Step>

  <Step title="Enter the URL">
    Provide the URL of the webpage you want to import.
  </Step>

  <Step title="Enable crawling (optional)">
    Toggle crawling on to import up to 100 pages from the same domain, or leave it off to import only the single page.
  </Step>

  <Step title="Import content">
    Content is scraped and imported from the specified URL.
  </Step>
</Steps>

<Callout icon="link" color="#6B7280">Learn more about [managing knowledge folders](/documentation/automate/knowledge/overview)</Callout>

## How it works

Browser rendering scrapes website content for accurate extraction:

* Main page content and text.
* Formatted content (headings, lists, paragraphs).
* JavaScript-rendered content.
* Publicly accessible information only.

<AccordionGroup>
  <Accordion title="Single page import" icon="file">
    By default, content is imported from only the single URL you provide. This is ideal for:

    * Specific documentation pages.
    * Help articles.
    * FAQ pages.
    * Individual blog posts.
  </Accordion>

  <Accordion title="Website crawling" icon="cloud-download">
    Enable crawling to import entire documentation sites automatically:

    * **Automatic discovery**: Links within the same domain are followed to discover and import connected pages.
    * **Breadth-first crawling**: Pages are crawled level by level for comprehensive coverage.
    * **Page limit**: Maximum of 100 pages crawled to prevent overloading your website.
    * **Folder structure**: Crawled sites are organized under a root folder named after the domain.
    * **Same-domain only**: Crawling stays within the original domain, external links are filtered out.

    <Note>Crawling respects the same domain as the starting URL. External links are automatically filtered out.</Note>
  </Accordion>
</AccordionGroup>

## Requirements

<Warning>
  * Website must be publicly accessible (no authentication required).
  * Content must be available without login or paywalls.
</Warning>

## Managing imported content

After import:

* Archive pages to exclude them from agent responses.
* Delete pages that are no longer needed.

<Callout icon="link" color="#6B7280">Learn more about [managing documents](/documentation/automate/knowledge/overview#managing-documents)</Callout>

## Auto-sync

When auto-sync is enabled, website content stays up-to-date:

* Page content is re-scraped during sync to capture updates.
* Changes to the webpage are reflected in your knowledge base.
* If the page becomes unavailable, the sync fails and you're notified.

<Warning>Auto-sync re-scrapes the same URLs only. It does not discover new pages or follow links, even if crawling was initially enabled.</Warning>