Crawl your site & the website index

What this is: How to index your existing web pages so the AI understands what your site already covers, and how to check coverage and re-crawl.

Prerequisite: Connect a CMS first: WordPress or Webflow. The crawl and index tools live alongside your CMS settings.

The Website Content Index panel

In Settings → Integrations, the Website Content Index panel shows how much of your site is indexed (for example, "N pages indexed") plus the date of the last crawl.

From this panel you can:

• Explore index: opens the dedicated index viewer at the website index.
• Re-crawl website (WordPress) or Import from Webflow (Webflow): refresh what's indexed.

WordPress crawl options

When re-crawling a WordPress site, you can toggle:

• Send crawler header
• Disable proxy
• Chrome user agent

Tip: Leave these off unless a crawl comes back empty or partial. If pages are missing, try turning on Chrome user agent (some sites only serve full content to a browser-like request).

How crawling works

• A one-time full crawl that you trigger yourself.
• Then an automatic daily delta refresh that re-indexes changed and new pages, so the index stays current without you touching it.

Check your coverage in the index viewer

Open the index viewer with Explore index at the website index. It has two tabs:

• Browse: a list of every indexed page, with its URL, title, chunk count, and index date.
• Search: type a query to see which page chunks the AI would retrieve. Use it to confirm specific content is actually indexed.

Confirm full coverage

1. Check the pages indexed count against your sitemap. The numbers should be close.
2. Use the Browse tab to spot-check that important pages are there.
3. Use the Search tab to confirm key topics come back.

Why it matters

Indexing your site lets the AI understand what you already cover. That feeds coverage scoring, internal-link suggestions, and audits across the Website overview area, so new content complements what you have instead of duplicating it.