Docs
KNet Crawlers
KNet Crawlers
Web crawlers that collect sites into structured data and searchable indexes.
KNet Crawlers crawls websites and turns them into data you can search or store: pages, documents and extracted fields, indexed for hybrid (keyword and semantic) search.
Pages
| Page | What it is for |
|---|---|
| Marketplace | Crawler agents to start a new crawler from: the Native Web Crawler (with Playground and New Crawler) plus K-Nets published with the crawler tag |
| Crawlers | Your saved crawlers, each a configured crawl job |
| Runs | Execution history; each run belongs to a crawler |
| Settings | Defaults for this workspace |
Typical tasks
- Try the Native Web Crawler in its Playground, then New Crawler to save it with a start URL and limits, and run it.
- Send its pages to a Knowledge Base to answer from them.
See also KNet Crawler.